Abstracts
Abstract
Although massive open online courses (MOOCs) have attracted much worldwide attention, scholars still understand little about the specific elements that students find engaging in these large open courses. This study offers a new original contribution by using a machine learning classifier to analyze 24,612 reflective sentences posted by 5,884 students, who participated in one or more of 18 highly rated MOOCs. Highly rated MOOCs were sampled because they exemplify good practices or teaching strategies. We selected highly rated MOOCs from Coursetalk, an open user-driven aggregator and discovery website that allows students to search and review various MOOCs. We defined a highly rated MOOC as a free online course that received an overall five-star course quality rating, and received at least 50 reviews from different learners within a specific subject area. We described six specific themes found across the entire data corpus: (a) structure and pace, (b) video, (c) instructor, (d) content and resources, (e) interaction and support, and (f) assignment and assessment. The findings of this study provide valuable insight into factors that students find engaging in large-scale open online courses.
Keywords:
- MOOCs,
- massive open online courses,
- engagement,
- text mining,
- machine learning
Appendices
Bibliography
- Altman, N. S. (1992). An introduction to kernel and nearest-neighbor nonparametric regression. The American Statistician, 46(3), 175-185. doi: 10.2307/2685209
- Anderson, A., Huttenlocher, D., Kleinberg, J., & Leskovec, J. (2014). Engaging with massive online courses. Proceedings of the 23rd international conference on World Wide Web, 687-698. doi: 10.1145/2566486.2568042
- Ames, C. (1992). Achievement goals, motivational climate, and motivational processes. In G. Roberts (Ed.), Motivation in Sports and Exercise (pp. 161-176). Champaign, IL: Human Kinetics Books.
- Baxter, J. A., & Haycock, J. (2014). Roles and student identities in online large course forums: Implications for practice. The International Review of Research in Open and Distributed Learning, 15(1), 20-40. doi: 10.19173/irrodl.v15i1.1593
- Beldarrain, Y. (2006). Distance education trends: Integrating new technologies to foster student interaction and collaboration. Distance Education, 27(2), 139-153. doi: 10.1080/01587910600789498
- Bishop, C. M. (2006). Pattern recognition and machine learning. Singapore: Springer.
- Bonk, C. J., & Wisher, R. A. (2000). Applying collaborative and e-learning tools to military distance learning: A research framework. (Technical Report #1107), US Army Research Institute for the Behavioral and Social Sciences, Alexandria, VA.
- Box, G. E. P., & Draper, N. R. (1987). Empirical model building and response surfaces. New York, NY: John Wiley & Sons.
- Business Wire. (2014). Coursetalk and edX collaborate to integrate online course reviews platform. Retrieved from http://www.businesswire.com/news/home/20140417005360/en#.VCohj2eSx8F
- Chiu, K. F., & Hew, K. F. (2018). Factors influencing peer learning and performance in MOOC asynchronous online discussion forum. Australasian Journal of Educational Technology, 34(4), 16-28. doi: 10.14742/ajet.3240
- Coetzee D., Fox, A., Hearst, M.A., & Hartmann, B. (2014). Should your MOOC forum use a reputation system? In Proceedings of CSCW 2014, ACM Press, 1176-1187. doi: 10.1145/2531602.2531657
- Coffrin, C., de Barba, P., Corrin, L., & Kennedy, G. (2014). Visualizing patterns of student engagement and performance in MOOCs. In Proceedings of LAK 2014, ACM Press (2014), 83-92. doi: 10.1145/2567574.2567586
- Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46. doi: 10.1177/001316446002000104
- Conole, G. (2013). MOOCs as disruptive technologies: Strategies for enhancing the learner experience and quality of MOOCs. RED, Revista de Educación a Distancia. Número 39. doi: 10.6018/red/50/2
- Cortes, C., & Vapnik, V. (1995). Support vector machine. Machine Learning, 20(3), 273-297. Retrieved from http://image.diku.dk/imagecanon/material/cortes_vapnik95.pdf
- Cox, D. R. (1958). The regression analysis of binary sequences. Journal of the Royal Statistical Society. Series B (Methodological), 215-242. Retrieved from https://www.jstor.org/stable/2983890?seq=1#page_scan_tab_contents
- Daniel, J. (2012). Making sense of MOOCs: Musings in a maze of myth, paradox, and possibility. Journal of Interactive Media in Education, 3. doi: 10.5334/2012-18
- de Barba, P. G., Kennedy, G. E., & Ainley, M. D. (2016). The role of students' motivation and participation in predicting performance in a MOOC. Journal of Computer Assisted Learning, 32, 218-231. doi: 10.1111/jcal.12130
- Dillahunt, T. R., Wang, B. Z., & Teasley, S. (2014). Democratizing higher education: Exploring MOOC use among those who cannot afford a formal education. International Review of Research in Open and Distance Learning, 15(5), 177-196. doi: 10.19173/irrodl.v15i5.1841
- Elliot, A. J., & Church, M. A. (1997). A hierarchical model of approach and avoidance achievement motivation. Journal of Personality and Social Psychology, 72, 218-232. doi: 10.1037/0022-3514.72.1.218
- Fredricks, J. A., Blumenfeld, P. C., & Paris, A. (2004). School engagement: Potential of the concept: State of the evidence. Review of Educational Research, 74, 59-119. doi: 10.3102/00346543074001059
- Friedman, J. H. (2001). Greedy function approximation: A gradient boosting machine. Annals of statistics, 1189-1232. Retrieved from https://statweb.stanford.edu/~jhf/ftp/trebst.pdf
- Gore, H. (2014). Massive Open Online Courses (MOOCs) and their impact on academic library services: Exploring the issues and challenges. New Review of Academic Librarianship, 20(1), 4-28. doi: 10.1080/13614533.2013.851609
- Guo, P. J., Kim, J., & Rubin, R. (2014). How video production affects student engagement: An empirical study of MOOC videos. In Proceedings of the first ACM conference on Learning@ scale conference (pp. 41-50). ACM. doi: 10.1145/2556325.2566239
- Hazari, S., North, A., & Moreland, D. (2009). Investigating pedagogical value of wiki technology. Journal of Information Systems Education, 20(2), 187-198. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.476.714&rep=rep1&type=pdf
- Hew, K. F. (2015). Towards a model of engaging online students: Lessons from MOOCs and four policy documents. International Journal of Information and Education Technology, 5(6), 425-431. doi: 10.7763/IJIET.2015.V5.543
- Hew, K. F. (2016). Promoting engagement in online courses: What strategies can we learn from three highly rated MOOCs. British Journal of Educational Technology, 47(2), 320-341. doi: 10.1111/bjet.12235
- Hew, K. F., & Cheung, W. S. (2014). Students' and Instructors' Use of Massive Open Online Courses (MOOCs): Motivations and Challenges. Educational Research Review, 12, 45-58. https://doi.org/10.1016/j.edurev.2014.05.001
- Kearsley, G., & Shneiderman, B. (1998). Engagement theory: A framework for technology-based teaching and learning. Educational technology, 38(5), 20-23. Retrieved from https://www.jstor.org/stable/44428478?seq=1#page_scan_tab_contents
- Keller, J. M. (1987). Development and use of the ARCS model of instructional design. Journal of Instructional Development, 10(3), 2-10. doi: 10.1007/BF02905780
- Kizilcec, R., Piech, C., & Schneider, E. (2013). Deconstructing disengagement: analyzing learner subpopulations in massive open online courses. In Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp. 170-179). New York, NY: ACM. doi: 10.1145/2460296.2460330
- Kloft, M., Stiehler, F., Zheng, Z., & Pinkwart,N. (2014). Predicting MOOC dropout over weeks using machine learning models. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pp. 60-65. Association for Computational Linguistics. Retrieved from http://www.aclweb.org/anthology/W/W14/W14-41.pdf#page=67
- Knowles, M. S., Holton, E. F., & Swanson, R. A. (2011). The adult learner: The definitive classic in adult education and human resource development. (7th ed.). New York: Elsevier Inc.
- Knowlton, D. S. (2000). A theoretical framework for the online classroom: A defense and delineation of a student‐centered pedagogy. New Directions for Teaching and Learning, 2000(84), 5-14. doi: 10.1002/tl.841
- Kolowich, S. (2013). The professors who make the MOOCs. Chronicle of Higher Education, 59(28), A20-A23. Retrieved from http://publicservicesalliance.org/wp-content/uploads/2013/03/The-Professors-Behind-the-MOOC-Hype-Technology-The-Chronicle-of-Higher-Education.pdf
- Lombardi, M. M. (2013). The inside story: Campus decision making in the wake of the latest MOOC Tsunami. MERLOT Journal of Online Learning and Teaching, 9(2), 239-248. Retrieved from https://search.proquest.com/docview/1500421747?pq-origsite=gscholar
- McHugh, M. L. (2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22(3), 276-282. doi: 10.11613/BM.2012.031
- Meece, J. L., Blemenfeld, P. C., & Hoyle, R. H. (1988). Students' goal orientations and cognitive engagement in classroom activities. Journal of Educational Psychology, 80, 514-523. doi: 10.1037/0022-0663.80.4.514
- Murphy, K. P. (2012). Machine learning: a probabilistic perspective. Cambridge, MA: MIT press.
- Robertson, S. (2004). Understanding inverse document frequency: On theoretical arguments for IDF. Journal of Documentation, 60(5), 503-520. doi: 10.1108/00220410410560582
- Rodriguez. C. O. (2012). MOOCs and the AI-Stanford like courses: Two successful and distinct course formats for massive open online courses. European Journal of Open, Distance and E-Learning. Retrieved from http://www.eurodl.org/index.php?p=archives&year=2013&halfyear=2&article=516
- Shah, D. (2016, December 25). By the numbers: MOOCS in 2016. Retrieved from https://www.class-central.com/report/mooc-stats-2016/
- Shenton, A. K. (2004). Strategies for ensuring trustworthiness in qualitative research projects. Education for Information, 22, 63-75. doi: 10.3233/EFI-2004-22201
- Sims, R. (2003). Promises of interactivity: Aligning learner perceptions and expectations with strategies for flexible and online learning. Distance Education, 24(1), 87-103. doi: 10.1080/01587910303050
- Stehman, S. V. (1997). Selecting and interpreting measures of thematic classification accuracy. Remote sensing of Environment, 62(1), 77-89. doi: 10.1016/S0034-4257(97)00083-7
- Szpunar, K. K., Khan, N. Y., & Schacter, D. L. (2013). Interpolated memory tests reduce mind wandering and improve learning of online lectures. Proceedings of the National Academy of Sciences, 110(16), 6313-6317. doi: 10.1073/pnas.1221764110
- Szpunar, K. K., Jing, G. H., & Schacter, D. L. (2014). Overcoming overconfidence in learning from video-recorded lectures: Implications of interpolated testing for online education. Journal of Applied Research in Memory and Cognition, 3(3), 161-164. doi: 10.1016/j.jarmac.2014.02.001
- Wanzer, M. B. (2002). Use of humor in the classroom: The good, the bad, and the not-so-funny things that teachers say and do. In J. L. Chesebro & J. C. McCroskey (Eds.), Communication for Teachers (pp. 116-125). Boston: Allyn & Bacon.
- Warren, J., Rixner, S., Greiner, J. & Wong, S. (2014). Facilitating human interaction in an online programming course, in Proceedings of SIGCSE 2014, ACM Press, 665-670. doi: 10.1145/2538862.2538893
- Wold, S., Esbensen, K., & Geladi, P. (1987). Principal component analysis. Chemometrics and Intelligent Laboratory Systems, 2(1-3), 37-52. doi: 10.1016/0169-7439(87)80084-9
- Wu, H. C., Luk, R. W. P., Wong, K. F., & Kwok, K. L. (2008). Interpreting tf-idf term weights as making relevance decisions. ACM Transactions on Information Systems (TOIS), 26(3), 13. doi: 10.1145/1361684.1361686
- Young, J. R. (2013). What professors can learn from "hard core" MOOC students. Chronicle of Higher Education, 59(37), A4. Retrieved from https://www.chronicle.com/article/What-Professors-Can-Learn-From/139367