Abstracts
Abstract
MOOCs (massive open online courses), because of their scale and accessibility, have become a major area of interest in contemporary education. However, despite their growing popularity, the question of their quality remains a central concern, partly due to the lack of consensus on the criteria establishing such quality. This study set out to fill this gap by carrying out a systematic review of the existing literature on MOOC quality and proposing a specific quality assurance framework at a micro level. The methodology employed in this research consisted of a careful analysis of MOOC success factor’s using Biggs’ classification scheme, conducted over a four-year period from 2018 to 2022. The results highlighted the compelling need to consider various indicators across presage, process, and product variables when designing and evaluating MOOCs. This implied paying particular attention to pedagogical quality, both from the learner’s and the teacher’s point of view. The quality framework thus developed is of significant importance. It offers valuable guidance to MOOC designers, learners, and researchers, providing them with an in-depth understanding of the key elements contributing to MOOC quality and facilitating their continuous improvement. In addition, this study highlighted the need to address aspects for future research, including large-scale automated evaluation of MOOCs. By focusing on pedagogical quality, MOOCs can play a vital role in providing meaningful learning experiences, maximizing learner satisfaction, and ensuring their success as innovative educational systems adapted to the changing needs of contemporary education.
Keywords:
- MOOC quality,
- quality assurance,
- pedagogical quality framework,
- MOOC success factors
Download the article in PDF to read it.
Download
Appendices
Bibliography
- Albelbisi, N., Yusop, F. D., & Salleh, U. K. M. (2018). Mapping the factors influencing success of massive open online courses (MOOC) in higher education. EURASIA Journal of Mathematics, Science and Technology Education, 14(7). https://doi.org/10.29333/ejmste/91486
- Alcarria, R., Bordel, B., Martín de Andrés, D., & Robles, T. (2018). Enhanced peer assessment in MOOC evaluation through assignment and review analysis. International Journal of Emerging Technologies in Learning, 13(1), 206. https://doi.org/10.3991/ijet.v13i01.7461
- Alemayehu, L., & Chen, H. -L. (2021). Learner and instructor-related challenges for learners’ engagement in MOOCs : A review of 2014–2020 publications in selected SSCI indexed journals. Interactive Learning Environments, 31, 1–23. https://doi.org/10.1080/10494820.2021.1920430
- Alexandron, G., Wiltrout, M. E., Berg, A., & Ruipérez-Valiente, J. A. (2020, March). Assessment that matters: Balancing reliability and learner-centered pedagogy in MOOC assessment. In Proceedings of the tenth international conference on learning analytics & knowledge (pp. 512-517). https://doi.org/10.1145/3375462.3375464
- Aloizou, V., Sobrino, S. L. V., Monés, A. M., & Sastre, S. G. (2019). Quality assurance methods assessing instructional design in MOOCs that implement active learning pedagogies: An evaluative case study. CEUR Workshop Proceedings. https://uvadoc.uva.es/handle/10324/38662
- Anyatasia, F. N., Santoso, H. B., & Junus, K. (2020). An evaluation of the Udacity MOOC based on instructional and interface design principles. Journal of Physics: Conference Series, 1566(1), 012053. https://doi.org/10.1088/1742-6596/1566/1/012053
- Aparicio, M., Oliveira, T., Bacao, F., & Painho, M. (2019). Gamification: A key determinant of massive open online course (MOOC) success. Information & Management, 56(1), 39–54. https://doi.org/10.1016/j.im.2018.06.003
- Askeroth, J. H., & Richardson, J. C. (2019). Instructor perceptions of quality learning in MOOCs they teach. Online Learning, 23(4), 135-159. https://doi.org/10.24059/olj.v23i4.2043
- Avgerinos, E., & Karageorgiadis, A. (2020). The importance of formative assessment and the different role of evaluation in MOOCS. 2020 IEEE Learning With MOOCS (pp. 168–173). https://doi.org/10.1109/LWMOOCS50143.2020.9234320
- Bai, S., Hew, K. F., & Huang, B. (2020). Does gamification improve student learning outcome? Evidence from a meta-analysis and synthesis of qualitative data in educational contexts. Educational Research Review, 30, 100322. https://doi.org/10.1016/j.edurev.2020.100322
- Biggs, J. B. (1993). From theory to practice : A cognitive systems approach. Higher Education Research & Development, 12(1), Article 1. https://doi.org/10.1080/0729436930120107
- Bingöl, I., Kursun, E., & Kayaduman, H. (2019). Factors for success and course completion in massive open online courses through the lens of participant types. Open Praxis, 12(2), 223. https://doi.org/10.5944/openpraxis.12.2.1067
- Bogdanova, D., & Snoeck, M. (2018). Using MOOC technology and formative assessment in a conceptual modelling course: An experience report. Proceedings of the 21st ACM/IEEE International Conference on Model Driven Engineering Languages and Systems: Companion Proceedings (pp. 67–73). https://doi.org/10.1145/3270112.3270120
- Bonk, C. J., Zhu, M., Kim, M., Xu, S., Sabir, N., & Sari, A. R. (2018). Pushing toward a more personalized MOOC: Exploring instructor selected activities, resources, and technologies for MOOC design and implementation. The International Review of Research in Open and Distributed Learning, 19(4), Article 4. https://doi.org/10.19173/irrodl.v19i4.3439
- Buchem, I., Carlino, C., Amenduni, F., & Poce, A. (2020). Meaningful gamification in MOOCS: Designing and examining learner engagement in the open virtual mobility learning hub. Proceedings of the 14th International Technology, Education and Development Conference (pp. 9529–9534). https://doi.org/10.21125/inted.2020.1661
- Chansanam, W., Poonpon, K., Manakul, T., & Detthamrong, U. (2021). Success and challenges in MOOCs: A literature systematic review technique. TEM Journal, 10(4), 1728–1732. https://doi.org/10.18421/TEM104-32
- Chen, Y., Gao, Q., Yuan, Q., & Tang, Y. (2019). Facilitating students’ interaction in MOOCs through timeline-anchored discussion. International Journal of Human–Computer Interaction, 35(19), 1781–1799. https://doi.org/10.1080/10447318.2019.1574056
- Costello, E., Brunton, J., Brown, M., & Daly, L. (2018). In MOOCs we trust: Learner perceptions of MOOC quality via trust and credibility. International Journal of Emerging Technologies in Learning, 13(06), 214. https://doi.org/10.3991/ijet.v13i06.8447
- Costello, E., Holland, J., & Kirwan, C. (2018). The future of online testing and assessment: Question quality in MOOCs. International Journal of Educational Technology in Higher Education, 15(1), 42. https://doi.org/10.1186/s41239-018-0124-z
- Cross, J. S., Keerativoranan, N., Carlon, M. K. J., Tan, Y. H., Rakhimberdina, Z., & Mori, H. (2019). Improving MOOC quality using learning analytics and tools. 2019 IEEE Learning With MOOCS (pp. 174–179). https://doi.org/10.1109/LWMOOCS47620.2019.8939617
- Dai, H. M., Teo, T., & Rappa, N. A. (2020). Understanding continuance intention among MOOC participants: The role of habit and MOOC performance. Computers in Human Behavior, 112, 106455. https://doi.org/10.1016/j.chb.2020.106455
- Dalipi, F., Imran, A. S., & Kastrati, Z. (2018). MOOC dropout prediction using machine learning techniques: Review and research challenges. 2018 IEEE Global Engineering Education Conference (pp. 1007–1014). https://doi.org/10.1109/EDUCON.2018.8363340
- Danka, I. (2020). Motivation by gamification: Adapting motivational tools of massively multiplayer online role-playing games (MMORPGs) for peer-to-peer assessment in connectivist massive open online courses (cMOOCs). International Review of Education, 66, 75–92. https://doi.org/10.1007/s11159-020-09821-6
- Davis, D., Seaton, D., Hauff, C., & Houben, G.-J. (2018). Toward large-scale learning design: Categorizing course designs in service of supporting learning outcomes. Proceedings of the Fifth Annual ACM Conference on Learning at Scale (pp. 1–10). https://doi.org/10.1145/3231644.3231663
- Demetriadis, S., Tegos, S., Psathas, G., Tsiatsos, T., Weinberger, A., Caballe, S., Dimitriadis, Y., Sanchez, E. G., Papadopoulos, P. M., & Karakostas, A. (2018). Conversational agents as group-teacher interaction mediators in MOOCs. 2018 Learning With MOOCS (pp. 43–46). https://doi.org/10.1109/LWMOOCS.2018.8534686
- Deng, R., Benckendorff, P., & Gannaway, D. (2020). Learner engagement in MOOCs: Scale development and validation. British Journal of Educational Technology, 51(1), 245–262. https://doi.org/10.1111/bjet.12810
- Douglas, K. A., Merzdorf, H. E., Hicks, N. M., Sarfraz, M. I., & Bermel, P. (2020). Challenges to assessing motivation in MOOC learners: An application of an argument-based approach. Computers & Education, 150, 103829. https://doi.org/10.1016/j.compedu.2020.103829
- Estrada-Molina, O., & Fuentes-Cancell, D.-R. (2022). Engagement and desertion in MOOCs: Systematic review. Comunicar, 30(70), 111–124. https://doi.org/10.3916/C70-2022-09
- Farrow, R., Ferguson, R., Weller, M., Pitt, R., Sanzgiri, J., & Habib, M. (2021). Assessment and recognition of MOOCs : The state of the art. 12. Journal of Innovation in PolyTechnic Education, 3(1), 15-26.
- Fassbinder, M., Fassbinder, A., Fioravanti, M. L., & Barbosa, E. F. (2019). Towards an educational design pattern language to support the development of open educational resources in videos for the MOOC context. Proceedings of the 26th Conference on Pattern Languages of Programs (pp. 1–10). https://dl.acm.org/doi/abs/10.5555/3492252.3492274
- Gamage, D., Perera, I., & Fernando, S. (2020). MOOCs lack interactivity and collaborativeness: Evaluating MOOC platforms. International Journal of Engineering Pedagogy, 10(2), 94. https://doi.org/10.3991/ijep.v10i2.11886
- Gamage, D., Staubitz, T., & Whiting, M. (2021). Peer assessment in MOOCs: Systematic literature review. Distance Education, 42(2), 268–289. https://doi.org/10.1080/01587919.2021.1911626
- Gamage, D., Whiting, M. E., Perera, I., & Fernando, S. (2018). Improving feedback and discussion in MOOC peer assessement using introduced peers. 2018 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (pp. 357–364). https://doi.org/10.1109/TALE.2018.8615307
- Giasiranis, S., & Sofos, L. (2020). The influence of instructional design and instructional material on learners’ motivation and completion rates of a MOOC course. Open Journal of Social Sciences, 8(11), 190–206. https://doi.org/10.4236/jss.2020.811018
- Gibbs, G. (2010, September). Dimensions of quality. The Higher Education Acadamy. https://support.webb.uu.se/digitalAssets/91/a_91639-f_Dimensions-of-Quality.pdf
- Goel, Y., & Goyal, R. (2020). On the effectiveness of self-training in MOOC dropout prediction. Open Computer Science, 10(1), 246–258. https://doi.org/10.1515/comp-2020-0153
- Gregori, E. B., Zhang, J., Galván-Fernández, C., & Fernández-Navarro, F. de A. (2018). Learner support in MOOCs: Identifying variables linked to completion. Computers & Education, 122, 153–168. https://doi.org/10.1016/j.compedu.2018.03.014
- Guajardo Leal, B. E., Navarro-Corona, C., & Valenzuela González, J. R. (2019). Systematic mapping study of academic engagement in MOOC. The International Review of Research in Open and Distributed Learning, 20(2). https://doi.org/10.19173/irrodl.v20i2.4018
- Guerra, E., Kon, F., & Lemos, P. (2022). Recommended guidelines for effective MOOCs based on a multiple-case study. ArXiv:2204.03405. https://doi.org/10.48550/arXiv.2204.03405
- Hew, K. F., Hu, X., Qiao, C., & Tang, Y. (2020). What predicts student satisfaction with MOOCs: A gradient boosting trees supervised machine learning and sentiment analysis approach. Computers & Education, 145, 103724. https://doi.org/10.1016/j.compedu.2019.103724
- Hooda, M. (2020). Learning analytics lens: Improving quality of higher education. International Journal of Emerging Trends in Engineering Research, 8(5), 1626–1646. https://doi.org/10.30534/ijeter/2020/24852020
- İnan, E., & Ebner, M. (2020). Learning analytics and MOOCs. In P. Zaphiris & A. Ioannou (Eds.), Learning and collaboration technologies: Designing, developing and deploying learning experiences (Vol. 12205, pp. 241–254). Springer International Publishing. https://doi.org/10.1007/978-3-030-50513-4_18
- Jarnac de Freitas, M., & Mira da Silva, M. (2020). Systematic literature review about gamification in MOOCs. Open Learning: The Journal of Open, Distance and e-Learning, 38(1), 1–23. https://doi.org/10.1080/02680513.2020.1798221
- Julia, K., Peter, V. R., & Marco, K. (2021). Educational scalability in MOOCs: Analysing instructional designs to find best practices. Computers & Education, 161, 104054. https://doi.org/10.1016/j.compedu.2020.104054
- Jung, E., Kim, D., Yoon, M., Park, S., & Oakley, B. (2019). The influence of instructional design on learner control, sense of achievement, and perceived effectiveness in a supersize MOOC course. Computers & Education, 128, 377–388. https://doi.org/10.1016/j.compedu.2018.10.001
- Khalil, M., Wong, J., de Koning, B., Ebner, M., & Paas, F. (2018). Gamification in MOOCs: A review of the state of the art. 2018 IEEE Global Engineering Education Conference (pp. 1629–1638). https://doi.org/10.1109/EDUCON.2018.8363430
- Kitchenham, B., Pretorius, R., Budgen, D., Brereton, O. P., Turner, M., Niazi, M., & Linkman, S. (2010). Systematic literature reviews in software engineering–a tertiary study. Information and Software Technology, 52(8), 792-805. https://doi.org/10.1016/j.infsof.2010.03.006
- Lemay, D. J., & Doleck, T. (2022). Predicting completion of massive open online course (MOOC) assignments from video viewing behavior. Interactive Learning Environments, 30(10), 1782–1793. https://doi.org/10.1080/10494820.2020.1746673
- Li, L., Johnson, J., Aarhus, W., & Shah, D. (2022). Key factors in MOOC pedagogy based on NLP sentiment analysis of learner reviews: What makes a hit. Computers & Education, 176, 104354. https://doi.org/10.1016/j.compedu.2021.104354
- Littlejohn, A., & Hood, N. (2018). Reconceptualising learning in the digital age. Springer. https://doi.org/10.1007/978-981-10-8893-3_5
- Liu, S., Liu, S., Liu, Z., Peng, X., & Yang, Z. (2022). Automated detection of emotional and cognitive engagement in MOOC discussions to predict learning achievement. Computers & Education, 181, 104461. https://doi.org/10.1016/j.compedu.2022.104461
- Mrhar, K., Douimi, O., & Abik, M. (2021). A dropout predictor system in MOOCs based on neural networks. Journal of Automation, Mobile Robotics and Intelligent Systems, 14(4), 72–80. https://doi.org/10.14313/JAMRIS/4-2020/48
- Nanda, G., A. Douglas, K., R. Waller, D., E. Merzdorf, H., & Goldwasser, D. (2021). Analyzing large collections of open-ended feedback from MOOC learners using LDA topic modeling and qualitative analysis. IEEE Transactions on Learning Technologies, 14(2), 146–160. https://doi.org/10.1109/TLT.2021.3064798
- Nie, Y., Luo, H., & Sun, D. (2021). Design and validation of a diagnostic MOOC evaluation method combining AHP and text mining algorithms. Interactive Learning Environments, 29(2), 315–328. https://doi.org/10.1080/10494820.2020.1802298
- OpenupEd. (n.d.). OpenupEd quality label. Retrieved October 7, 2022 from https://www.openuped.eu/quality-label
- Ossiannilsson, E. (2020). Quality models for open, flexible, and online learning. Journal of Computer Science Research, 2(4). https://doi.org/10.30564/jcsr.v2i4.2357
- Osuna-Acedo, S. (2021). Gamification and MOOCs. In D. Frau-Meigs, S. Osuna-Acedo, & C. Marta-Lazo (Eds.), MOOCs and the participatory challenge: From revolution to reality (pp. 89–101). Springer International. https://doi.org/10.1007/978-3-030-67314-7_6
- Pilli, O., Admiraal, W., & Salli, A. (2018). MOOCs: Innovation or stagnation? Turkish Online Journal of Distance Education, 19(3), 169–181. https://doi.org/10.17718/tojde.445121
- Quality Assurance Agency for Higher Education. (n.d.). The quality code. Retrieved October 7, 2022 from https://www.qaa.ac.uk/quality-code
- Quality Matters. (n.d.). Quality matters. Retrieved November 22, 2022, from https://www.qualitymatters.org/
- Quiliano-Terreros, R., Ramirez-Hernandez, D., & Barniol, P. (2019). Systematic Mapping Study 2012-2017 : Quality and Effectiveness Measurement in MOOC. Turkish Online Journal of Distance Education, 223‑247. https://doi.org/10.17718/tojde.522719
- Quintana, R. M., & Tan, Y. (2019). Characterizing MOOC Pedagogies : Exploring Tools and Methods for Learning Designers and Researchers. Online Learning, 23(4), Article 4. https://doi.org/10.24059/olj.v23i4.2084
- Rahardja, U., Aini, Q., Graha, Y. I., & Tangkaw, M. R. (2019). Gamification framework design of management education and development in industrial revolution 4.0. Journal of Physics: Conference Series, 1364. https://doi.org/10.1088/1742-6596/1364/1/012035
- Ray, S. (2019, February). A quick review of machine learning algorithms. In 2019 International conference on machine learning, big data, cloud and parallel computing (COMITCon; pp. 35-39). IEEE. https://doi.org/10.1109/COMITCon.2019.8862451
- Rincón-Flores, E. G., Mena, J., & Montoya, M. S. R. (2020). Gamification: A new key for enhancing engagement in MOOCs on energy? International Journal on Interactive Design and Manufacturing, 14, 1379–1393. https://doi.org/10.1007/s12008-020-00701-9
- Romero-Rodriguez, L. M., Ramirez-Montoya, M. S., & Gonzalez, J. R. V. (2019). Gamification in MOOCs : Engagement application test in energy sustainability courses. IEEE Access, 32093‑32101. https://doi.org/10.1109/ACCESS.2019.2903230
- Sabjan, A., Abd Wahab, A., Ahmad, A., Ahmad, R., Hassan, S., & Wahid, J. (2021). MOOC quality design criteria for programming and non-programming students. Asian Journal of University Education, 16(4), 61. https://doi.org/10.24191/ajue.v16i4.11941
- Sezgin, S., & Yüzer, T. V. (2022). Analysing adaptive gamification design principles for online courses. Behaviour & Information Technology, 41(3), 485–501. https://doi.org/10.1080/0144929X.2020.1817559
- Shukor, N. A., & Abdullah, Z. (2019). Using learning analytics to improve MOOC instructional design. International Journal of Emerging Technologies in Learning, 14(24), 6. https://doi.org/10.3991/ijet.v14i24.12185
- Smyrnova-Trybulska, E., McKay, E., Morze, N., Yakovleva, O., Issa, T., Issa, T. (2019). Develop and implement MOOCs unit: A pedagogical instruction for academics, case study. In E. Smyrnova-Trybulska, P., Kommers, N. Morze, & J. Malach (Eds). Universities in the networked society. Critical studies of education (vol. 10, pp. 103-132). Springer. https://doi.org/10.1007/978-3-030-05026-9_7
- Stoica, A. S., Heras, S., Palanca, J., Julián, V., & Mihaescu, M. C. (2021). Classification of educational videos by using a semi-supervised learning method on transcripts and keywords. Neurocomputing, 456, 637–647. https://doi.org/10.1016/j.neucom.2020.11.075
- Stracke, C. M., Tan, E., Moreira Texeira, A., Texeira Pinto, M. do C., Vassiliadis, B., Kameas, A., & Sgouropoulou, C. (2018). Gap between MOOC designers’ and MOOC learners’ perspectives on interaction and experiences in MOOCs: Findings from the global MOOC quality survey. 2018 IEEE 18th International Conference on Advanced Learning Technologies (pp. 1–5). https://doi.org/10.1109/ICALT.2018.00007
- Stracke , C. M., & Trisolini, G. (2021). A systematic literature review on the quality of MOOCs. Sustainability, 13(11), 5817. https://doi.org/10.3390/su13115817
- Su, P. -Y., Guo, J. -H., & Shao, Q. -G. (2021). Construction of the quality evaluation index system of MOOC platforms based on the user perspective. Sustainability, 13(20), 11163. https://doi.org/10.3390/su132011163
- Sun, G., & Bin, S. (2018). Topic Interaction Model Based on Local Community Detection in MOOC Discussion Forums and its Teaching. Educational Sciences: Theory & Practice, 18(6).
- Sun, Y., Ni, L., Zhao, Y., Shen, X., & Wang, N. (2019). Understanding students’ engagement in MOOCs: An integration of self‐determination theory and theory of relationship quality. British Journal of Educational Technology, 50(6), 3156–3174. https://doi.org/10.1111/bjet.12724
- Suwita, J., Kosala, R., Ranti, B., & Supangkat, S. H. (2019). Factors considered for the success of the massive open online course in the era of smart education: Systematic literature review. 2019 International Conference on ICT for Smart Society (pp. 1–5). https://doi.org/10.1109/ICISS48059.2019.8969844
- Tjoa, A. M., & Poecze, F. (2020). Gamification as an enabler of quality distant education: The need for guiding ethical principles towards an education for a global society leaving no one behind. Proceedings of the 22nd International Conference on Information Integration and Web-Based Applications & Services (pp. 340–346). https://doi.org/10.1145/3428757.3429145
- van der Zee, T., Davis, D., Saab, N., Giesbers, B., Ginn, J., van der Sluis, F., Paas, F., & Admiraal, W. (2018). Evaluating retrieval practice in a MOOC: How writing and reading summaries of videos affects student learning. Proceedings of the 8th International Conference on Learning Analytics and Knowledge (pp. 216–225). https://doi.org/10.1145/3170358.3170382
- Wang, Q., Khan, M. S., & Khan, M. K. (2021). Predicting user perceived satisfaction and reuse intentions toward massive open online courses (MOOCs) in the COVID-19 pandemic: An application of the UTAUT model and quality factors. International Journal of Research in Business and Social Science, 10(2), 1–11. https://doi.org/10.20525/ijrbs.v10i2.1045
- Wang, W., Guo, L., He, L., & Wu, Y. J. (2019). Effects of social-interactive engagement on the dropout ratio in online learning: Insights from MOOC. Behaviour & Information Technology, 38(6), 621–636. https://doi.org/10.1080/0144929X.2018.1549595
- Wang, X., Lee, Y., Lin, L., Mi, Y., & Yang, T. (2021). Analyzing instructional design quality and students’ reviews of 18 courses out of the Class Central Top 20 MOOCs through systematic and sentiment analyses. The Internet and Higher Education, 50, 100810. https://doi.org/10.1016/j.iheduc.2021.100810
- Wohlin, C. (2014). Guidelines for snowballing in systematic literature studies and a replication in software engineering. Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering (pp. 1–10). https://doi.org/10.1145/2601248.2601268
- Wong, B. T. (2021). A survey on the pedagogical features of language massive open online courses. Asian Association of Open Universities Journal, 16(1), 116–128. https://doi.org/10.1108/AAOUJ-03-2021-0028
- Xiao, C., Qiu, H., & Cheng, S. M. (2019). Challenges and opportunities for effective assessments within a quality assurance framework for MOOCs. Journal of Hospitality, Leisure, Sport & Tourism Education, 24, 1-16. https://doi.org/10.1016/j.jhlste.2018.10.005
- Xing, W. (2018). Exploring the influences of MOOC design features on student performance and persistence. Exploring the influences of MOOC design features on student performance and persistence. Distance Education, 40(1), 98-113. https://doi.org/10.1080/01587919.2018.1553560
- Yuniwati, I., Yustita, A. D., Hardiyanti, S. A., & Suardinata, I. W. (2020). Development of assesment instruments to measure quality of MOOC-platform in engineering mathematics 1 course. Journal of Physics: Conference Series, 1567(2), 022102. https://doi.org/10.1088/1742-6596/1567/2/022102
- Zhou, Y., & Li, M. (2020). Online course quality evaluation based on BERT. 2020 International Conference on Communications, Information System and Computer Engineering (pp. 255–258). https://doi.org/10.1109/CISCE50729.2020.00057