Résumés
Abstract
Introduction: Assessment can positively influence learning, however designing effective assessment-for-learning interventions has proved challenging. We implemented a mandatory assessment-for-learning system comprising a workplace-based assessment of non-medical expert competencies and a progress test in undergraduate medical education and evaluated its impact.
Methods: We conducted semi-structured interviews with year-3 and 4 medical students at McGill University to explore how the assessment system had influenced their learning in year 3. We conducted theory-informed thematic analysis of the data.
Results: Eleven students participated, revealing that the assessment influenced learning through several mechanisms. Some required little student engagement (i.e., feed-up, test-enhanced learning, looking things up after an exam). Others required substantial engagement (e.g., studying for tests, selecting raters for quality feedback, using feedback). Student engagement was moderated by the perceived credibility of the system and of the costs and benefits of engagement. Credibility was shaped by students’ goals-in-context: becoming a good doctor, contributing to the healthcare team, succeeding in assessments.
Discussion: Our assessment system failed to engage students enough to leverage its full potential. We discuss the inherent flaws and external factors that hindered student engagement. Assessment designers should leverage easy-to-control mechanisms to support assessment-for-learning and anticipate significant collaborative work to modify learning cultures.
Résumé
Introduction : L’évaluation peut influencer positivement l’apprentissage mais la conception de dispositifs d’évaluation pour l’apprentissage efficaces s’avère difficile. Nous avons implanté en formation prédoctorale un système obligatoire d’évaluation pour l’apprentissage comprenant une évaluation en milieu clinique des compétences transversales et un test de rendement progressif, puis évalué ses effets.
Méthodes : Nous avons mené des entretiens semi-dirigés avec des étudiants en troisième et quatrième années de médecine à l’Université McGill pour explorer la manière dont le système d’évaluation avait influencé leur apprentissage au cours de la troisième année. Nous avons effectué une analyse thématique, informée par la théorie, des données.
Résultats : Onze étudiants ont participé. Les résultats indiquent que l’évaluation a influencé leur apprentissage par le biais de plusieurs mécanismes différents. Certains d’entre eux nécessitaient une implication faible de la part de l’étudiant, comme l’identification des objectifs à atteindre (feed-up), l’apprentissage amélioré par les tests, la recherche d’informations après un examen. D’autres exigeaient une implication importante (par exemple, étudier pour les tests, sélectionner les évaluateurs pour obtenir une rétroaction de qualité, mettre à profit la rétroaction). L’implication des étudiants était modulée par leur perception des avantages et des inconvénients de s’impliquer, et de la crédibilité du système. Cette dernière était influencée par les objectifs-en-contexte des étudiants: devenir un bon médecin, contribuer à l’équipe soignante, réussir les épreuves d’évaluation.
Discussion : Notre système d’évaluation n’a pas réussi à impliquer suffisamment les étudiants que pour réaliser son potentiel. Nous abordons les défauts inhérents au système ainsi que les facteurs externes qui ont entravé l’implication des apprenants. Pour implanter efficacement un dispositif d’évaluation pour l’apprentissage, les concepteurs d’évaluations devraient optimiser les mécanismes qui sont faciles à contrôler et être prêts à s’investir dans un important travail de collaboration pour changer les cultures d’apprentissage.
Veuillez télécharger l’article en PDF pour le lire.
Télécharger
Parties annexes
Bibliography
- Kreiter C, Green J, Lenoch S, Saiki T. The overall impact of testing on medical student learning: quantitative estimation of consequential validity. Adv Health Sci Educ. 2013;18(4):835-844. https://doi.org/10.1007/s10459-012-9395-7
- Norcini J, Anderson MB, Bollela V, et al. 2018 Consensus framework for good assessment. Med Teach. 2018;40(11):1102-1109. https://doi.org/10.1080/0142159X.2018.1500016
- Cilliers FJ, Schuwirth LW, Herman N, Adendorff HJ, van der Vleuten CP. A model of the pre-assessment learning effects of summative assessment in medical education. Adv Health Sci Educ. 2012;17(1):39-53. https://doi.org/10.1007/s10459-011-9292-5
- Cilliers F, Schuwirth L, Adendorff H, Herman N, Van der Vleuten C. The mechanism of impact of summative assessment on medical students’ learning. Adv Health Sci Educ. 2010;15(5):695-715. https://doi.org/10.1007/s10459-010-9232-9
- Larsen DP, Butler AC, Roediger HL. Test-enhanced learning in medical education. Med Educ. 2008;42(10):959-966. https://doi.org/10.1111/j.1365-2923.2008.03124.x
- Larsen DP, Butler AC, Roediger Iii HL. Comparative effects of test-enhanced learning and self-explanation on long-term retention. Med Educ. 2013;47(7):674-682. https://doi.org/10.1111/medu.12141
- Hattie J, Timperley H. The power of feedback. Rev Educ Res. 2007;77(1):81-112. https://doi.org/10.3102/003465430298487
- Schuwirth LWT, Van der Vleuten CPM. Programmatic assessment: From assessment of learning to assessment for learning. Med Teach. 2011;33(6):478-485. https://doi.org/10.3109/0142159X.2011.565828
- Schuwirth L. Making the horse drink: use of mini-CEX in an assessment for learning view. Adv Health Sci Educ. 2013;18:1–4. https://doi.org/10.1007/s10459-012-9423-7
- Holmboe ES, Sherbino J, Long DM, Swing SR, Frank JR. The role of assessment in competency-based medical education. Med Teach. 2010;32(8):676-682. https://doi.org/10.3109/0142159X.2010.500704
- Harrison CJ, Könings KD, Molyneux A, Schuwirth LWT, Wass V, Van der Vleuten CPM. Web-based feedback after summative assessment: how do students engage? Med Educ. 2013;47(7):734-744. https://doi.org/10.1111/medu.12209
- Harrison C, Könings K, Schuwirth L, Wass V, van der Vleuten C. Barriers to the uptake and use of feedback in the context of summative assessment. Adv Health Sci Educ. 2015;20(1):229-245. https://doi.org/10.1007/s10459-014-9524-6
- Driessen EW, van Tartwijk J, Govaerts M, Teunissen P, van der Vleuten CP. The use of programmatic assessment in the clinical workplace: a Maastricht case report. Med Teach. 2012;34(3):226-231. https://doi.org/10.3109/0142159x.2012.652242
- Bok H, Teunissen P, Favier R, et al. Programmatic assessment of competency-based workplace learning: when theory meets practice. BMC Med Educ. 2013;13(1):123. https://doi.org/10.1186/1472-6920-13-123
- Heeneman S, Oudkerk Pool A, Schuwirth LW, van der Vleuten CP, Driessen EW. The impact of programmatic assessment on student learning: theory versus practice. Med Educ. 2015;49(5):487-498. https://doi.org/10.1111/medu.12645
- Bindal T, Wall D, Goodyear HM. Trainee doctors’ views on workplace-based assessments: Are they just a tick box exercise? Med Teach. 2011;33(11):919-927. https://doi.org/10.3109/0142159X.2011.558140
- Rees CE, Cleland JA, Dennis A, Kelly N, Mattick K, Monrouxe LV. Supervised learning events in the Foundation Programme: a UK-wide narrative interview study. BMJ Open. 2014;4(10). https://doi.org/10.1136/bmjopen-2014-005980
- Barrett A, Galvin R, Scherpbier AJJA, Teunissen PW, O’Shaughnessy A, Horgan M. Is the learning value of workplace-based assessment being realised? A qualitative study of trainer and trainee perceptions and experiences. Postgrad Med J. 2017;93(1097):138-142. https://doi.org/10.1136/postgradmedj-2015-133917
- Wiliam D, Thompson M. Integrating assessment with learning: What will it take to make it work? In: The Future of Assessment: Shaping Teaching and Learning. 1st ed. Lawrence Erlbaum Associates; 2008.
- Freeman A, Van Der Vleuten C, Nouns Z, Ricketts C. Progress testing internationally. Med Teach. 2010;32(6):451-455. https://doi.org/10.3109/0142159X.2010.485231
- van der Vleuten CPM, Verwijnen GM, Wijnen WHFW. Fifteen years of experience with progress testing in a problem-based learning curriculum. Med Teach. 1996;18(2):103-109. https://doi.org/10.3109/01421599609034142
- Yielder J, Wearn A, Chen Y, et al. A qualitative exploration of student perceptions of the impact of progress tests on learning and emotional wellbeing. BMC Med Educ. 2017;17(1):148. https://doi.org/10.1186/s12909-017-0984-2
- Cruess R, McIlroy JH, Cruess S, Ginsburg S, Steinert Y. The Professionalism Mini-Evaluation Exercise: A Preliminary Investigation. Acad Med. 2006;81(10):S74-S78. https://doi.org/10.1097/00001888-200610001-00019
- Sargeant J, Mann K, Van der Vleuten C, Metsemakers J. Reflection: a link between receiving and using assessment feedback. Adv Health Sci Educ. 2009;14(3):399-410. https://doi.org/0.1007/s10459-008-9124-4
- Saldaña J. The coding manual for qualitative researchers. 3rd ed. Sage; 2016.
- Kiger ME, Varpio L. Thematic analysis of qualitative data: AMEE Guide No. 131. Med Teach. Published online May 1, 2020:1-9. https://doi.org/10.1080/0142159X.2020.1755030
- Dory V, Gomez-Garibello C, Cruess R, Cruess S, Cummings BA, Young M. The challenges of detecting progress in generic competencies in the clinical setting. Med Educ. 2018;52(12):1259-1270. https://doi.org/10.1111/medu.13749
- Ricci M, St-Onge C, Xiao J, Young M. Students as stakeholders in assessment: how students perceive the value of an assessment. Perspect Med Educ. 2018;7(6):352-361. https://doi.org/10.1007/s40037-018-0480-3
- Boudreau JD, Cassell E, Fuks A. A healing curriculum. Med Educ. 2007;41(12):1193-1201. https://doi.org/10.1111/j.1365-2923.2007.02905.x
- Young M, Ryan A. Postpositivism in health professions education scholarship. Acad Med. 2020;95(5):695-699. https://doi.org/10.1097/ACM.0000000000003089
- Molloy E, Ajjawi R, Bearman M, Noble C, Rudland J, Ryan A. Challenging feedback myths: values, learner involvement and promoting effects beyond the immediate task. Med Educ. 2020;54(1):33-39. https://doi.org/10.1111/medu.13802
- Ajjawi R, Regehr G. When I say ... feedback. Med Educ. 2019;53(7):652-654. https://doi.org/10.1111/medu.13746
- Tavares W, Eppich W, Cheng A, et al. Learning conversations: an analysis of their theoretical roots and their manifestations of feedback and debriefing in medical education. Acad Med. 2020;95(7):1020-1025. https://doi.org/10.1097/acm.0000000000002932
- Watling CJ, Kenyon CF, Zibrowski EM, et al. Rules of Engagement: residents’ perceptions of the in-training evaluation process. Acad Med. 2008;83(10):S97-S100. https://doi.org/10.1097/ACM.0b013e318183e78c
- Scarff CE, Bearman M, Chiavaroli N, Trumble S. Trainees’ perspectives of assessment messages: a narrative systematic review. Med Educ. 2019;53(3):221-233. https://doi.org/10.1111/medu.13775
- Eva K, Armson H, Holmboe E, et al. Factors influencing responsiveness to feedback: on the interplay between fear, confidence, and reasoning processes. Adv Health Sci Educ. 2012;17(1):15-26. https://doi.org/10.1007/s10459-011-9290-7
- Gaunt A, Patel A, Rusius V, Royle TJ, Markham DH, Pawlikowska T. ‘Playing the game’: how do surgical trainees seek feedback using workplace-based assessment? Med Educ. 2017;51(9):953-962. https://doi.org/10.1111/medu.13380
- Bowen L, Marshall M, Murdoch-Eaton D. Medical student perceptions of feedback and feedback behaviors within the context of the “educational alliance.” Acad Med. 2017;92(9):1303-1312. https://doi.org/10.1097/acm.0000000000001632
- Telio S, Regehr G, Ajjawi R. Feedback and the educational alliance: examining credibility judgements and their consequences. Med Educ. 2016;50(9):933-942. https://doi.org/10.1111/medu.13063
- Moroz A, Horlick M, Mandalaywala N, T Stern D. Faculty feedback that begins with resident self-assessment: motivation is the key to success. Med Educ. 2018;52(3):314-323. https://doi.org/10.1111/medu.13484
- Delva D, Sargeant J, Miller S, et al. Encouraging residents to seek feedback. Med Teach. 2013;35(12):e1625-31. https://doi.org/10.3109/0142159X.2013.806791
- Bok HGJ, Teunissen PW, Spruijt A, et al. Clarifying students’ feedback-seeking behaviour in clinical clerkships. Med Educ. 2013;47(3):282-291. https://doi.org/10.1111/medu.12054
- Bing-You R, Hayes V, Palka T, Ford M, Trowbridge R. The art (and artifice) of seeking feedback: clerkship students’ approaches to asking for feedback. Acad Med. 2018;93(8):1218-1226. https://doi.org/10.1097/acm.0000000000002256
- Watling C, LaDonna KA, Lingard L, Voyer S, Hatala R. ‘Sometimes the work just needs to be done’: socio-cultural influences on direct observation in medical training. Med Educ. 2016;50(10):1054-1064. https://doi.org/10.1111/medu.13062
- Ramani S, Könings KD, Mann KV, Pisarski EE, van der Vleuten CPM. About politeness, face, and feedback: exploring resident and faculty perceptions of how institutional feedback culture influences feedback practices. Acad Med. 2018;93(9):1348-1358. https://doi.org/10.1097/acm.0000000000002193
- Voyer S, Cuncic C, Butler DL, MacNeil K, Watling C, Hatala R. Investigating conditions for meaningful feedback in the context of an evidence-based feedback programme. Med Educ. 2016;50(9):943-954. https://doi.org/10.1111/medu.13067
- Molloy E, Boud D, Henderson M. Developing a learning-centred framework for feedback literacy. Assess Eval High Educ. 2020;45(4):527-540. https://doi.org/10.1080/02602938.2019.1667955
- Winstone NE, Nash RA, Parker M, Rowntree J. Supporting learners’ agentic engagement with feedback: a systematic review and a taxonomy of recipience processes. Educ Psychol. 2017;52(1):17-37. https://doi.org/10.1080/00461520.2016.1207538
- Royal College of Physicians and Surgeons of Canada. CanMEDS 2015: physician competency framework. (Frank JR, Snell L, Sherbino J, eds.).; 2015.