Résumés
Abstract
Background: Competency based residency programs depend on high quality feedback from the assessment of entrustable professional activities (EPA). The Quality of Assessment for Learning (QuAL) score is a tool developed to rate the quality of narrative comments in workplace-based assessments; it has validity evidence for scoring the quality of narrative feedback provided to emergency medicine residents, but it is unknown whether the QuAL score is reliable in the assessment of narrative feedback in other postgraduate programs.
Methods: Fifty sets of EPA narratives from a single academic year at our competency based medical education post-graduate anesthesia program were selected by stratified sampling within defined parameters [e.g. resident gender and stage of training, assessor gender, Competency By Design training level, and word count (≥17 or <17 words)]. Two competency committee members and two medical students rated the quality of narrative feedback using a utility score and QuAL score. We used Kendall’s tau-b co-efficient to compare the perceived utility of the written feedback to the quality assessed with the QuAL score. The authors used generalizability and decision studies to estimate the reliability and generalizability coefficients.
Results: Both the faculty’s utility scores and QuAL scores (r = 0.646, p < 0.001) and the trainees’ utility scores and QuAL scores (r = 0.667, p < 0.001) were moderately correlated. Results from the generalizability studies showed that utility scores were reliable with two raters for both faculty (Epsilon=0.87, Phi=0.86) and trainees (Epsilon=0.88, Phi=0.88).
Conclusions: The QuAL score is correlated with faculty- and trainee-rated utility of anesthesia EPA feedback. Both faculty and trainees can reliability apply the QuAL score to anesthesia EPA narrative feedback. This tool has the potential to be used for faculty development and program evaluation in Competency Based Medical Education. Other programs could consider replicating our study in their specialty.
Résumé
Contexte : La qualité de la rétroaction à la suite de l’évaluation d’activités professionnelles confiables (APC) est d’une importance capitale dans les programmes de résidence fondés sur les compétences. Le score QuAL (Quality of Assessment for Learning) est un outil développé pour évaluer la qualité de la rétroaction narrative dans les évaluations en milieu de travail. Sa validité a été démontrée dans le cas des commentaires narratifs fournis aux résidents en médecine d'urgence, mais sa fiabilité n’a pas été évaluée dans d'autres programmes de formation postdoctorale.
Méthodes : Cinquante ensembles de commentaires portant sur des APC d'une seule année universitaire dans notre programme postdoctoral en anesthésiologie – un programme fondé sur les compétences – ont été sélectionnés par échantillonnage stratifié selon des paramètres préétablis [par exemple, le sexe du résident et son niveau de formation, le sexe de l'évaluateur, le niveau de formation en Compétence par conception, et le nombre de mots (≥17 ou <17 mots)]. Deux membres du comité de compétence et deux étudiants en médecine ont évalué la qualité de la rétroaction narrative à l'aide d'un score d'utilité et d'un score QuAL. Nous avons utilisé le coefficient tau-b de Kendall pour comparer l'utilité perçue de la rétroaction écrite et sa qualité évaluée à l’aide du score QuAL. Les auteurs ont utilisé des études de généralisabilité et de décision pour estimer les coefficients de fiabilité et de généralisabilité.
Résultats : Les scores d'utilité et les scores QuAL des enseignants (r = 0,646, p < 0,001) et ceux des étudiants (r = 0,667, p < 0,001) étaient modérément corrélés. Les résultats des études de généralisabilité ont montré qu’avec deux évaluateurs les scores d'utilité étaient fiables tant pour les enseignants (Epsilon=0,87, Phi=0,86) que pour les étudiants (Epsilon=0,88, Phi=0,88).
Conclusions : Le score QuAL est en corrélation avec l'utilité de la rétroaction sur les APC en anesthésiologie évaluée par les enseignants et les étudiants. Les uns et les autres peuvent appliquer de manière fiable le score QuAL aux commentaires narratifs sur les APC en anesthésiologie. Cet outil pourrait être utilisé pour le perfectionnement professoral et l'évaluation des programmes dans le cadre d’une formation médicale fondée sur les compétences. D'autres programmes pourraient envisager de reproduire notre étude dans leur spécialité.
Veuillez télécharger l’article en PDF pour le lire.
Télécharger
Parties annexes
Bibliography
- Frank JR, Snell LS, Cate OT, et al. Competency-based medical education: theory to practice. Med Teach. 2010;32(8):638–45. https://doi.org/10.3109/0142159x.2010.501190
- Holmboe ES, Sherbino J, Long DM, Swing SR, Frank JR. The role of assessment in competency-based medical education. Med Teach. 2010 Aug 1;32(8):676–82. https://doi.org/10.3109/0142159x.2010.500704
- Caverzagie KJ, Nousiainen MT, Ferguson PC, et al. Overarching challenges to the implementation of competency-based medical education. Med Teach. 2017 Jun 3;39(6):588–93. https://doi.org/10.1080/0142159x.2017.1315075
- Jensen AR, Wright AS, Kim S, Horvath KD, Calhoun KE. Educational feedback in the operating room: a gap between resident and faculty perceptions. Amer J Surg. 2012 Aug 1;204(2):248–55. https://doi.org/10.1016/j.amjsurg.2011.08.019
- Upadhyaya S, Rashid M, Davila-Cervantes A, Oswald A. Exploring resident perceptions of initial competency based medical education implementation. Can Med Educ J. 2021 Apr;12(2):e42–56. https://doi.org/10.36834/cmej.70943
- Weller JM, Naik VN, Diego RJS. Systematic review and narrative synthesis of competency-based medical education in anaesthesia. Brit J Anaesthesia. 2020 Jun 1;124(6):748–60. https://doi.org/10.1016/j.bja.2019.10.025
- Yilmaz Y, Carey R, Chan T et al. Developing a dashboard for faculty development in competency-based training programs: a design-based research project. Can Med Ed J. 2021 Sep 15;12(4):48–64. https://doi.org/10.36834/cmej.72067
- Chan TM, Sebok-Syer SS, Sampson C, Monteiro S. The Quality of Assessment of Learning (Qual) Score: validity evidence for a scoring system aimed at rating short, workplace-based comments on trainee performance. Teach Learn Med. 2020 Jul;32(3):319–29. https://doi.org/10.1080/10401334.2019.1708365
- Woods RA, Singh S, Thoma B, et al. Validity evidence for the QuAL (Quality of Assessment for Learning) score: a quality metric for supervisor comments in Competency Based Medical Education. Can Med Ed J. 2022; 13(6); 19-35. https://doi.org/10.36834/cmej.74860
- St-Onge C, Young M, Eva KW, Hodges B. Validity: one word with a plurality of meanings. Adv in Health Sci Educ. 2017 Oct;22(4):853–67. https://doi.org/10.1007%2Fs40037-018-0433-x
- Cook DA, Brydges R, Ginsburg S, Hatala R. A contemporary approach to validity arguments: a practical guide to Kane’s framework. Med Educ. 2015 Jun;49(6):560–75. https://doi.org/10.1111/medu.12678
- Streiner DL, Norman GR. Health measurement scales: a practical guide to their development and use. 4th ed. Oxford: Oxford University Press; 2008. 452 p. Available from: https://oxford.universitypressscholarship.com/10.1093/acprof:oso/9780199231881.001.0001/acprof-9780199231881 [Accessed on Oct 3, 2021].
- Field A. Discovering statistics using SPSS. 3rd ed., SAGE Publications.2009
- Streiner DL. Starting at the beginning: an introduction to coefficient alpha and internal consistency. J Pers Assess. 2003 Feb;80(1):99-103. https://doi.org/10.1207/S15327752JPA8001_18
- Telio S, Regehr G, Ajjawi R. Feedback and the educational alliance: examining credibility judgements and their consequences. Med Educ. 2016 Sep;50(9):933–42. https://doi.org/10.1111/medu.13063
- Acai A, Li SA, Sherbino J, Chan TM. Attending emergency physicians’ perceptions of a programmatic workplace-based assessment system: the McMaster Modular Assessment Program (McMAP). Teach Learn Med. 2019 Aug 8;31(4):434-44. https://doi.org/10.1080/10401334.2019.1574581
- Chan T, Oswald A, Hauer KE, et al. Diagnosing conflict: conflicting data, interpersonal conflict, and conflicts of interest in clinical competency committees. Med Teach. 2021 Jul 3;43(7):765-73. https://doi.org/10.1080/0142159x.2021.1925101
- Dudek N, Dojeiji S. Twelve tips for completing quality in-training evaluation reports. Med Teach. 2014 Dec;36(12):1038–42. https://doi.org/10.3109/0142159x.2014.932897
- Gray JD. Global rating scales in residency education. Acad Med. 1996 Jan;71(1 Suppl):S55-63. https://doi.org/10.1097/00001888-199601000-00043
- Hatala R, Sawatsky AP, Dudek N, Ginsburg S, Cook DA. Using In-Training Evaluation Report (ITER) qualitative comments to assess medical students and residents: a systematic review. Acad Med. 2017 Jun 1;92(6):868–79. https://doi.org/10.1097/acm.0000000000001506
- Ginsburg S, van der Vleuten C, Eva KW, Lingard L. Hedging to save face: a linguistic analysis of written comments on in-training evaluation reports. Adv Health Sci Educ. 2016 Mar;21:175-88. https://doi.org/10.1007/s40037-021-00681-w
- Ginsburg S, Regehr G, Lingard L, Eva KW. Reading between the lines: faculty interpretations of narrative evaluation comments. Med Ed. 2015 Mar;49(3):296-306. https://doi.org/10.1111/medu.12637
- Ginsburg S, Eva K, Regehr G. Do in-training evaluation reports deserve their bad reputations? A study of the reliability and predictive ability of ITER scores and narrative comments. Acad Med. 2013 Oct 1;88(10):1539-44. https://doi.org/10.1097/acm.0b013e3182a36c3d
- Ginsburg S, van der Vleuten CP, Eva KW, Lingard L. Cracking the code: residents’ interpretations of written assessment comments. Med Ed. 2017 Apr;51(4):401-10. https://doi.org/10.1111/medu.13158
- Ginsburg S, Watling CJ, Schumacher DJ, Gingerich A, Hatala R. Numbers encapsulate, words elaborate: toward the best use of comments for assessment and feedback on entrustment ratings. Acad Med. 2021 Jul 1;96(7S):S81-6. https://doi.org/10.1097/acm.0000000000004089
- Li SA, Sherbino J, Chan TM. McMaster Modular Assessment Program (McMAP) through the years: residents' experience with an evolving feedback culture over a 3‐year period. AEM Educ Training. 2017 Jan;1(1):5-14. http://dx.doi.org/10.1002/aet2.10009
- Chan TM, Sherbino J, Mercuri M. Nuance and noise: lessons learned from longitudinal aggregated assessment data. J Grad Med Ed. 2017 Dec;9(6):724-9. https://doi.org/10.4300/jgme-d-17-00086.1
- Sebok-Syer SS, Klinger DA, Sherbino J, Chan TM. “It’s complicated”: understanding the relationships between checklists, rating scales, and written comments in workplace-based assessments. Acad Med. 2016 Nov 1;91(11):S10. https://doi.org/10.1097/ACM.000000000001373
- Yilmaz Y, Carey R, Chan TM et al. Developing a dashboard for faculty development in competency-based training programs: a design-based research project. Can Med Ed J. 2021 Oct 20;12(4):48-64. https://doi.org/10.36834/cmej.72067
- Miller J, Katz D. Gender differences in perception of workplace experience among anesthesiology residents. JEPM. 2018 Jan;20(1).
- Pearce G, Sidhu N, Cavadino A, Shrivathsa A, Seglenieks R. Gender effects in anaesthesia training in Australia and New Zealand. Brit j anaesthesia. 2020 Mar 1;124(3):e70-6. https://doi.org/10.1016/j.bja.2019.12.020
- Dayal A, O’Connor DM, Qadri U, Arora VM. Comparison of male vs female resident milestone evaluations by faculty during emergency medicine residency training. JAMA intern med. 2017 May 1;177(5):651-7. https://doi.org/10.1001/jamainternmed.2016.9616
- Mamtani M, Shofer F, Scott K, et al. Gender differences in emergency medicine attending physician comments to residents: a qualitative analysis. JAMA Network Open. 2022 Nov 1;5(11):e2243134-. https://doi.org/10.1001/jamanetworkopen.2022.43134
- Menchetti I, Eagles D, Ghanem D, Leppard J, Fournier K, Cheung WJ. Gender differences in emergency medicine resident assessment: a scoping review. AEM Educ Training. 2022 Oct;6(5):e10808. https://doi.org/10.1007/s11606-019-04884-0
- Santen SA, Yamazaki K, Holmboe ES, Yarris LM, Hamstra SJ. Comparison of male and female resident milestone assessments during emergency medicine residency training: a national study. Acad Med. 2020 Feb;95(2):263. https://doi.org/10.1097/acm.0000000000002988