(021) Using OpenEvidence to Generate Multiple-Choice Examinations from Learning Opportunity Topics

Introduction: High-quality multiple-choice examinations (MCEs) are essential for resident assessment in otolaryngology, but developing psychometrically sound questions is time-intensive. This study evaluates the performance characteristics of an artificial intelligence (AI)-generated MCE aligned with clinical topics in areas of historic underperformance on previous assessments.

Methods: A 50-question MCE was created using OpenEvidence based on 34 predefined learning opportunity topics. Twenty-one otolaryngology residents (PGY1–5) completed the examination in January 2026. Descriptive statistics summarized performance. Association between PGY level and examination score was assessed using Spearman correlation. Internal consistency was measured with Cronbach’s alpha. Item difficulty (proportion correct) and discrimination (point-biserial correlation) were calculated. Answers and explanations were cross-checked using ChatGPT and Microsoft Copilot. The mean number of cited resources per topic was calculated.

Results: The mean examination score was 73.3 ± 7.4 (median 76; range 60–90), with no significant correlation between score and PGY level (ρ = –0.11, p = 0.64). Internal consistency was excellent (α=0.996). Mean item difficulty was 0.73 (range 0–1; median 0.82): 5 questions (10%) were hard (answered correctly by < 30% of residents), 13 questions (26%) were appropriately challenging (30–70% correct), and 32 questions (64%) were easy (>70% correct). Most items differentiated between higher- and lower-performing residents (mean discrimination 0.28; range –0.34–1.14). Mean resources per topic was 5.9.

Conclusions: AI-assisted MCE generation using OpenEvidence produced a reliable, content-aligned assessment. These findings support its feasibility as a scalable adjunct for resident education and formative assessment in otolaryngology, with faculty oversight required to ensure content validity.