The Role of Artificial Intelligence in Evaluating Otolaryngology Residency Applicants

Saturday, October 17, 2026

9:54 AM - 10:00 AM PT

Location: 406A

Presenting Author(s)

NS

Nadia L. Samaha, MD

Resident Physician
University of Pennsylvania School of Medicine
Philadelphia, Pennsylvania, United States

Co-Author(s)

SS

Sajad Sotudeh, PhD

Computer Scientist
Georgetown University Department of Computer Science, United States
Jenny X. Chen, MD, EdM

Assistant Professor
Johns Hopkins School of Medicine
Baltimore, MD, United States
Leila J. Mady, MD, PhD, MPH

Associate Professor
Johns Hopkins University School of Medicine
Baltimore, MD, United States
Tiffany N. Chao, MD MSEd

Assistant Professor, Residency Program Director
University of Pennsylvania
Philadelphia, PA, United States
SM

Sonya M. Malekzadeh, MD, FACS

Professor
Georgetown University School of Medicine
Washington, DC, United States

Senior Author(s)

MH

Michael Hoa, MD

Professor
Georgetown University Medical Center
Potomac, MD, United States

Disclosure(s):

Tiffany N. Chao, MD MSEd:

faculty for this accredited education activity has no relationship(s) with ineligible companies to disclose.

Michael Hoa, MD:

faculty for this accredited education activity has no relationship(s) with ineligible companies to disclose.

Introduction: Residency programs face an overwhelming volume of applications, often leading to using inequitable screening metrics like licensing exam scores. Large language models (LLM) offer a solution by synthesizing complex information into structured summaries. This study evaluates the performance of a LLM in generating summaries of otolaryngology applications to support holistic initial screening.

Methods: Ten applications from the 2024–2025 otolaryngology residency cycle were summarized using Phi-3 vision, Llama 3.3, and GPT4-o. A prompt was iteratively engineered to produce 2-page summaries. Six faculty reviewers evaluated the summaries via survey, rating comprehensiveness, accuracy, errors, holistic quality, bias, readability, review time, and utility. Descriptive statistics were calculated on SPSS.

Results: Reviewers rated 71% of summaries as extremely or very comprehensive (mean score, 2.33 ± 0.75, 5-point Likert scale; lower scores more favorable). Factual accuracy was rated as completely or mostly accurate in 71% of evaluations (2.19 ± 0.71). Overall, 58% of summaries were rated as extremely or very holistic (2.55 ± 0.83), 54% as extremely or very unbiased (2.50 ± 0.77), and 77% as excellent or good in clarity/readability (1.91 ± 0.82). Reviewers reported spending a mean of 15.8 ± 14.5 minutes reviewing a full application versus 4.7 ± 3.0 minutes reviewing a summary. Summaries were rated as very or extremely useful by 83%, both for initial application screening (2.00 ± 0.58) and for interview preparation (1.83 ± 0.69). Inter-rater reliability ranged from κ = –0.60 to 1.00, with moderate to high agreement overall. Qualitative feedback highlighted improved efficiency with limitations in capturing nuance from letters of recommendation, personal statements, and occasional publication summary inaccuracies.

Conclusions: Faculty perceived LLM-generated summaries as comprehensive, holistic, efficient, and useful with limitations in nuance and occasional factual omissions. These findings support the use of LLM summarization as an adjunct—not a replacement—to human residency application review.