(013) Leveraging Large Language Models (LLM) to Estimate the Active U.S. Otolaryngology Workforce

Introduction: Current estimates require an aggregation of significantly varying and often outdated national and private claims databases followed by intensive manual verification. This study sought to devise a novel LLM modality to identify actively practicing Otolaryngologists.

Methods: 50 Otolaryngologists were randomly selected from the most recent (2023) Medicare Physician and Other Practitioners’ dataset. Manual telephone calls rendered Gold Standard identification of active practice, practice location, and fellowship training status. Three prompts varying in specificity of instructions were entered into ChatGPT 5.2 (OpenAI). Using the Gold Standard classification as the reference, diagnostic performance metrics were calculated for the LLM’s determination of active practice status.

Results: Despite their active presence in the most recent publicly available Medicare database, only 39 (78%) of Otolaryngologists were identified as actively practicing while 8 (16%) were inactive. Of these eight, seven were retired and one deceased. The activity status of 3 (6%) was unclear due to inability to connect to their practice and were excluded in analysis of predictive characteristics. The LLM’s most specific prompt demonstrated a 100% sensitivity but a 50% specificity, struggling with correct identification of inactive practice. Overall, the model displayed a 91.5% accuracy, 90.7% positive predictive value, and 100% negative predictive value.

Conclusions: Given the high prevalence of actively practicing Otolaryngologists in this subset, the accuracy of this model remains high despite a 50% specificity rate. Large scale feasibility and incorporation into contemporary workforce estimate modalities will require expansion of this model across private claims datasets as well as larger workforce subsets.