(800) A Blinded Comparative Evaluation of Clinical and AI-Generated Responses to Otologic Patient Queries

Introduction: Artificial intelligence (AI) chatbots powered by large language models (LLMs) are increasingly used for patient education and health information delivery. Despite their growing visibility, concerns persist regarding the clinical appropriateness and reliability of AI-generated medical content. This study compares the quality, empathy, and readability of responses generated by leading LLMs with physician responses addressing common otologic symptoms in a public online forum.

Methods: This comparative study included a sample of 49 otology-related questions posted on Reddit r/AskDocs between January 2020 and June 2025 and were selected using search terms including “hearing loss,” “ear infection,” “tinnitus,” “ear pain,” and “vertigo.” Posts were retrieved using Reddit’s “Top” filter. Each question was answered by a verified doctor on Reddit and three AI LLMs (ChatGPT-4o, ClaudeAI, Google Gemini). Five medical professional evaluators independently scored responses. Outcome measures included response quality, empathy, readability scores, word count and Flesch–Kincaid grade level. Independent variables included responder type and LLM model.

Results: Common otologic concerns posed in patient questions were otalgia (38.7%), vertigo (28.6%), tinnitus (24.5%), hearing loss (22.4%), and aural fullness (20.4%). LLM responses were longer than physician responses (mean 145 vs 67 words; p < .05) and rated higher in quality (10.95 vs 9.58), empathy (7.26 vs 5.18), and readability (4.00 vs 3.73); (all p < .05). Evaluators correctly identified AI versus physician responses in 89.4% of cases with higher sensitivity for detecting physician responses (93.5%). By Flesch-Kincaid grade level, ChatGPT produced the most readable content (mean 7.25), while ClaudeAI responses were more complex (11.86; p < .05).

Conclusions: LLM responses received higher ratings in quality, empathy, and readability than those of physicians in response to a variety of otologic concerns. When appropriately implemented, such systems may enhance access to understandable otologic information and complement clinician-delivered care.