(1301) Assessing Artificial Intelligence Chatbot Responses for Patient Education on Maxillomandibular Advancement for Obstructive Sleep Apnea

Introduction: Artificial intelligence (AI) chatbots are often used to obtain healthcare information. This study examined generative AI responses to common questions about maxillomandibular advancement (MMA) for quality, accuracy, and readability to determine the reliability of these tools as patient education resources.

Methods: Four AI chatbots (ChatGPT, Perplexity, Copilot, Gemini) were asked an identical set of 20 questions about MMA (n=80 responses). Responses were graded by 4 fellowship-trained sleep surgeons using 2 separate 5-point Likert scales for accuracy and quality. A representative response score was determined using the median surgeon grade. Proportions of responses receiving a score of at least 4 for each Likert scale (mostly accurate, good quality) were compared by chatbot using Fisher’s exact tests. Readability metrics including Flesch-Kincaid (FK) grade level, Gunning Fog Index (GFI), and word count were compared by chatbot using Friedman tests with multiple comparisons. Statistical significance was defined as p< 0.05.

Results: Following grading, accuracy scores were at least 4 for most ChatGPT (80%), Perplexity (65%), Copilot (70%), and Gemini (85%) responses (p=0.51). Quality scores were at least 4 for most ChatGPT (95%), Perplexity (65%), Copilot (75%), and Gemini (85%) responses (p=0.09). Mean FK grade level was lowest for ChatGPT (9.44) responses versus Perplexity (12.89, p< 0.0001), Copilot (10.61, p=0.018), and Gemini (11.52, p< 0.0001). Mean GFI score was lowest for ChatGPT (10.35) responses versus Perplexity (14.92, p< 0.0001), Copilot (11.54, p=0.11), and Gemini (12.95, p< 0.0001). Mean word count was shortest for Perplexity (194.70) versus ChatGPT (346.3, p=0.0002), Copilot (300.1, p=0.06) and Gemini (426.8, p< 0.0001).

Conclusions: In response MMA questions, AI chatbots demonstrated good response quality in 65-95% and high response accuracy in 65-85% of instances. Readability differed significantly across interfaces, in favor of ChatGPT. Knowledge obtained from generative AI should remain supplementary to information provided by trusted healthcare professionals.