(103) Are AI-Generated Otolaryngology Patient Education Materials Clinically Ready? A Review of Readability, Quality, and Safety

Introduction: Concerns persist regarding the usability and reliability of patient education materials (PEMs) produced by artificial intelligence. Our systematic review evaluates how large language models (LLMs) perform in generating PEMs for otolaryngology patients, focusing on readability, quality, and safety.

Methods: A systematic review was conducted on February 19, 2025 using MEDLINE, EMBASE, CENTRAL, CINAHL, Web of Science, and SCOPUS databases. PRISMA guidelines were adhered to. Cross-sectional studies evaluating LLM-generated PEMs or responses to hypothetical patient questions were included. Primary outcomes with recommended ranges were Flesh Reading Ease (FRE, ≥ 70), Flesh-Kincaid Grade Level (FKGL, ≤ 6.0), and SMOG (≤ 8) scores for readability, and the DISCERN (≥ 51) score for quality.

Results: Of 225 screened studies, 34 were included. All studies used ChatGPT (v3.5 or v4 when specified). The most common subspecialties included rhinology (29.4%), otology (26.5%), and head and neck oncology (23.5%). Median reported mean scores for FRE, FKGL, SMOG, and DISCERN were 40.7 (IQR 14.6), 12.1 (IQR 4.05), 11.3 (IQR 2.0), and 34.9 (IQR 24.9). When LLMs were asked to provide simpler answers, FRE scores increased by 45% while FKGL scores decreased by 31%. Twenty studies (58.9%) reported inaccurate, guideline-discordant, or potentially harmful responses, while six (17.6%) reported accuracy >90%.

Conclusions: LLM-generated PEMs were difficult to read and written at a 12th grade level, well above the recommended 6th grade level, raising concerns about accessibility. Based on DISCERN scores, PEM quality was poor. The high rate of errors in generated responses suggests that without oversight, LLM-generated materials are currently unsuitable for unsupervised patient education. Clinician review and quality checks are essential for effective and safe use.