(104) Assessing the Clinical Promise of AI Translation Tools: Otolaryngologist Perspectives and Language-Specific Performance

Introduction: AI translation has shown to cost-effectively improve communication for patients with limited English proficiency (LEP). We aim to understand otolaryngologist perspectives on opportunities and barriers for AI use to translate patient instructions and quantitatively evaluate AI translation quality across languages.

Methods: Otolaryngologists at an urban safety net teaching hospital were recruited via networking and snowball sampling. For aim 1, semi-structured Zoom interviews were analyzed by two coders using inductive and deductive thematic analysis (NVivo 1.7.2) until thematic saturation was achieved. For aim 2, ChatGPT-5 translations of allergy instructions in Haitian Creole (HC) and Spanish were compared with professional translations using BLEU and chrF metrics.

Results: For aim 1, thematic analysis revealed 4 themes: 1) Dissatisfaction with existing translation systems; 2) Recognition of the value of AI translation for patient education materials and custom discharge instructions; 3) Identification of systemic barriers: HIPAA, administrative constraints, variable attitudes toward AI, concerns regarding accuracy and cultural nuance; 4) Trainees reported greater openness to innovation, whereas attendings expressed greater acceptance of status quo.

For aim 2, scores for Spanish vs HC (BLEU 0.54 vs 0.40; chrF 0.77 vs 0.70) indicate consistent phrasing and alignment with professional standards in Spanish, while HC translations showed greater variability (BLEU) despite preserved semantic similarity across both languages (chrF).

Conclusions: Advances in AI have narrowed the gap between AI-generated and professional translations, particularly in more widely spoken languages, highlighting the potential to improve communication and health literacy for patients with LEP. Providers emphasize the importance of institutional support and safeguards for safe integration. These findings inform an ongoing prospective study of allergy-instruction translation combining provider perspectives, accuracy metrics, and patient feedback to guide evidence-based implementation.