(009) Friend or Foe: Are Large Language Models a Safe Substitute for Human Translation?

Introduction: Non-English speaking patients encounter barriers to care due to lack of access to medical information and education in their own language. The aim of this study is to determine whether large language models (LLM) can assist medical professionals to provide high-quality Spanish translations of preoperative information for common otolaryngology surgical procedures.

Methods: Cross-sectional study comparing translations performed by LLMs to those by medical professionals. Preoperative handouts were generated by Open Evidence, following American Academy of Otolaryngology guidelines. The LLMs used for translation were public versions of Open AI, Claude, Mistral, Llama, and Gemini. Translation quality was assessed manually using the Multidimensional Quality Metrics (MQM) and machine assisted using the Bilingual Evaluation Understudy (BLEU) score.

Results: Five handouts were generated by Open Evidence. For each source, three human professional translations, and five LLM translations were performed and compared. The average MQM score for human translations was 2.6, whereas the average for LLM was 18.0. These results show a disparity in quality between LLM and medical professional translations, with a mean difference of 15.40 (95% CI, 6.9-23.9). BLEU score for all translations averaged 0.040 ± 0.037 across all LLMs. Individual BLEU scores for each LLM were Claude: 0.031 - 0.035, Gemini: 0.065 - 0.091, Llama: 0.028 - 0.049, O4: 0.022 - 0.037, Mistral: 0.028 - 0.059. There was no statistically significant difference between the LLM performance based on BLEU scores (p = 0.06).

Conclusions: This study revealed that LLM translations varied in quality compared to those produced by medical professionals. However, both averages were considered satisfactory for MQM scoring. BLEU score analysis suggests that different LLMs produce translations that vary in lexical similarity than human translations. These results indicate that LLM exhibited a level of proficiency that is advantageous in overcoming language barriers.