Academic Journal
Evaluation of LLMs for Automated Dental-to-Medical Consult Letter Generation: Comparative Study.
| Τίτλος: | Evaluation of LLMs for Automated Dental-to-Medical Consult Letter Generation: Comparative Study. |
|---|---|
| Συγγραφείς: | Eom D; Interdisciplinary Program of Medical Informatics, College of Medicine, Seoul National University, Seoul, Republic of Korea., Koh K; Department of Conservative Dentistry, Eunpyeong St. Mary's Hospital, College of Medicine, The Catholic University of Korea, Seoul, Republic of Korea; Department of Dentistry, Graduate School of Chosun University, Gwangju, Republic of Korea., Chung H; Department of Transdisciplinary Medicine, Seoul National University Hospital, Seoul, Republic of Korea., Kim K; Department of Transdisciplinary Medicine, Seoul National University Hospital, Seoul, Republic of Korea; Center for Data Science, Healthcare AI Research Institute, Seoul National University Hospital, Seoul, Republic of Korea; Department of Medicine, College of Medicine, Seoul National University, Seoul, Republic of Korea., Lee SH; Division of Oral & Maxillofacial Surgery Department of Dentistry, Eunpyeong St. Mary's Hospital, College of Medicine, The Catholic University of Korea, Seoul, Republic of Korea. Electronic address: justina@catholic.ac.kr. |
| Πηγή: | International dental journal [Int Dent J] 2026 Oct; Vol. 76 (5), pp. 111088. Date of Electronic Publication: 2026 Sep 03. |
| Τύπος έκδοσης: | Journal Article; Comparative Study; Evaluation Study |
| Γλώσσα: | English |
| Στοιχεία περιοδικού: | Publisher: Elsevier Country of Publication: England NLM ID: 0374714 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 1875-595X (Electronic) Linking ISSN: 00206539 NLM ISO Abbreviation: Int Dent J Subsets: MEDLINE |
| Imprint Name(s): | Publication: 2021- : London : Elsevier Original Publication: London : FDI World Dental |
| Ιατρικοί όροι (MeSH): | Large Language Models* , Referral and Consultation* , Correspondence as Topic*, Humans ; Feasibility Studies |
| Περίληψη: | Introduction and Aims: For dental patients with systemic diseases, multidisciplinary care via referral letters is essential for clinical safety. This study evaluated the clinical feasibility and quality of dental-to-medical referral letters generated by Large Language Models (LLMs). Methods: Using 153 paired datasets, three LLMs - Gemini-2.5-Pro, GPT-5 and MedGemma - generated referral letters via a standardised prompting framework. Two dental clinicians performed a blinded assessment across nine sub-criteria covering content, style and correctness. GPT-5 also performed an automated evaluation. Statistical significance was analysed using Friedman and Wilcoxon signed-rank tests with Holm correction. Results: In the human clinical evaluation, Gemini-2.5-Pro achieved the highest overall performance (TOTAL mean SCORE: 2.9487 with a bootstrap 95% confidence interval [CI] of 2.9345-2.9606), significantly outperforming GPT-5 (2.8818 with 95% CI of 2.8613-2.9008) and MedGemma (2.8613 with 95% CI of 2.8338-2.8865) (P < .05). While GPT-5 showed high clarity in systemic problem descriptions, MedGemma excelled in structural adherence. The overall performance difference between GPT-5 and MedGemma was not statistically significant (P = .3519). Inter-rater reliability was strong in the prompt search dataset (QWK: 0.7175) but lower in the independent test dataset (QWK: 0.1505), reflecting the subjective nature of clinical document assessment; therefore, final human scores were reported as average ratings from two blinded clinicians with bootstrap confidence intervals. Notably, GPT-5's self-evaluation results deviated from human judgment, exhibiting potential model-based bias. Conclusion: This preliminary study suggests the potential of Gemini-2.5-Pro for automated consult letter generation, though the low test-set inter-rater reliability (QWK: 0.1505) highlights the subjective complexity of clinical text. To integrate these assistive prototypes into routine clinical application, more systematic studies are required. Clinical Relevance: Integrating LLMs offers a promising approach to assisting with dental-to-medical referrals, presenting a potential future benefit of reducing administrative workloads and enabling healthcare providers to focus on safe, empathetic patient care. (Copyright © 2026 The Authors. Published by Elsevier Inc. All rights reserved.) |
| Competing Interests: | Declaration of competing interest None disclosed. |
| Contributed Indexing: | Keywords: Artificial intelligence; Dental records; Large language models; Referral and consultation |
| Entry Date(s): | Date Created: 20260903 Date Completed: 20260916 Latest Revision: 20260916 |
| Update Code: | 20260916 |
| PubMed Central ID: | PMC13571579 |
| DOI: | 10.1016/j.identj.2026.111088 |
| PMID: | 42691762 |
| Βάση Δεδομένων: | MEDLINE |
| ISSN: | 1875-595X |
|---|---|
| DOI: | 10.1016/j.identj.2026.111088 |