Academic Journal
Προβολή σε EDS

Evaluation of LLMs for Automated Dental-to-Medical Consult Letter Generation: Comparative Study.

Λεπτομέρειες βιβλιογραφικής εγγραφής
Τίτλος: Evaluation of LLMs for Automated Dental-to-Medical Consult Letter Generation: Comparative Study.
Συγγραφείς: Eom D; Interdisciplinary Program of Medical Informatics, College of Medicine, Seoul National University, Seoul, Republic of Korea., Koh K; Department of Conservative Dentistry, Eunpyeong St. Mary's Hospital, College of Medicine, The Catholic University of Korea, Seoul, Republic of Korea; Department of Dentistry, Graduate School of Chosun University, Gwangju, Republic of Korea., Chung H; Department of Transdisciplinary Medicine, Seoul National University Hospital, Seoul, Republic of Korea., Kim K; Department of Transdisciplinary Medicine, Seoul National University Hospital, Seoul, Republic of Korea; Center for Data Science, Healthcare AI Research Institute, Seoul National University Hospital, Seoul, Republic of Korea; Department of Medicine, College of Medicine, Seoul National University, Seoul, Republic of Korea., Lee SH; Division of Oral & Maxillofacial Surgery Department of Dentistry, Eunpyeong St. Mary's Hospital, College of Medicine, The Catholic University of Korea, Seoul, Republic of Korea. Electronic address: justina@catholic.ac.kr.
Πηγή: International dental journal [Int Dent J] 2026 Oct; Vol. 76 (5), pp. 111088. Date of Electronic Publication: 2026 Sep 03.
Τύπος έκδοσης: Journal Article; Comparative Study; Evaluation Study
Γλώσσα: English
Στοιχεία περιοδικού: Publisher: Elsevier Country of Publication: England NLM ID: 0374714 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 1875-595X (Electronic) Linking ISSN: 00206539 NLM ISO Abbreviation: Int Dent J Subsets: MEDLINE
Imprint Name(s): Publication: 2021- : London : Elsevier
Original Publication: London : FDI World Dental
Ιατρικοί όροι (MeSH): Large Language Models* , Referral and Consultation* , Correspondence as Topic*, Humans ; Feasibility Studies
Περίληψη: Introduction and Aims: For dental patients with systemic diseases, multidisciplinary care via referral letters is essential for clinical safety. This study evaluated the clinical feasibility and quality of dental-to-medical referral letters generated by Large Language Models (LLMs).
Methods: Using 153 paired datasets, three LLMs - Gemini-2.5-Pro, GPT-5 and MedGemma - generated referral letters via a standardised prompting framework. Two dental clinicians performed a blinded assessment across nine sub-criteria covering content, style and correctness. GPT-5 also performed an automated evaluation. Statistical significance was analysed using Friedman and Wilcoxon signed-rank tests with Holm correction.
Results: In the human clinical evaluation, Gemini-2.5-Pro achieved the highest overall performance (TOTAL mean SCORE: 2.9487 with a bootstrap 95% confidence interval [CI] of 2.9345-2.9606), significantly outperforming GPT-5 (2.8818 with 95% CI of 2.8613-2.9008) and MedGemma (2.8613 with 95% CI of 2.8338-2.8865) (P < .05). While GPT-5 showed high clarity in systemic problem descriptions, MedGemma excelled in structural adherence. The overall performance difference between GPT-5 and MedGemma was not statistically significant (P = .3519). Inter-rater reliability was strong in the prompt search dataset (QWK: 0.7175) but lower in the independent test dataset (QWK: 0.1505), reflecting the subjective nature of clinical document assessment; therefore, final human scores were reported as average ratings from two blinded clinicians with bootstrap confidence intervals. Notably, GPT-5's self-evaluation results deviated from human judgment, exhibiting potential model-based bias.
Conclusion: This preliminary study suggests the potential of Gemini-2.5-Pro for automated consult letter generation, though the low test-set inter-rater reliability (QWK: 0.1505) highlights the subjective complexity of clinical text. To integrate these assistive prototypes into routine clinical application, more systematic studies are required.
Clinical Relevance: Integrating LLMs offers a promising approach to assisting with dental-to-medical referrals, presenting a potential future benefit of reducing administrative workloads and enabling healthcare providers to focus on safe, empathetic patient care.
(Copyright © 2026 The Authors. Published by Elsevier Inc. All rights reserved.)
Competing Interests: Declaration of competing interest None disclosed.
Contributed Indexing: Keywords: Artificial intelligence; Dental records; Large language models; Referral and consultation
Entry Date(s): Date Created: 20260903 Date Completed: 20260916 Latest Revision: 20260916
Update Code: 20260916
PubMed Central ID: PMC13571579
DOI: 10.1016/j.identj.2026.111088
PMID: 42691762
Βάση Δεδομένων: MEDLINE
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://www.doi.org/10.1016/j.identj.2026.111088?
    Name: ScienceDirect (all content) (s7799221)
    Category: fullText
    Text: View record from ScienceDirect
    MouseOverText: View record from ScienceDirect
  – Url: https://resolver.ebsco.com/c/fiv2js/result?sid=EBSCO:cmedm&genre=article&issn=1875595X&ISBN=&volume=76&issue=5&date=20261001&spage=111088&pages=111088&title=International dental journal&atitle=Evaluation%20of%20LLMs%20for%20Automated%20Dental-to-Medical%20Consult%20Letter%20Generation%3A%20Comparative%20Study.&aulast=Eom%20D&id=DOI:10.1016/j.identj.2026.111088
    Name: Full Text Finder (for New FTF UI) (ns324271)
    Category: fullText
    Text: Full Text Finder
    MouseOverText: Full Text Finder
Header DbId: cmedm
DbLabel: MEDLINE
An: 42691762
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Evaluation of LLMs for Automated Dental-to-Medical Consult Letter Generation: Comparative Study.
– Name: Author
  Label: Authors
  Group: Au
  Data: &lt;searchLink fieldCode=&quot;AU&quot; term=&quot;%22Eom+D%22&quot;&gt;Eom D&lt;/searchLink&gt;; Interdisciplinary Program of Medical Informatics, College of Medicine, Seoul National University, Seoul, Republic of Korea.&lt;br /&gt;&lt;searchLink fieldCode=&quot;AU&quot; term=&quot;%22Koh+K%22&quot;&gt;Koh K&lt;/searchLink&gt;; Department of Conservative Dentistry, Eunpyeong St. Mary&#39;s Hospital, College of Medicine, The Catholic University of Korea, Seoul, Republic of Korea; Department of Dentistry, Graduate School of Chosun University, Gwangju, Republic of Korea.&lt;br /&gt;&lt;searchLink fieldCode=&quot;AU&quot; term=&quot;%22Chung+H%22&quot;&gt;Chung H&lt;/searchLink&gt;; Department of Transdisciplinary Medicine, Seoul National University Hospital, Seoul, Republic of Korea.&lt;br /&gt;&lt;searchLink fieldCode=&quot;AU&quot; term=&quot;%22Kim+K%22&quot;&gt;Kim K&lt;/searchLink&gt;; Department of Transdisciplinary Medicine, Seoul National University Hospital, Seoul, Republic of Korea; Center for Data Science, Healthcare AI Research Institute, Seoul National University Hospital, Seoul, Republic of Korea; Department of Medicine, College of Medicine, Seoul National University, Seoul, Republic of Korea.&lt;br /&gt;&lt;searchLink fieldCode=&quot;AU&quot; term=&quot;%22Lee+SH%22&quot;&gt;Lee SH&lt;/searchLink&gt;; Division of Oral &amp; Maxillofacial Surgery Department of Dentistry, Eunpyeong St. Mary&#39;s Hospital, College of Medicine, The Catholic University of Korea, Seoul, Republic of Korea. Electronic address: justina@catholic.ac.kr.
– Name: TitleSource
  Label: Source
  Group: Src
  Data: &lt;searchLink fieldCode=&quot;JN&quot; term=&quot;%220374714%22&quot;&gt;International dental journal&lt;/searchLink&gt; [Int Dent J] 2026 Oct; Vol. 76 (5), pp. 111088. &lt;i&gt;Date of Electronic Publication: &lt;/i&gt;2026 Sep 03.
– Name: TypePub
  Label: Publication Type
  Group: TypPub
  Data: Journal Article; Comparative Study; Evaluation Study
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: TitleSource
  Label: Journal Info
  Group: Src
  Data: &lt;i&gt;Publisher: &lt;/i&gt;&lt;searchLink fieldCode=&quot;PB&quot; term=&quot;%22Elsevier%22&quot;&gt;Elsevier &lt;/searchLink&gt;&lt;i&gt;Country of Publication: &lt;/i&gt;England &lt;i&gt;NLM ID: &lt;/i&gt;0374714 &lt;i&gt;Publication Model: &lt;/i&gt;Print-Electronic &lt;i&gt;Cited Medium: &lt;/i&gt;Internet &lt;i&gt;ISSN: &lt;/i&gt;1875-595X (Electronic) &lt;i&gt;Linking ISSN: &lt;/i&gt;&lt;searchLink fieldCode=&quot;IS&quot; term=&quot;%2200206539%22&quot;&gt;00206539 &lt;/searchLink&gt;&lt;i&gt;NLM ISO Abbreviation: &lt;/i&gt;Int Dent J &lt;i&gt;Subsets: &lt;/i&gt;MEDLINE
– Name: PublisherInfo
  Label: Imprint Name(s)
  Group: PubInfo
  Data: &lt;i&gt;Publication&lt;/i&gt;: 2021- : London : Elsevier&lt;br /&gt;&lt;i&gt;Original Publication&lt;/i&gt;: London : FDI World Dental
– Name: SubjectMESH
  Label: MeSH Terms
  Group: Su
  Data: &lt;searchLink fieldCode=&quot;MM&quot; term=&quot;%22Large+Language+Models%22&quot;&gt;Large Language Models*&lt;/searchLink&gt; &lt;br /&gt;&lt;searchLink fieldCode=&quot;MM&quot; term=&quot;%22Referral+and+Consultation%22&quot;&gt;Referral and Consultation*&lt;/searchLink&gt; &lt;br /&gt;&lt;searchLink fieldCode=&quot;MM&quot; term=&quot;%22Correspondence+as+Topic%22&quot;&gt;Correspondence as Topic*&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;MH&quot; term=&quot;%22Humans%22&quot;&gt;Humans&lt;/searchLink&gt; ; &lt;searchLink fieldCode=&quot;MH&quot; term=&quot;%22Feasibility+Studies%22&quot;&gt;Feasibility Studies&lt;/searchLink&gt;
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Introduction and Aims: For dental patients with systemic diseases, multidisciplinary care via referral letters is essential for clinical safety. This study evaluated the clinical feasibility and quality of dental-to-medical referral letters generated by Large Language Models (LLMs).&lt;br /&gt;Methods: Using 153 paired datasets, three LLMs - Gemini-2.5-Pro, GPT-5 and MedGemma - generated referral letters via a standardised prompting framework. Two dental clinicians performed a blinded assessment across nine sub-criteria covering content, style and correctness. GPT-5 also performed an automated evaluation. Statistical significance was analysed using Friedman and Wilcoxon signed-rank tests with Holm correction.&lt;br /&gt;Results: In the human clinical evaluation, Gemini-2.5-Pro achieved the highest overall performance (TOTAL mean SCORE: 2.9487 with a bootstrap 95% confidence interval [CI] of 2.9345-2.9606), significantly outperforming GPT-5 (2.8818 with 95% CI of 2.8613-2.9008) and MedGemma (2.8613 with 95% CI of 2.8338-2.8865) (P &amp;lt; .05). While GPT-5 showed high clarity in systemic problem descriptions, MedGemma excelled in structural adherence. The overall performance difference between GPT-5 and MedGemma was not statistically significant (P = .3519). Inter-rater reliability was strong in the prompt search dataset (QWK: 0.7175) but lower in the independent test dataset (QWK: 0.1505), reflecting the subjective nature of clinical document assessment; therefore, final human scores were reported as average ratings from two blinded clinicians with bootstrap confidence intervals. Notably, GPT-5&#39;s self-evaluation results deviated from human judgment, exhibiting potential model-based bias.&lt;br /&gt;Conclusion: This preliminary study suggests the potential of Gemini-2.5-Pro for automated consult letter generation, though the low test-set inter-rater reliability (QWK: 0.1505) highlights the subjective complexity of clinical text. To integrate these assistive prototypes into routine clinical application, more systematic studies are required.&lt;br /&gt;Clinical Relevance: Integrating LLMs offers a promising approach to assisting with dental-to-medical referrals, presenting a potential future benefit of reducing administrative workloads and enabling healthcare providers to focus on safe, empathetic patient care.&lt;br /&gt; (Copyright &#169; 2026 The Authors. Published by Elsevier Inc. All rights reserved.)
– Name: Abstract
  Label: Competing Interests
  Group: Ab
  Data: Declaration of competing interest None disclosed.
– Name: SubjectMinor
  Label: Contributed Indexing
  Group:
  Data: &lt;i&gt;Keywords: &lt;/i&gt;Artificial intelligence; Dental records; Large language models; Referral and consultation
– Name: DateEntry
  Label: Entry Date(s)
  Group: Date
  Data: &lt;i&gt;Date Created: &lt;/i&gt;20260903 &lt;i&gt;Date Completed: &lt;/i&gt;20260916 &lt;i&gt;Latest Revision: &lt;/i&gt;20260916
– Name: DateUpdate
  Label: Update Code
  Group: Date
  Data: 20260916
– Name: PubmedCentralID
  Label: PubMed Central ID
  Group: ID
  Data: PMC13571579
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1016/j.identj.2026.111088
– Name: AN
  Label: PMID
  Group: ID
  Data: 42691762
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=cmedm&AN=42691762
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1016/j.identj.2026.111088
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        StartPage: 111088
    Subjects:
      – SubjectFull: Humans
        Type: general
      – SubjectFull: Feasibility Studies
        Type: general
      – SubjectFull: Large Language Models
        Type: general
      – SubjectFull: Referral and Consultation
        Type: general
      – SubjectFull: Correspondence as Topic
        Type: general
    Titles:
      – TitleFull: Evaluation of LLMs for Automated Dental-to-Medical Consult Letter Generation: Comparative Study.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Eom D
      – PersonEntity:
          Name:
            NameFull: Koh K
      – PersonEntity:
          Name:
            NameFull: Chung H
      – PersonEntity:
          Name:
            NameFull: Kim K
      – PersonEntity:
          Name:
            NameFull: Lee SH
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 10
              Text: 2026 Oct
              Type: published
              Y: 2026
          Identifiers:
            – Type: issn-electronic
              Value: 1875-595X
          Numbering:
            – Type: volume
              Value: 76
            – Type: issue
              Value: 5
          Titles:
            – TitleFull: International dental journal
              Type: main
ResultId 1