Academic Journal
Προβολή σε EDS

Reasoning or reciting? A temporal contamination audit of large language models in clinical medicine.

Λεπτομέρειες βιβλιογραφικής εγγραφής
Τίτλος: Reasoning or reciting? A temporal contamination audit of large language models in clinical medicine.
Συγγραφείς: Sheppert AP; Internal Medicine, Legacy Health System, Portland, OR 98686, United States.; Internal Medicine, The Vancouver Clinic, Vancouver, WA 98686, United States.; Artificial Intelligence, Capitol Technology University, Laurel, MD 20708, United States., Adams B; Vascular Surgery, Intermountain Healthcare, Salt Lake City, UT 84604, United States.; Bioinformatics, University of Utah, Salt Lake City, UT 84112, United States., Sheppert AD; Internal Medicine, Legacy Health System, Portland, OR 98686, United States.; Internal Medicine, The Vancouver Clinic, Vancouver, WA 98686, United States., Riley S; Internal Medicine, Legacy Health System, Portland, OR 98686, United States.; Internal Medicine, The Vancouver Clinic, Vancouver, WA 98686, United States.
Πηγή: Journal of the American Medical Informatics Association : JAMIA [J Am Med Inform Assoc] 2026 Sep 01; Vol. 33 (9), pp. 1617-1626.
Τύπος έκδοσης: Journal Article
Γλώσσα: English
Στοιχεία περιοδικού: Publisher: Oxford University Press Country of Publication: England NLM ID: 9430800 Publication Model: Print Cited Medium: Internet ISSN: 1527-974X (Electronic) Linking ISSN: 10675027 NLM ISO Abbreviation: J Am Med Inform Assoc Subsets: MEDLINE
Imprint Name(s): Publication: 2015- : Oxford : Oxford University Press
Original Publication: Philadelphia, PA : Hanley & Belfus, c1993-
Ιατρικοί όροι (MeSH): Large Language Models* , Clinical Reasoning* , Artificial Intelligence* , Clinical Medicine*, Humans ; Semantics
Περίληψη: Objective: Evaluate whether large language models reason or simply regurgitate training data in clinical diagnosis.
Materials and Methods: We audited 2000 clinical case reports from PubMed Central: 1000 from 2021 to 2022 (within training data) and 1000 from 2025 (after training cutoffs). Five frontier LLMs generated diagnoses evaluated by an independent AI judge validated against physician consensus (n = 10 000 evaluations).
Results: Diagnostic accuracy was virtually identical across temporal cohorts (66.8% contaminated vs 66.9% clean), directly contradicting the memorization hypothesis. Lexical similarity was uniformly low (mean ROUGE-L 0.057), and semantic similarity measured by BERTScore showed no memorization signal (F1 0.8182 contaminated vs 0.8195 clean, Δ = +0.0013), confirming that models generate novel reasoning rather than regurgitating training data.
Discussion: This large-scale audit, using both lexical and semantic similarity metrics, provides compelling evidence that LLMs engage in genuine clinical reasoning rather than regurgitating memorized training data.
Conclusion: Models demonstrated equivalent accuracy on cases they could not have seen during training, suggesting they have internalized generalizable medical knowledge rather than memorizing specific cases.
(© The Author(s) 2026. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For commercial re-use, please contact reprints@oup.com for reprints and translation rights for reprints. All other permissions can be obtained through our RightsLink service via the Permissions link on the article page on our site—for further information please contact journals.permissions@oup.com.)
Contributed Indexing: Keywords: artificial intelligence; clinical reasoning; data contamination; large language models; medical diagnosis
Entry Date(s): Date Created: 20260612 Date Completed: 20260916 Latest Revision: 20260918
Update Code: 20260918
PubMed Central ID: PMC13580742
DOI: 10.1093/jamia/ocag069
PMID: 42281378
Βάση Δεδομένων: MEDLINE
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://resolver.ebsco.com/c/fiv2js/result?sid=EBSCO:cmedm&genre=article&issn=1527974X&ISBN=&volume=33&issue=9&date=20260901&spage=1617&pages=1617-1626&title=Journal of the American Medical Informatics Association : JAMIA&atitle=Reasoning%20or%20reciting%3F%20A%20temporal%20contamination%20audit%20of%20large%20language%20models%20in%20clinical%20medicine.&aulast=Sheppert%20AP&id=DOI:10.1093/jamia/ocag069
    Name: Full Text Finder (for New FTF UI) (ns324271)
    Category: fullText
    Text: Full Text Finder
    MouseOverText: Full Text Finder
Header DbId: cmedm
DbLabel: MEDLINE
An: 42281378
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Reasoning or reciting? A temporal contamination audit of large language models in clinical medicine.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AU" term="%22Sheppert+AP%22">Sheppert AP</searchLink>; Internal Medicine, Legacy Health System, Portland, OR 98686, United States.; Internal Medicine, The Vancouver Clinic, Vancouver, WA 98686, United States.; Artificial Intelligence, Capitol Technology University, Laurel, MD 20708, United States.<br /><searchLink fieldCode="AU" term="%22Adams+B%22">Adams B</searchLink>; Vascular Surgery, Intermountain Healthcare, Salt Lake City, UT 84604, United States.; Bioinformatics, University of Utah, Salt Lake City, UT 84112, United States.<br /><searchLink fieldCode="AU" term="%22Sheppert+AD%22">Sheppert AD</searchLink>; Internal Medicine, Legacy Health System, Portland, OR 98686, United States.; Internal Medicine, The Vancouver Clinic, Vancouver, WA 98686, United States.<br /><searchLink fieldCode="AU" term="%22Riley+S%22">Riley S</searchLink>; Internal Medicine, Legacy Health System, Portland, OR 98686, United States.; Internal Medicine, The Vancouver Clinic, Vancouver, WA 98686, United States.
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%229430800%22">Journal of the American Medical Informatics Association : JAMIA</searchLink> [J Am Med Inform Assoc] 2026 Sep 01; Vol. 33 (9), pp. 1617-1626.
– Name: TypePub
  Label: Publication Type
  Group: TypPub
  Data: Journal Article
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: TitleSource
  Label: Journal Info
  Group: Src
  Data: <i>Publisher: </i><searchLink fieldCode="PB" term="%22Oxford+University+Press%22">Oxford University Press </searchLink><i>Country of Publication: </i>England <i>NLM ID: </i>9430800 <i>Publication Model: </i>Print <i>Cited Medium: </i>Internet <i>ISSN: </i>1527-974X (Electronic) <i>Linking ISSN: </i><searchLink fieldCode="IS" term="%2210675027%22">10675027 </searchLink><i>NLM ISO Abbreviation: </i>J Am Med Inform Assoc <i>Subsets: </i>MEDLINE
– Name: PublisherInfo
  Label: Imprint Name(s)
  Group: PubInfo
  Data: <i>Publication</i>: 2015- : Oxford : Oxford University Press<br /><i>Original Publication</i>: Philadelphia, PA : Hanley & Belfus, c1993-
– Name: SubjectMESH
  Label: MeSH Terms
  Group: Su
  Data: <searchLink fieldCode="MM" term="%22Large+Language+Models%22">Large Language Models*</searchLink> <br /><searchLink fieldCode="MM" term="%22Clinical+Reasoning%22">Clinical Reasoning*</searchLink> <br /><searchLink fieldCode="MM" term="%22Artificial+Intelligence%22">Artificial Intelligence*</searchLink> <br /><searchLink fieldCode="MM" term="%22Clinical+Medicine%22">Clinical Medicine*</searchLink><br /><searchLink fieldCode="MH" term="%22Humans%22">Humans</searchLink> ; <searchLink fieldCode="MH" term="%22Semantics%22">Semantics</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Objective: Evaluate whether large language models reason or simply regurgitate training data in clinical diagnosis.<br />Materials and Methods: We audited 2000 clinical case reports from PubMed Central: 1000 from 2021 to 2022 (within training data) and 1000 from 2025 (after training cutoffs). Five frontier LLMs generated diagnoses evaluated by an independent AI judge validated against physician consensus (n = 10 000 evaluations).<br />Results: Diagnostic accuracy was virtually identical across temporal cohorts (66.8% contaminated vs 66.9% clean), directly contradicting the memorization hypothesis. Lexical similarity was uniformly low (mean ROUGE-L 0.057), and semantic similarity measured by BERTScore showed no memorization signal (F1 0.8182 contaminated vs 0.8195 clean, Δ = +0.0013), confirming that models generate novel reasoning rather than regurgitating training data.<br />Discussion: This large-scale audit, using both lexical and semantic similarity metrics, provides compelling evidence that LLMs engage in genuine clinical reasoning rather than regurgitating memorized training data.<br />Conclusion: Models demonstrated equivalent accuracy on cases they could not have seen during training, suggesting they have internalized generalizable medical knowledge rather than memorizing specific cases.<br /> (© The Author(s) 2026. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For commercial re-use, please contact reprints@oup.com for reprints and translation rights for reprints. All other permissions can be obtained through our RightsLink service via the Permissions link on the article page on our site—for further information please contact journals.permissions@oup.com.)
– Name: SubjectMinor
  Label: Contributed Indexing
  Group:
  Data: <i>Keywords: </i>artificial intelligence; clinical reasoning; data contamination; large language models; medical diagnosis
– Name: DateEntry
  Label: Entry Date(s)
  Group: Date
  Data: <i>Date Created: </i>20260612 <i>Date Completed: </i>20260916 <i>Latest Revision: </i>20260918
– Name: DateUpdate
  Label: Update Code
  Group: Date
  Data: 20260918
– Name: PubmedCentralID
  Label: PubMed Central ID
  Group: ID
  Data: PMC13580742
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1093/jamia/ocag069
– Name: AN
  Label: PMID
  Group: ID
  Data: 42281378
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=cmedm&AN=42281378
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1093/jamia/ocag069
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        StartPage: 1617
    Subjects:
      – SubjectFull: Humans
        Type: general
      – SubjectFull: Semantics
        Type: general
      – SubjectFull: Large Language Models
        Type: general
      – SubjectFull: Clinical Reasoning
        Type: general
      – SubjectFull: Artificial Intelligence
        Type: general
      – SubjectFull: Clinical Medicine
        Type: general
    Titles:
      – TitleFull: Reasoning or reciting? A temporal contamination audit of large language models in clinical medicine.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Sheppert AP
      – PersonEntity:
          Name:
            NameFull: Adams B
      – PersonEntity:
          Name:
            NameFull: Sheppert AD
      – PersonEntity:
          Name:
            NameFull: Riley S
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 09
              Text: 2026 Sep 01
              Type: published
              Y: 2026
          Identifiers:
            – Type: issn-electronic
              Value: 1527-974X
          Numbering:
            – Type: volume
              Value: 33
            – Type: issue
              Value: 9
          Titles:
            – TitleFull: Journal of the American Medical Informatics Association : JAMIA
              Type: main
ResultId 1