Academic Journal
Προβολή σε EDS

Chain-of-verification prompting for NIH stroke scale extraction using small and frontier large language models.

Λεπτομέρειες βιβλιογραφικής εγγραφής
Τίτλος: Chain-of-verification prompting for NIH stroke scale extraction using small and frontier large language models.
Συγγραφείς: Lott J; School of Medicine, Queen's University, 80 Barrie Street, Kingston, ON K7L 3N6, Canada., Stubbert B; Schulich School of Medicine & Dentistry, Western University, 1151 Richmond St, London, ON N6A 5C1, Canada., McShannon D; Faculty of Engineering, McMaster University, 1280 Main St W, Hamilton, ON L8S 4E7, Canada., Bellissimo J; School of Medicine, Queen's University, 80 Barrie Street, Kingston, ON K7L 3N6, Canada., Patel D; School of Medicine, Queen's University, 80 Barrie Street, Kingston, ON K7L 3N6, Canada., Dietrich N; Temerty Faculty of Medicine, University of Toronto, 1 King's College Cir, Toronto, ON M5S 1A8, Canada. Electronic address: nicholas.dietrich@mail.utoronto.ca.
Πηγή: International journal of medical informatics [Int J Med Inform] 2026 Oct; Vol. 219, pp. 106573. Date of Electronic Publication: 2026 Jun 25.
Τύπος έκδοσης: Journal Article
Γλώσσα: English
Στοιχεία περιοδικού: Publisher: Elsevier Science Ireland Ltd Country of Publication: Ireland NLM ID: 9711057 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 1872-8243 (Electronic) Linking ISSN: 13865056 NLM ISO Abbreviation: Int J Med Inform Subsets: MEDLINE
Imprint Name(s): Original Publication: Shannon, Co. Clare, Ireland : Elsevier Science Ireland Ltd., c1997-
Ιατρικοί όροι (MeSH): Stroke*/diagnosis , Electronic Health Records* , Natural Language Processing*, Large Language Models ; United States ; Humans ; National Institutes of Health (U.S.)
Περίληψη: Background: The National Institutes of Health Stroke Scale (NIHSS) is critical to acute stroke care but is often documented in unstructured notes. Large language models (LLMs) can enable automated extraction, though smaller models often underperform relative to frontier systems. Chain-of-Verification (CoVe) prompting introduces a structured self-verification step that may improve performance.
Methods: We evaluated eight LLMs on 312 discharge summaries. Small models included LLaMA 3.2 3B, Ministral 3B, Gemma 3 4B, and Qwen 3 4B. Frontier models included GPT-5.2, Gemini 3 Pro, Claude Opus 4.5, and Grok 4. Each model was tested under a baseline and CoVe prompt. Outcomes were subscore exact-match accuracy, subscore mean absolute error (MAE), total score exact-match accuracy, and total score MAE.
Results: At baseline, small models achieved 53.2 ± 10.0% subscore accuracy and subscore MAE 0.84 ± 0.22, compared with 88.5 ± 10.1% and 0.15 ± 0.16 in frontier models (both p < 0.001). Total exact accuracy was low in both groups (7.7 ± 12.9% vs 35.9 ± 32.4%). CoVe significantly improved small-model performance (subscore accuracy 65.0 ± 10.9%; subscore MAE 0.55 ± 0.21; total MAE 4.84 ± 2.30 vs 7.19 ± 3.54 at baseline; all p < 0.001), although total exact accuracy remained modest (9.6 ± 15.7%). Frontier models showed no significant group-level change with CoVe.
Conclusion: CoVe prompting substantially improves NIHSS extraction in small LLMs while producing negligible effects in frontier models. Although smaller model performance remains insufficient for standalone clinical deployment, CoVe prompting offers a promising avenue for further exploration.
(Copyright © 2026 The Author(s). Published by Elsevier B.V. All rights reserved.)
Competing Interests: Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Contributed Indexing: Keywords: Chain-of-verification; Clinical information extraction; Large language models; NIH stroke scale; Natural language processing
Entry Date(s): Date Created: 20260627 Date Completed: 20260716 Latest Revision: 20260716
Update Code: 20260717
DOI: 10.1016/j.ijmedinf.2026.106573
PMID: 42364465
Βάση Δεδομένων: MEDLINE
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://www.doi.org/10.1016/j.ijmedinf.2026.106573?
    Name: ScienceDirect (all content) (s7799221)
    Category: fullText
    Text: View record from ScienceDirect
    MouseOverText: View record from ScienceDirect
  – Url: https://resolver.ebsco.com/c/fiv2js/result?sid=EBSCO:cmedm&genre=article&issn=18728243&ISBN=&volume=219&issue=&date=20261001&spage=106573&pages=106573&title=International journal of medical informatics&atitle=Chain-of-verification%20prompting%20for%20NIH%20stroke%20scale%20extraction%20using%20small%20and%20frontier%20large%20language%20models.&aulast=Lott%20J&id=DOI:10.1016/j.ijmedinf.2026.106573
    Name: Full Text Finder (for New FTF UI) (ns324271)
    Category: fullText
    Text: Full Text Finder
    MouseOverText: Full Text Finder
Header DbId: cmedm
DbLabel: MEDLINE
An: 42364465
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Chain-of-verification prompting for NIH stroke scale extraction using small and frontier large language models.
– Name: Author
  Label: Authors
  Group: Au
  Data: &lt;searchLink fieldCode=&quot;AU&quot; term=&quot;%22Lott+J%22&quot;&gt;Lott J&lt;/searchLink&gt;; School of Medicine, Queen&#39;s University, 80 Barrie Street, Kingston, ON K7L 3N6, Canada.&lt;br /&gt;&lt;searchLink fieldCode=&quot;AU&quot; term=&quot;%22Stubbert+B%22&quot;&gt;Stubbert B&lt;/searchLink&gt;; Schulich School of Medicine &amp; Dentistry, Western University, 1151 Richmond St, London, ON N6A 5C1, Canada.&lt;br /&gt;&lt;searchLink fieldCode=&quot;AU&quot; term=&quot;%22McShannon+D%22&quot;&gt;McShannon D&lt;/searchLink&gt;; Faculty of Engineering, McMaster University, 1280 Main St W, Hamilton, ON L8S 4E7, Canada.&lt;br /&gt;&lt;searchLink fieldCode=&quot;AU&quot; term=&quot;%22Bellissimo+J%22&quot;&gt;Bellissimo J&lt;/searchLink&gt;; School of Medicine, Queen&#39;s University, 80 Barrie Street, Kingston, ON K7L 3N6, Canada.&lt;br /&gt;&lt;searchLink fieldCode=&quot;AU&quot; term=&quot;%22Patel+D%22&quot;&gt;Patel D&lt;/searchLink&gt;; School of Medicine, Queen&#39;s University, 80 Barrie Street, Kingston, ON K7L 3N6, Canada.&lt;br /&gt;&lt;searchLink fieldCode=&quot;AU&quot; term=&quot;%22Dietrich+N%22&quot;&gt;Dietrich N&lt;/searchLink&gt;; Temerty Faculty of Medicine, University of Toronto, 1 King&#39;s College Cir, Toronto, ON M5S 1A8, Canada. Electronic address: nicholas.dietrich@mail.utoronto.ca.
– Name: TitleSource
  Label: Source
  Group: Src
  Data: &lt;searchLink fieldCode=&quot;JN&quot; term=&quot;%229711057%22&quot;&gt;International journal of medical informatics&lt;/searchLink&gt; [Int J Med Inform] 2026 Oct; Vol. 219, pp. 106573. &lt;i&gt;Date of Electronic Publication: &lt;/i&gt;2026 Jun 25.
– Name: TypePub
  Label: Publication Type
  Group: TypPub
  Data: Journal Article
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: TitleSource
  Label: Journal Info
  Group: Src
  Data: &lt;i&gt;Publisher: &lt;/i&gt;&lt;searchLink fieldCode=&quot;PB&quot; term=&quot;%22Elsevier+Science+Ireland+Ltd%22&quot;&gt;Elsevier Science Ireland Ltd &lt;/searchLink&gt;&lt;i&gt;Country of Publication: &lt;/i&gt;Ireland &lt;i&gt;NLM ID: &lt;/i&gt;9711057 &lt;i&gt;Publication Model: &lt;/i&gt;Print-Electronic &lt;i&gt;Cited Medium: &lt;/i&gt;Internet &lt;i&gt;ISSN: &lt;/i&gt;1872-8243 (Electronic) &lt;i&gt;Linking ISSN: &lt;/i&gt;&lt;searchLink fieldCode=&quot;IS&quot; term=&quot;%2213865056%22&quot;&gt;13865056 &lt;/searchLink&gt;&lt;i&gt;NLM ISO Abbreviation: &lt;/i&gt;Int J Med Inform &lt;i&gt;Subsets: &lt;/i&gt;MEDLINE
– Name: PublisherInfo
  Label: Imprint Name(s)
  Group: PubInfo
  Data: &lt;i&gt;Original Publication&lt;/i&gt;: Shannon, Co. Clare, Ireland : Elsevier Science Ireland Ltd., c1997-
– Name: SubjectMESH
  Label: MeSH Terms
  Group: Su
  Data: &lt;searchLink fieldCode=&quot;MM&quot; term=&quot;%22Stroke%22&quot;&gt;Stroke*&lt;/searchLink&gt;/&lt;searchLink fieldCode=&quot;MM&quot; term=&quot;%22Stroke+diagnosis%22&quot;&gt;diagnosis&lt;/searchLink&gt; &lt;br /&gt;&lt;searchLink fieldCode=&quot;MM&quot; term=&quot;%22Electronic+Health+Records%22&quot;&gt;Electronic Health Records*&lt;/searchLink&gt; &lt;br /&gt;&lt;searchLink fieldCode=&quot;MM&quot; term=&quot;%22Natural+Language+Processing%22&quot;&gt;Natural Language Processing*&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;MH&quot; term=&quot;%22Large+Language+Models%22&quot;&gt;Large Language Models&lt;/searchLink&gt; ; &lt;searchLink fieldCode=&quot;MH&quot; term=&quot;%22United+States%22&quot;&gt;United States&lt;/searchLink&gt; ; &lt;searchLink fieldCode=&quot;MH&quot; term=&quot;%22Humans%22&quot;&gt;Humans&lt;/searchLink&gt; ; &lt;searchLink fieldCode=&quot;MH&quot; term=&quot;%22National+Institutes+of+Health+%28U%2ES%2E%29%22&quot;&gt;National Institutes of Health (U.S.)&lt;/searchLink&gt;
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Background: The National Institutes of Health Stroke Scale (NIHSS) is critical to acute stroke care but is often documented in unstructured notes. Large language models (LLMs) can enable automated extraction, though smaller models often underperform relative to frontier systems. Chain-of-Verification (CoVe) prompting introduces a structured self-verification step that may improve performance.&lt;br /&gt;Methods: We evaluated eight LLMs on 312 discharge summaries. Small models included LLaMA 3.2 3B, Ministral 3B, Gemma 3 4B, and Qwen 3 4B. Frontier models included GPT-5.2, Gemini 3 Pro, Claude Opus 4.5, and Grok 4. Each model was tested under a baseline and CoVe prompt. Outcomes were subscore exact-match accuracy, subscore mean absolute error (MAE), total score exact-match accuracy, and total score MAE.&lt;br /&gt;Results: At baseline, small models achieved 53.2&#160;&#177;&#160;10.0% subscore accuracy and subscore MAE 0.84&#160;&#177;&#160;0.22, compared with 88.5&#160;&#177;&#160;10.1% and 0.15&#160;&#177;&#160;0.16 in frontier models (both p&#160;&amp;lt;&#160;0.001). Total exact accuracy was low in both groups (7.7&#160;&#177;&#160;12.9% vs 35.9&#160;&#177;&#160;32.4%). CoVe significantly improved small-model performance (subscore accuracy 65.0&#160;&#177;&#160;10.9%; subscore MAE 0.55&#160;&#177;&#160;0.21; total MAE 4.84&#160;&#177;&#160;2.30 vs 7.19&#160;&#177;&#160;3.54 at baseline; all p&#160;&amp;lt;&#160;0.001), although total exact accuracy remained modest (9.6&#160;&#177;&#160;15.7%). Frontier models showed no significant group-level change with CoVe.&lt;br /&gt;Conclusion: CoVe prompting substantially improves NIHSS extraction in small LLMs while producing negligible effects in frontier models. Although smaller model performance remains insufficient for standalone clinical deployment, CoVe prompting offers a promising avenue for further exploration.&lt;br /&gt; (Copyright &#169; 2026 The Author(s). Published by Elsevier B.V. All rights reserved.)
– Name: Abstract
  Label: Competing Interests
  Group: Ab
  Data: Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
– Name: SubjectMinor
  Label: Contributed Indexing
  Group:
  Data: &lt;i&gt;Keywords: &lt;/i&gt;Chain-of-verification; Clinical information extraction; Large language models; NIH stroke scale; Natural language processing
– Name: DateEntry
  Label: Entry Date(s)
  Group: Date
  Data: &lt;i&gt;Date Created: &lt;/i&gt;20260627 &lt;i&gt;Date Completed: &lt;/i&gt;20260716 &lt;i&gt;Latest Revision: &lt;/i&gt;20260716
– Name: DateUpdate
  Label: Update Code
  Group: Date
  Data: 20260717
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1016/j.ijmedinf.2026.106573
– Name: AN
  Label: PMID
  Group: ID
  Data: 42364465
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=cmedm&AN=42364465
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1016/j.ijmedinf.2026.106573
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        StartPage: 106573
    Subjects:
      – SubjectFull: Large Language Models
        Type: general
      – SubjectFull: United States
        Type: general
      – SubjectFull: Humans
        Type: general
      – SubjectFull: National Institutes of Health (U.S.)
        Type: general
      – SubjectFull: Stroke diagnosis
        Type: general
      – SubjectFull: Electronic Health Records
        Type: general
      – SubjectFull: Natural Language Processing
        Type: general
    Titles:
      – TitleFull: Chain-of-verification prompting for NIH stroke scale extraction using small and frontier large language models.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Lott J
      – PersonEntity:
          Name:
            NameFull: Stubbert B
      – PersonEntity:
          Name:
            NameFull: McShannon D
      – PersonEntity:
          Name:
            NameFull: Bellissimo J
      – PersonEntity:
          Name:
            NameFull: Patel D
      – PersonEntity:
          Name:
            NameFull: Dietrich N
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 10
              Text: 2026 Oct
              Type: published
              Y: 2026
          Identifiers:
            – Type: issn-electronic
              Value: 1872-8243
          Numbering:
            – Type: volume
              Value: 219
          Titles:
            – TitleFull: International journal of medical informatics
              Type: main
ResultId 1