Academic Journal
Chain-of-verification prompting for NIH stroke scale extraction using small and frontier large language models.
| Title: | Chain-of-verification prompting for NIH stroke scale extraction using small and frontier large language models. |
|---|---|
| Authors: | Lott J; School of Medicine, Queen's University, 80 Barrie Street, Kingston, ON K7L 3N6, Canada., Stubbert B; Schulich School of Medicine & Dentistry, Western University, 1151 Richmond St, London, ON N6A 5C1, Canada., McShannon D; Faculty of Engineering, McMaster University, 1280 Main St W, Hamilton, ON L8S 4E7, Canada., Bellissimo J; School of Medicine, Queen's University, 80 Barrie Street, Kingston, ON K7L 3N6, Canada., Patel D; School of Medicine, Queen's University, 80 Barrie Street, Kingston, ON K7L 3N6, Canada., Dietrich N; Temerty Faculty of Medicine, University of Toronto, 1 King's College Cir, Toronto, ON M5S 1A8, Canada. Electronic address: nicholas.dietrich@mail.utoronto.ca. |
| Source: | International journal of medical informatics [Int J Med Inform] 2026 Oct; Vol. 219, pp. 106573. Date of Electronic Publication: 2026 Jun 25. |
| Publication Type: | Journal Article |
| Language: | English |
| Journal Info: | Publisher: Elsevier Science Ireland Ltd Country of Publication: Ireland NLM ID: 9711057 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 1872-8243 (Electronic) Linking ISSN: 13865056 NLM ISO Abbreviation: Int J Med Inform Subsets: MEDLINE |
| Imprint Name(s): | Original Publication: Shannon, Co. Clare, Ireland : Elsevier Science Ireland Ltd., c1997- |
| MeSH Terms: | Stroke*/diagnosis , Electronic Health Records* , Natural Language Processing*, Large Language Models ; United States ; Humans ; National Institutes of Health (U.S.) |
| Abstract: | Background: The National Institutes of Health Stroke Scale (NIHSS) is critical to acute stroke care but is often documented in unstructured notes. Large language models (LLMs) can enable automated extraction, though smaller models often underperform relative to frontier systems. Chain-of-Verification (CoVe) prompting introduces a structured self-verification step that may improve performance. Methods: We evaluated eight LLMs on 312 discharge summaries. Small models included LLaMA 3.2 3B, Ministral 3B, Gemma 3 4B, and Qwen 3 4B. Frontier models included GPT-5.2, Gemini 3 Pro, Claude Opus 4.5, and Grok 4. Each model was tested under a baseline and CoVe prompt. Outcomes were subscore exact-match accuracy, subscore mean absolute error (MAE), total score exact-match accuracy, and total score MAE. Results: At baseline, small models achieved 53.2 ± 10.0% subscore accuracy and subscore MAE 0.84 ± 0.22, compared with 88.5 ± 10.1% and 0.15 ± 0.16 in frontier models (both p < 0.001). Total exact accuracy was low in both groups (7.7 ± 12.9% vs 35.9 ± 32.4%). CoVe significantly improved small-model performance (subscore accuracy 65.0 ± 10.9%; subscore MAE 0.55 ± 0.21; total MAE 4.84 ± 2.30 vs 7.19 ± 3.54 at baseline; all p < 0.001), although total exact accuracy remained modest (9.6 ± 15.7%). Frontier models showed no significant group-level change with CoVe. Conclusion: CoVe prompting substantially improves NIHSS extraction in small LLMs while producing negligible effects in frontier models. Although smaller model performance remains insufficient for standalone clinical deployment, CoVe prompting offers a promising avenue for further exploration. (Copyright © 2026 The Author(s). Published by Elsevier B.V. All rights reserved.) |
| Competing Interests: | Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. |
| Contributed Indexing: | Keywords: Chain-of-verification; Clinical information extraction; Large language models; NIH stroke scale; Natural language processing |
| Entry Date(s): | Date Created: 20260627 Date Completed: 20260716 Latest Revision: 20260716 |
| Update Code: | 20260717 |
| DOI: | 10.1016/j.ijmedinf.2026.106573 |
| PMID: | 42364465 |
| Database: | MEDLINE |
| ISSN: | 1872-8243 |
|---|---|
| DOI: | 10.1016/j.ijmedinf.2026.106573 |