Academic Journal
View in EDS

Process-Oriented, Behaviorally Anchored Assessment of Clinical Reasoning in Large Language Models and the Effect of Extended Thinking: Protocol for a Prospective, Multigroup, Comparative Study.

Bibliographic Details
Title: Process-Oriented, Behaviorally Anchored Assessment of Clinical Reasoning in Large Language Models and the Effect of Extended Thinking: Protocol for a Prospective, Multigroup, Comparative Study.
Authors: Agha A; Department of Internal Medicine, College of Medicine and Health Sciences, United Arab Emirates University, AlAin, Abu Dhabi, United Arab Emirates., Jalil M; College of Business and Economics, United Arab Emirates University, AlAin, Abu Dhabi, United Arab Emirates., Anwar E; Department of Internal Medicine, College of Medicine and Health Sciences, United Arab Emirates University, AlAin, Abu Dhabi, United Arab Emirates., Bakoush O; Department of Internal Medicine, College of Medicine and Health Sciences, United Arab Emirates University, AlAin, Abu Dhabi, United Arab Emirates.
Source: JMIR research protocols [JMIR Res Protoc] 2026 Sep 22; Vol. 15, pp. e103220. Date of Electronic Publication: 2026 Sep 22.
Publication Type: Journal Article; Comparative Study
Language: English
Journal Info: Publisher: JMIR Publications Country of Publication: Canada NLM ID: 101599504 Publication Model: Electronic Cited Medium: Internet ISSN: 1929-0748 (Electronic) Linking ISSN: 19290748 NLM ISO Abbreviation: JMIR Res Protoc Subsets: MEDLINE
Imprint Name(s): Original Publication: Toronto : JMIR Publications
MeSH Terms: Large Language Models* , Clinical Reasoning* , Thinking*, Humans ; Prospective Studies
Abstract: Background: Most clinical reasoning evaluations in large language models (LLMs) score only the final answer, usually multiple-choice accuracy, which explains little about model reasoning. Two developments stress this gap: reasoning-optimized models are now common, and several expose an explicit extended thinking control. Whether that deliberation improves the reasoning process remains untested with a validated instrument.
Objective: This protocol introduces and aims to validate the Rapid Evaluation Assessment of Clinical Reasoning Tool (REACT)-AI, a Behaviorally Anchored Rating Scale (BARS) with 13 subdomains for process-oriented assessment of AI clinical reasoning, that is, the quality of the externalized reasoning rather than its faithfulness to the model's internal computation. It also builds a conflict-of-interest-controlled LLM-as-judge pipeline and tests whether extended thinking improves reasoning quality.
Methods: This 2-phase, prospective, comparative study has a within-model thinking-mode factor. Six flagship models are run in standard and extended thinking or reasoning modes, giving 12 conditions: 4 providers are toggled within the same model, and 2 pair a standard model with a dedicated reasoning model. In phase 1, 5 standardized urgent care vignettes are answered under all 12 conditions, 3 runs per condition (180 AI outputs), and by an independent expert clinician panel (n=5) and senior and junior medical students. Every output is scored by dual-blind human raters and, in parallel, by LLM judges, with no model judging its own family. Acceptance criteria for deploying the judge at scale are a weighted κ of at least 0.60 and an intraclass correlation coefficient above 0.75. In phase 2, the validated pipeline scores 3600 AI outputs. A self-correction turn and a paired Gulf English condition run alongside and are reported separately, with the AI metacognition assessment rubric and disinformation generation rate as secondary instruments.
Results: As of June 2026, ethics approval was obtained (ERSC_2025_6124), and the study is registered on the Open Science Framework under embargo. Responses to the 5 phase 1 vignettes were collected from senior and junior medical students between January and June 2026; 173 scripts were received and remain sealed, unopened, and unscored. No AI outputs have been generated, and the expert panel remains unrecruited. Following the June 1, 2026, model-version lock, AI generation and scoring will begin in September 2026, with phase 1 calibration through December 2026 and phase 2 from January to April 2027; results are expected in winter 2027. The study will report the validated instrument, judge calibration against human experts, self-preference bias by model family, and the effect of extended thinking on reflection and metacognition, including null findings.
Conclusions: REACT-AI and its judge pipeline are built to be reusable. They close the gap between accuracy benchmarks and genuine reasoning assessment and provide a template for studying reasoning mode effects.
Trial Registration: OSF Registries osf.io/jep5t; https://osf.io/jep5t.
International Registered Report Identifier (irrid): DERR1-10.2196/103220.
(©Adnan Agha, Muhammad Jalil, Eram Anwar, Omran Bakoush. Originally published in JMIR Research Protocols (https://www.researchprotocols.org), 22.09.2026.)
References: Nat Med. 2025 Jan;31(1):60-69. (PMID: 39779929)
JAMA Netw Open. 2026 Apr 01;9(4):e264003. (PMID: 41973425)
Nat Med. 2026 Feb;32(2):616-623. (PMID: 41652123)
NPJ Digit Med. 2025 Oct 7;8(1):600. (PMID: 41057566)
Nat Med. 2025 Jun;31(6):1873-1881. (PMID: 40195448)
Cureus. 2025 Oct 14;17(10):e94563. (PMID: 41246764)
Nat Commun. 2025 Nov 6;16(1):9799. (PMID: 41198657)
BMJ Open. 2024 Jul 18;14(7):e087469. (PMID: 39025818)
Nature. 2023 Aug;620(7972):172-180. (PMID: 37438534)
JAMA Netw Open. 2025 Aug 1;8(8):e2526021. (PMID: 40779272)
Acad Med. 2019 Jun;94(6):902-912. (PMID: 30720527)
BMJ. 2020 Sep 9;370:m3210. (PMID: 32907797)
JAMA. 2018 Jun 12;319(22):2267-2268. (PMID: 29800012)
Nat Med. 2026 Mar;32(3):943-951. (PMID: 41559415)
JAMA Netw Open. 2024 Oct 1;7(10):e2440969. (PMID: 39466245)
Acad Med. 2020 Aug;95(8):1166-1171. (PMID: 31577583)
J Appl Psychol. 2009 Sep;94(5):1336-44. (PMID: 19702375)
Nat Med. 2020 Sep;26(9):1364-1374. (PMID: 32908283)
J Gen Intern Med. 2022 Jul;37(9):2224-2229. (PMID: 35710662)
J Oral Facial Pain Headache. 2026 Mar;40(2):140-150. (PMID: 41914067)
Contributed Indexing: Keywords: LLM-as-judge; behaviorally anchored rating scale; clinical decision support; clinical reasoning assessment; extended thinking; large language models; metacognition; process-oriented evaluation; reasoning models; self-preference bias
Entry Date(s): Date Created: 20260922 Date Completed: 20260922 Latest Revision: 20261009
Update Code: 20261009
PubMed Central ID: PMC13643257
DOI: 10.2196/103220
PMID: 42772746
Database: MEDLINE
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://resolver.ebsco.com/c/fiv2js/result?sid=EBSCO:cmedm&genre=article&issn=19290748&ISBN=&volume=15&issue=&date=20260922&spage=e103220&pages=e103220&title=JMIR research protocols&atitle=Process-Oriented%2C%20Behaviorally%20Anchored%20Assessment%20of%20Clinical%20Reasoning%20in%20Large%20Language%20Models%20and%20the%20Effect%20of%20Extended%20Thinking%3A%20Protocol%20for%20a%20Prospective%2C%20Multigroup%2C%20Comparative%20Study.&aulast=Agha%20A&id=DOI:10.2196/103220
    Name: Full Text Finder (for New FTF UI) (ns324271)
    Category: fullText
    Text: Full Text Finder
    MouseOverText: Full Text Finder
Header DbId: cmedm
DbLabel: MEDLINE
An: 42772746
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Process-Oriented, Behaviorally Anchored Assessment of Clinical Reasoning in Large Language Models and the Effect of Extended Thinking: Protocol for a Prospective, Multigroup, Comparative Study.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AU" term="%22Agha+A%22">Agha A</searchLink>; Department of Internal Medicine, College of Medicine and Health Sciences, United Arab Emirates University, AlAin, Abu Dhabi, United Arab Emirates.<br /><searchLink fieldCode="AU" term="%22Jalil+M%22">Jalil M</searchLink>; College of Business and Economics, United Arab Emirates University, AlAin, Abu Dhabi, United Arab Emirates.<br /><searchLink fieldCode="AU" term="%22Anwar+E%22">Anwar E</searchLink>; Department of Internal Medicine, College of Medicine and Health Sciences, United Arab Emirates University, AlAin, Abu Dhabi, United Arab Emirates.<br /><searchLink fieldCode="AU" term="%22Bakoush+O%22">Bakoush O</searchLink>; Department of Internal Medicine, College of Medicine and Health Sciences, United Arab Emirates University, AlAin, Abu Dhabi, United Arab Emirates.
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%22101599504%22">JMIR research protocols</searchLink> [JMIR Res Protoc] 2026 Sep 22; Vol. 15, pp. e103220. <i>Date of Electronic Publication: </i>2026 Sep 22.
– Name: TypePub
  Label: Publication Type
  Group: TypPub
  Data: Journal Article; Comparative Study
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: TitleSource
  Label: Journal Info
  Group: Src
  Data: <i>Publisher: </i><searchLink fieldCode="PB" term="%22JMIR+Publications%22">JMIR Publications </searchLink><i>Country of Publication: </i>Canada <i>NLM ID: </i>101599504 <i>Publication Model: </i>Electronic <i>Cited Medium: </i>Internet <i>ISSN: </i>1929-0748 (Electronic) <i>Linking ISSN: </i><searchLink fieldCode="IS" term="%2219290748%22">19290748 </searchLink><i>NLM ISO Abbreviation: </i>JMIR Res Protoc <i>Subsets: </i>MEDLINE
– Name: PublisherInfo
  Label: Imprint Name(s)
  Group: PubInfo
  Data: <i>Original Publication</i>: Toronto : JMIR Publications
– Name: SubjectMESH
  Label: MeSH Terms
  Group: Su
  Data: <searchLink fieldCode="MM" term="%22Large+Language+Models%22">Large Language Models*</searchLink> <br /><searchLink fieldCode="MM" term="%22Clinical+Reasoning%22">Clinical Reasoning*</searchLink> <br /><searchLink fieldCode="MM" term="%22Thinking%22">Thinking*</searchLink><br /><searchLink fieldCode="MH" term="%22Humans%22">Humans</searchLink> ; <searchLink fieldCode="MH" term="%22Prospective+Studies%22">Prospective Studies</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Background: Most clinical reasoning evaluations in large language models (LLMs) score only the final answer, usually multiple-choice accuracy, which explains little about model reasoning. Two developments stress this gap: reasoning-optimized models are now common, and several expose an explicit extended thinking control. Whether that deliberation improves the reasoning process remains untested with a validated instrument.<br />Objective: This protocol introduces and aims to validate the Rapid Evaluation Assessment of Clinical Reasoning Tool (REACT)-AI, a Behaviorally Anchored Rating Scale (BARS) with 13 subdomains for process-oriented assessment of AI clinical reasoning, that is, the quality of the externalized reasoning rather than its faithfulness to the model's internal computation. It also builds a conflict-of-interest-controlled LLM-as-judge pipeline and tests whether extended thinking improves reasoning quality.<br />Methods: This 2-phase, prospective, comparative study has a within-model thinking-mode factor. Six flagship models are run in standard and extended thinking or reasoning modes, giving 12 conditions: 4 providers are toggled within the same model, and 2 pair a standard model with a dedicated reasoning model. In phase 1, 5 standardized urgent care vignettes are answered under all 12 conditions, 3 runs per condition (180 AI outputs), and by an independent expert clinician panel (n=5) and senior and junior medical students. Every output is scored by dual-blind human raters and, in parallel, by LLM judges, with no model judging its own family. Acceptance criteria for deploying the judge at scale are a weighted κ of at least 0.60 and an intraclass correlation coefficient above 0.75. In phase 2, the validated pipeline scores 3600 AI outputs. A self-correction turn and a paired Gulf English condition run alongside and are reported separately, with the AI metacognition assessment rubric and disinformation generation rate as secondary instruments.<br />Results: As of June 2026, ethics approval was obtained (ERSC_2025_6124), and the study is registered on the Open Science Framework under embargo. Responses to the 5 phase 1 vignettes were collected from senior and junior medical students between January and June 2026; 173 scripts were received and remain sealed, unopened, and unscored. No AI outputs have been generated, and the expert panel remains unrecruited. Following the June 1, 2026, model-version lock, AI generation and scoring will begin in September 2026, with phase 1 calibration through December 2026 and phase 2 from January to April 2027; results are expected in winter 2027. The study will report the validated instrument, judge calibration against human experts, self-preference bias by model family, and the effect of extended thinking on reflection and metacognition, including null findings.<br />Conclusions: REACT-AI and its judge pipeline are built to be reusable. They close the gap between accuracy benchmarks and genuine reasoning assessment and provide a template for studying reasoning mode effects.<br />Trial Registration: OSF Registries osf.io/jep5t; https://osf.io/jep5t.<br />International Registered Report Identifier (irrid): DERR1-10.2196/103220.<br /> (©Adnan Agha, Muhammad Jalil, Eram Anwar, Omran Bakoush. Originally published in JMIR Research Protocols (https://www.researchprotocols.org), 22.09.2026.)
– Name: Ref
  Label: References
  Group: RefInfo
  Data: Nat Med. 2025 Jan;31(1):60-69. (PMID: <searchLink fieldCode="PM" term="%2239779929%22">39779929)</searchLink><br />JAMA Netw Open. 2026 Apr 01;9(4):e264003. (PMID: <searchLink fieldCode="PM" term="%2241973425%22">41973425)</searchLink><br />Nat Med. 2026 Feb;32(2):616-623. (PMID: <searchLink fieldCode="PM" term="%2241652123%22">41652123)</searchLink><br />NPJ Digit Med. 2025 Oct 7;8(1):600. (PMID: <searchLink fieldCode="PM" term="%2241057566%22">41057566)</searchLink><br />Nat Med. 2025 Jun;31(6):1873-1881. (PMID: <searchLink fieldCode="PM" term="%2240195448%22">40195448)</searchLink><br />Cureus. 2025 Oct 14;17(10):e94563. (PMID: <searchLink fieldCode="PM" term="%2241246764%22">41246764)</searchLink><br />Nat Commun. 2025 Nov 6;16(1):9799. (PMID: <searchLink fieldCode="PM" term="%2241198657%22">41198657)</searchLink><br />BMJ Open. 2024 Jul 18;14(7):e087469. (PMID: <searchLink fieldCode="PM" term="%2239025818%22">39025818)</searchLink><br />Nature. 2023 Aug;620(7972):172-180. (PMID: <searchLink fieldCode="PM" term="%2237438534%22">37438534)</searchLink><br />JAMA Netw Open. 2025 Aug 1;8(8):e2526021. (PMID: <searchLink fieldCode="PM" term="%2240779272%22">40779272)</searchLink><br />Acad Med. 2019 Jun;94(6):902-912. (PMID: <searchLink fieldCode="PM" term="%2230720527%22">30720527)</searchLink><br />BMJ. 2020 Sep 9;370:m3210. (PMID: <searchLink fieldCode="PM" term="%2232907797%22">32907797)</searchLink><br />JAMA. 2018 Jun 12;319(22):2267-2268. (PMID: <searchLink fieldCode="PM" term="%2229800012%22">29800012)</searchLink><br />Nat Med. 2026 Mar;32(3):943-951. (PMID: <searchLink fieldCode="PM" term="%2241559415%22">41559415)</searchLink><br />JAMA Netw Open. 2024 Oct 1;7(10):e2440969. (PMID: <searchLink fieldCode="PM" term="%2239466245%22">39466245)</searchLink><br />Acad Med. 2020 Aug;95(8):1166-1171. (PMID: <searchLink fieldCode="PM" term="%2231577583%22">31577583)</searchLink><br />J Appl Psychol. 2009 Sep;94(5):1336-44. (PMID: <searchLink fieldCode="PM" term="%2219702375%22">19702375)</searchLink><br />Nat Med. 2020 Sep;26(9):1364-1374. (PMID: <searchLink fieldCode="PM" term="%2232908283%22">32908283)</searchLink><br />J Gen Intern Med. 2022 Jul;37(9):2224-2229. (PMID: <searchLink fieldCode="PM" term="%2235710662%22">35710662)</searchLink><br />J Oral Facial Pain Headache. 2026 Mar;40(2):140-150. (PMID: <searchLink fieldCode="PM" term="%2241914067%22">41914067)</searchLink>
– Name: SubjectMinor
  Label: Contributed Indexing
  Group:
  Data: <i>Keywords: </i>LLM-as-judge; behaviorally anchored rating scale; clinical decision support; clinical reasoning assessment; extended thinking; large language models; metacognition; process-oriented evaluation; reasoning models; self-preference bias
– Name: DateEntry
  Label: Entry Date(s)
  Group: Date
  Data: <i>Date Created: </i>20260922 <i>Date Completed: </i>20260922 <i>Latest Revision: </i>20261009
– Name: DateUpdate
  Label: Update Code
  Group: Date
  Data: 20261009
– Name: PubmedCentralID
  Label: PubMed Central ID
  Group: ID
  Data: PMC13643257
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.2196/103220
– Name: AN
  Label: PMID
  Group: ID
  Data: 42772746
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=cmedm&AN=42772746
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.2196/103220
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        StartPage: e103220
    Subjects:
      – SubjectFull: Humans
        Type: general
      – SubjectFull: Prospective Studies
        Type: general
      – SubjectFull: Large Language Models
        Type: general
      – SubjectFull: Clinical Reasoning
        Type: general
      – SubjectFull: Thinking
        Type: general
    Titles:
      – TitleFull: Process-Oriented, Behaviorally Anchored Assessment of Clinical Reasoning in Large Language Models and the Effect of Extended Thinking: Protocol for a Prospective, Multigroup, Comparative Study.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Agha A
      – PersonEntity:
          Name:
            NameFull: Jalil M
      – PersonEntity:
          Name:
            NameFull: Anwar E
      – PersonEntity:
          Name:
            NameFull: Bakoush O
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 22
              M: 09
              Text: 2026 Sep 22
              Type: published
              Y: 2026
          Identifiers:
            – Type: issn-electronic
              Value: 1929-0748
          Numbering:
            – Type: volume
              Value: 15
          Titles:
            – TitleFull: JMIR research protocols
              Type: main
ResultId 1