Academic Journal
Process-Oriented, Behaviorally Anchored Assessment of Clinical Reasoning in Large Language Models and the Effect of Extended Thinking: Protocol for a Prospective, Multigroup, Comparative Study.
| Τίτλος: | Process-Oriented, Behaviorally Anchored Assessment of Clinical Reasoning in Large Language Models and the Effect of Extended Thinking: Protocol for a Prospective, Multigroup, Comparative Study. |
|---|---|
| Συγγραφείς: | Agha A; Department of Internal Medicine, College of Medicine and Health Sciences, United Arab Emirates University, AlAin, Abu Dhabi, United Arab Emirates., Jalil M; College of Business and Economics, United Arab Emirates University, AlAin, Abu Dhabi, United Arab Emirates., Anwar E; Department of Internal Medicine, College of Medicine and Health Sciences, United Arab Emirates University, AlAin, Abu Dhabi, United Arab Emirates., Bakoush O; Department of Internal Medicine, College of Medicine and Health Sciences, United Arab Emirates University, AlAin, Abu Dhabi, United Arab Emirates. |
| Πηγή: | JMIR research protocols [JMIR Res Protoc] 2026 Sep 22; Vol. 15, pp. e103220. Date of Electronic Publication: 2026 Sep 22. |
| Τύπος έκδοσης: | Journal Article; Comparative Study |
| Γλώσσα: | English |
| Στοιχεία περιοδικού: | Publisher: JMIR Publications Country of Publication: Canada NLM ID: 101599504 Publication Model: Electronic Cited Medium: Internet ISSN: 1929-0748 (Electronic) Linking ISSN: 19290748 NLM ISO Abbreviation: JMIR Res Protoc Subsets: MEDLINE |
| Imprint Name(s): | Original Publication: Toronto : JMIR Publications |
| Ιατρικοί όροι (MeSH): | Large Language Models* , Clinical Reasoning* , Thinking*, Humans ; Prospective Studies |
| Περίληψη: | Background: Most clinical reasoning evaluations in large language models (LLMs) score only the final answer, usually multiple-choice accuracy, which explains little about model reasoning. Two developments stress this gap: reasoning-optimized models are now common, and several expose an explicit extended thinking control. Whether that deliberation improves the reasoning process remains untested with a validated instrument. Objective: This protocol introduces and aims to validate the Rapid Evaluation Assessment of Clinical Reasoning Tool (REACT)-AI, a Behaviorally Anchored Rating Scale (BARS) with 13 subdomains for process-oriented assessment of AI clinical reasoning, that is, the quality of the externalized reasoning rather than its faithfulness to the model's internal computation. It also builds a conflict-of-interest-controlled LLM-as-judge pipeline and tests whether extended thinking improves reasoning quality. Methods: This 2-phase, prospective, comparative study has a within-model thinking-mode factor. Six flagship models are run in standard and extended thinking or reasoning modes, giving 12 conditions: 4 providers are toggled within the same model, and 2 pair a standard model with a dedicated reasoning model. In phase 1, 5 standardized urgent care vignettes are answered under all 12 conditions, 3 runs per condition (180 AI outputs), and by an independent expert clinician panel (n=5) and senior and junior medical students. Every output is scored by dual-blind human raters and, in parallel, by LLM judges, with no model judging its own family. Acceptance criteria for deploying the judge at scale are a weighted κ of at least 0.60 and an intraclass correlation coefficient above 0.75. In phase 2, the validated pipeline scores 3600 AI outputs. A self-correction turn and a paired Gulf English condition run alongside and are reported separately, with the AI metacognition assessment rubric and disinformation generation rate as secondary instruments. Results: As of June 2026, ethics approval was obtained (ERSC_2025_6124), and the study is registered on the Open Science Framework under embargo. Responses to the 5 phase 1 vignettes were collected from senior and junior medical students between January and June 2026; 173 scripts were received and remain sealed, unopened, and unscored. No AI outputs have been generated, and the expert panel remains unrecruited. Following the June 1, 2026, model-version lock, AI generation and scoring will begin in September 2026, with phase 1 calibration through December 2026 and phase 2 from January to April 2027; results are expected in winter 2027. The study will report the validated instrument, judge calibration against human experts, self-preference bias by model family, and the effect of extended thinking on reflection and metacognition, including null findings. Conclusions: REACT-AI and its judge pipeline are built to be reusable. They close the gap between accuracy benchmarks and genuine reasoning assessment and provide a template for studying reasoning mode effects. Trial Registration: OSF Registries osf.io/jep5t; https://osf.io/jep5t. International Registered Report Identifier (irrid): DERR1-10.2196/103220. (©Adnan Agha, Muhammad Jalil, Eram Anwar, Omran Bakoush. Originally published in JMIR Research Protocols (https://www.researchprotocols.org), 22.09.2026.) |
| References: | Nat Med. 2025 Jan;31(1):60-69. (PMID: 39779929) JAMA Netw Open. 2026 Apr 01;9(4):e264003. (PMID: 41973425) Nat Med. 2026 Feb;32(2):616-623. (PMID: 41652123) NPJ Digit Med. 2025 Oct 7;8(1):600. (PMID: 41057566) Nat Med. 2025 Jun;31(6):1873-1881. (PMID: 40195448) Cureus. 2025 Oct 14;17(10):e94563. (PMID: 41246764) Nat Commun. 2025 Nov 6;16(1):9799. (PMID: 41198657) BMJ Open. 2024 Jul 18;14(7):e087469. (PMID: 39025818) Nature. 2023 Aug;620(7972):172-180. (PMID: 37438534) JAMA Netw Open. 2025 Aug 1;8(8):e2526021. (PMID: 40779272) Acad Med. 2019 Jun;94(6):902-912. (PMID: 30720527) BMJ. 2020 Sep 9;370:m3210. (PMID: 32907797) JAMA. 2018 Jun 12;319(22):2267-2268. (PMID: 29800012) Nat Med. 2026 Mar;32(3):943-951. (PMID: 41559415) JAMA Netw Open. 2024 Oct 1;7(10):e2440969. (PMID: 39466245) Acad Med. 2020 Aug;95(8):1166-1171. (PMID: 31577583) J Appl Psychol. 2009 Sep;94(5):1336-44. (PMID: 19702375) Nat Med. 2020 Sep;26(9):1364-1374. (PMID: 32908283) J Gen Intern Med. 2022 Jul;37(9):2224-2229. (PMID: 35710662) J Oral Facial Pain Headache. 2026 Mar;40(2):140-150. (PMID: 41914067) |
| Contributed Indexing: | Keywords: LLM-as-judge; behaviorally anchored rating scale; clinical decision support; clinical reasoning assessment; extended thinking; large language models; metacognition; process-oriented evaluation; reasoning models; self-preference bias |
| Entry Date(s): | Date Created: 20260922 Date Completed: 20260922 Latest Revision: 20261008 |
| Update Code: | 20261008 |
| DOI: | 10.2196/103220 |
| PMID: | 42772746 |
| Βάση Δεδομένων: | MEDLINE |
καταχωρήστε σχόλιο πρώτοι!