Patient-Facing AI System for Symptom Guidance Using Simulated Encounters and Physician Review: Analytical Validation Study.

Λεπτομέρειες βιβλιογραφικής εγγραφής
Τίτλος:	Patient-Facing AI System for Symptom Guidance Using Simulated Encounters and Physician Review: Analytical Validation Study.
Συγγραφείς:	Patwardhan A; Verily Life Sciences LLC (United States), Dallas, TX, United States., Verma N; Verily Life Sciences LLC (United States), Dallas, TX, United States., Barman P; Verily Life Sciences LLC (United States), Dallas, TX, United States., Mao B; Verily Life Sciences LLC (United States), Dallas, TX, United States., Li N; Verily Life Sciences LLC (United States), Dallas, TX, United States., Varghese P; Verily Life Sciences LLC (United States), Dallas, TX, United States., Peri V; Verily Life Sciences LLC (United States), Dallas, TX, United States., Lehmann LS; Verily Life Sciences LLC (United States), Dallas, TX, United States.
Πηγή:	JMIR formative research [JMIR Form Res] 2026 Aug 19; Vol. 10, pp. e103419. Date of Electronic Publication: 2026 Aug 19.
Τύπος έκδοσης:	Journal Article; Validation Study; Research Support, Non-U.S. Gov't
Γλώσσα:	English
Στοιχεία περιοδικού:	Publisher: JMIR Publications Country of Publication: Canada NLM ID: 101726394 Publication Model: Electronic Cited Medium: Internet ISSN: 2561-326X (Electronic) Linking ISSN: 2561326X NLM ISO Abbreviation: JMIR Form Res Subsets: MEDLINE
Imprint Name(s):	Original Publication: Toronto, ON, Canada : JMIR Publications, [2017]-
Ιατρικοί όροι (MeSH):	Artificial Intelligence/standards , Symptom Assessment/methods , Physicians*/statistics & numerical data, Humans ; Large Language Models ; Prospective Studies ; Patient Simulation ; Female
Περίληψη:	Background: Rapid advances in large language models (LLMs) have expanded interest in patient-facing health applications that support symptom assessment and care-seeking decisions. Although AI-enabled symptom-guidance tools could improve patient navigation and recognition of clinically serious conditions, inappropriate recommendations may result in missed needed care or unnecessary escalation. Structured predeployment evaluation is therefore needed before prospective clinical use. Objective: This formative predeployment study aimed to analytically validate a locked configuration of the Personal Health Assistant (PHA; Verily Health), an LLM-enabled patient-facing symptom-guidance tool, under controlled conditions using simulated patient encounters and expert physician review. Methods: We conducted a prospective predeployment analytical validation study using synthetic patient data. The analysis set included 772 synthetic cases generated using published telephone-triage protocols and persona-based, LLM-assisted case generation. Each case included a clinical vignette, structured medical history information, and a simulated multiturn patient conversation. All cases were reviewed by a panel of 5 independent physicians without access to PHA outputs in order to provide level-of-care recommendations. Case-level ground truth was derived from aggregated physician ratings using prespecified plurality and tie-handling rules. Urgent undertriage, nonurgent undertriage, and overtriage were evaluated against prespecified performance criteria. Secondary analyses assessed the clinical appropriateness of PHA-generated possible conditions and escalation, self-care, and laboratory-testing recommendations. Results: The final analysis set contained 772 cases, including 2 with unevaluable ground truth that were excluded from primary Endpoint denominators. Urgent undertriage was identified in 28 of 406 cases (6.9%, 95% CI 4.6%-9.8%), nonurgent undertriage in 56 of 305 (18.4%, 95% CI 14.2%-23.2%), and overtriage in 40 of 364 (11.0%, 95% CI 8.0%-14.7%). Overtriage met the prespecified criterion of less than 30%, whereas urgent and nonurgent undertriage did not meet their respective criteria of less than 5% and less than 15%, respectively. Secondary findings were generally supportive of the clinical appropriateness of the additional PHA outputs. Conclusions: PHA met the prespecified overtriage criterion but did not meet the urgent or nonurgent undertriage criteria. The findings identified undertriage as the principal performance gap requiring further product refinement. Because the study used synthetic cases, simulated conversations, and enriched urgency categories under controlled conditions, the results may not reflect performance in real-world populations or deployment settings. Prospective evaluation with actual users in the intended-use population is needed before broader clinical use. (©Anil Patwardhan, Nishant Verma, Poulami Barman, Bingyu Mao, Nan Li, Paul Varghese, Vidya Peri, Lisa Soleymani Lehmann. Originally published in JMIR Formative Research (https://formative.jmir.org), 19.08.2026.)
References:	Nat Med. 2026 May;32(5):1671-1675. (PMID: 41731097) JMIR Mhealth Uhealth. 2022 Mar 28;10(3):e32340. (PMID: 35343909) JMIR AI. 2024 Apr 29;3:e46875. (PMID: 38875676) BMJ Open. 2020 Dec 16;10(12):e040269. (PMID: 33328258) Lancet. 2018 Nov 24;392(10161):2263-2264. (PMID: 30413281) Commun Med (Lond). 2026 Feb 25;6(1):. (PMID: 41735505) JMIR Hum Factors. 2026 Jan 16;13:e76168. (PMID: 41544248) NPJ Digit Med. 2025 Mar 25;8(1):178. (PMID: 40133390) J Med Internet Res. 2022 May 10;24(5):e31810. (PMID: 35536633) Mayo Clin Proc Digit Health. 2025 Jun 10;3(3):100242. (PMID: 40686623) BMC Fam Pract. 2020 May 9;21(1):84. (PMID: 32386511) NPJ Digit Med. 2026 Feb 6;9(1):. (PMID: 41652180) NPJ Digit Med. 2022 Aug 17;5(1):118. (PMID: 35977992) BMJ. 2015 Jul 08;351:h3480. (PMID: 26157077)
Contributed Indexing:	Keywords: analytical validation; digital health; large language model; overtriage; remote care; simulation; symptom checker; symptom guidance; triage; undertriage
Entry Date(s):	Date Created: 20260819 Date Completed: 20260819 Latest Revision: 20260908
Update Code:	20260908
PubMed Central ID:	PMC13539170
DOI:	10.2196/103419
PMID:	42615486
Βάση Δεδομένων:	MEDLINE

Full Text Finder

Περιγραφή
ISSN:	2561-326X
DOI:	10.2196/103419