Academic Journal
Large Language Model Chatbot Responses to Cancer Survivorship Questions in Hong Kong: Bilingual Evaluation and Prompt Optimization Study.
| Τίτλος: | Large Language Model Chatbot Responses to Cancer Survivorship Questions in Hong Kong: Bilingual Evaluation and Prompt Optimization Study. |
|---|---|
| Συγγραφείς: | An H; Department of Family Medicine and Primary Care, School of Clinical Medicine, Li Ka Shing Faculty of Medicine, University of Hong Kong, Hong Kong, SAR, China (Hong Kong)., Huang W; Department of Family Medicine and Primary Care, School of Clinical Medicine, Li Ka Shing Faculty of Medicine, University of Hong Kong, Hong Kong, SAR, China (Hong Kong)., Cheung RTM; Independent researcher, Hong Kong, SAR, China (Hong Kong)., Du Q; Department of Clinical Oncology, Pamela Youde Nethersole Eastern Hospital, Hong Kong, SAR, China (Hong Kong)., Na R; Division of Urology, Department of Surgery, School of Clinical Medicine, Li Ka Shing Faculty of Medicine, University of Hong Kong, Hong Kong, SAR, China (Hong Kong)., Wong DKK; Department of Family Medicine and Primary Care, School of Clinical Medicine, Li Ka Shing Faculty of Medicine, University of Hong Kong, Hong Kong, SAR, China (Hong Kong).; Department of Family Medicine, University of Hong Kong - Shenzhen Hospital, Shenzhen, Guangdong, China. |
| Πηγή: | Journal of medical Internet research [J Med Internet Res] 2026 Oct 08; Vol. 28, pp. e105778. Date of Electronic Publication: 2026 Oct 08. |
| Τύπος έκδοσης: | Journal Article |
| Γλώσσα: | English |
| Στοιχεία περιοδικού: | Publisher: JMIR Publications Country of Publication: Canada NLM ID: 100959882 Publication Model: Electronic Cited Medium: Internet ISSN: 1438-8871 (Electronic) Linking ISSN: 14388871 NLM ISO Abbreviation: J Med Internet Res Subsets: MEDLINE |
| Imprint Name(s): | Publication: <2011- > : Toronto : JMIR Publications Original Publication: [Pittsburgh, PA? : s.n., 1999- |
| Ιατρικοί όροι (MeSH): | Large Language Models* , Multilingualism* , Neoplasms* , Cancer Survivors*, Hong Kong ; Humans ; Comprehension |
| Περίληψη: | Background: As the population of survivors of cancer grows and new information technologies become widespread in Hong Kong, survivors of cancer increasingly seek ongoing care and support information from large language models (LLMs). While these tools provide immediate conversational responses, they carry substantial risks of generating inaccurate, unsafe, or generic medical advice. Evaluating LLM response performance is particularly critical in Hong Kong's bilingual health care context. Objective: This study aimed to evaluate the clinical utility, safety, understandability, and readability of 4 popular commercial LLMs answering cancer survivorship questions in English and Traditional Chinese and to explore whether a structured prompting strategy improves response performance. Methods: We conducted a 2-phase evaluation study combining quantitative scoring with qualitative classification of unsafe responses. In phase 1, 25 expert-curated questions spanning 5 survivorship domains were input into 4 LLMs: Google Gemini 3.1 Pro, HK Chat-0.6.2, DeepSeek-V3.2, and Kimi-K2.5. A Delphi expert panel evaluated the baseline responses for clinical utility using a 5-point scale and assessed safety via binary categorization. Understandability was measured using the Patient Education Materials Assessment Tool (PEMAT). Readability was assessed via the Flesch-Kincaid Grade Level for English and lexical richness for Traditional Chinese. In phase 2, experts developed an optimized context, role, audience, format, task, and tone (CRAFT) prompt, and the resulting LLM responses were compared with baseline outputs in an exploratory within-sample analysis using paired statistical tests. Results: Baseline evaluations identified Google Gemini 3.1 Pro as the most clinically useful model in English (mean 3.99/5, SD 0.16) and Traditional Chinese (mean 4.21/5, SD 0.20). Unconstrained models demonstrated language-dependent safety vulnerabilities: English errors (up to 24%) centered on service mismatches and definitive interpretations, whereas Traditional Chinese errors (up to 20%) involved prescriptive disease management and unverified adjunctive therapies. Within each language, the models producing the most readable text were not those achieving the highest clinical utility; this relationship was not formally tested. Applying the CRAFT prompt to Gemini 3.1 Pro significantly improved clinical utility across both languages (English: mean 4.68, SD 0.18, P<.001; Traditional Chinese: 4.66, SD 0.23, P<.001). After prompting, the 2 unsafe English baseline responses were reclassified as safe (English 23/25 to 25/25 safe; Traditional Chinese 25/25 at both time points). This transition was not statistically significant on a post hoc exact McNemar test (P=.50) and is reported as descriptive. Conclusions: LLMs exhibited language-divergent clinical safety vulnerabilities when answering cancer survivorship queries, and within each language, the most readable models were not the most clinically useful. In an exploratory within-sample comparison, expert-designed CRAFT prompting improved clinical utility and corrected the unsafe outputs seen at baseline without reducing understandability. Safe deployment of LLMs in multilingual oncology contexts requires strict validation combined with structured, role-restrictive constraints. (©Haoyu An, Wanshu Huang, Rachel Tsoi Man Cheung, Qijun Du, Rong Na, David Ka Ki Wong. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 08.10.2026.) |
| References: | Oncologist. 2025 Apr 4;30(4):. (PMID: 40285677) NPJ Digit Med. 2026 Feb 13;9(1):. (PMID: 41688533) J Med Internet Res. 2025 Oct 7;27:e78625. (PMID: 41057043) J Adv Nurs. 2000 Oct;32(4):1008-15. (PMID: 11095242) BMC Med Inform Decis Mak. 2025 Sep 29;25(1):350. (PMID: 41023674) Eur J Pediatr. 2025 Jul 18;184(8):491. (PMID: 40679644) JMIR Cancer. 2026 Mar 26;12:e84234. (PMID: 41886611) CA Cancer J Clin. 2025 Jul-Aug;75(4):308-340. (PMID: 40445120) JAMA Intern Med. 2023 Jun 1;183(6):589-596. (PMID: 37115527) Med Teach. 2005 Nov;27(7):639-43. (PMID: 16332558) BMC Med. 2025 Aug 1;23(1):447. (PMID: 40745595) JNCI Cancer Spectr. 2023 Mar 1;7(2):. (PMID: 36808255) Patient Educ Couns. 2014 Sep;96(3):395-403. (PMID: 24973195) J Natl Cancer Inst. 2019 Nov 1;111(11):1120-1130. (PMID: 31095326) Arch Intern Med. 2005 Dec 12-26;165(22):2618-24. (PMID: 16344419) PLOS Digit Health. 2023 Feb 9;2(2):e0000198. (PMID: 36812645) Musculoskelet Sci Pract. 2025 Apr;76:103275. (PMID: 39899928) J Med Internet Res. 2023 Oct 4;25:e50638. (PMID: 37792434) J Med Internet Res. 2025 Jul 22;27:e73226. (PMID: 40694436) |
| Contributed Indexing: | Keywords: Hong Kong; LLM; chatbots; health education; large language model; prompt engineering; survivors of cancer |
| Entry Date(s): | Date Created: 20261008 Date Completed: 20261008 Latest Revision: 20261025 |
| Update Code: | 20261025 |
| PubMed Central ID: | PMC13691464 |
| DOI: | 10.2196/105778 |
| PMID: | 42849027 |
| Βάση Δεδομένων: | MEDLINE |
| ISSN: | 1438-8871 |
|---|---|
| DOI: | 10.2196/105778 |