Academic Journal
Proof-of-concept study of a small language model chatbot for breast cancer decision support – a transparent, source-controlled, explainable and data-secure approach
| Title: | Proof-of-concept study of a small language model chatbot for breast cancer decision support – a transparent, source-controlled, explainable and data-secure approach |
|---|---|
| Authors: | Sebastian Griewing, Fabian Lechner, Niklas Gremke, Stefan Lukac, Wolfgang Janni, Markus Wallwiener, Uwe Wagner, Martin Hirsch, Sebastian Kuhn |
| Source: | J Cancer Res Clin Oncol |
| Publisher Information: | Springer Science and Business Media LLC, 2024. |
| Publication Year: | 2024 |
| Subject Terms: | ddc:610, Female [MeSH], Decision Support Systems, Clinical [MeSH], Humans [MeSH], Breast cancer, Artificial intelligence, Large language model, Practice Guidelines as Topic/standards [MeSH], Decision Support Techniques [MeSH], Breast Neoplasms [MeSH], Clinical oncology, Small language model, Research, Proof of Concept Study [MeSH], Practice Guidelines as Topic, Humans, Breast Neoplasms, Female, Decision Support Systems, Clinical, Proof of Concept Study, Decision Support Techniques |
| Description: | Purpose Large language models (LLM) show potential for decision support in breast cancer care. Their use in clinical care is currently prohibited by lack of control over sources used for decision-making, explainability of the decision-making process and health data security issues. Recent development of Small Language Models (SLM) is discussed to address these challenges. This preclinical proof-of-concept study tailors an open-source SLM to the German breast cancer guideline (BC-SLM) to evaluate initial clinical accuracy and technical functionality in a preclinical simulation. Methods A multidisciplinary tumor board (MTB) is used as the gold-standard to assess the initial clinical accuracy in terms of concordance of the BC-SLM with MTB and comparing it to two publicly available LLM, ChatGPT3.5 and 4. The study includes 20 fictional patient profiles and recommendations for 5 treatment modalities, resulting in 100 binary treatment recommendations (recommended or not recommended). Statistical evaluation includes concordance with MTB in % including Cohen’s Kappa statistic (κ). Technical functionality is assessed qualitatively in terms of local hosting, adherence to the guideline and information retrieval. Results The overall concordance amounts to 86% for BC-SLM (κ = 0.721, p p p Conclusion The tailored BC-SLM shows initial clinical accuracy and technical functionality, with concordance to the MTB that is comparable to publicly-available LLMs like ChatGPT4 and 3.5. This serves as a proof-of-concept for adapting a SLM to an oncological disease and its guideline to address prevailing issues with LLM by ensuring decision transparency, explainability, source control, and data security, which represents a necessary step towards clinical validation and safe use of language models in clinical oncology. |
| Document Type: | Article Other literature type |
| File Description: | application/pdf |
| Language: | English |
| ISSN: | 1432-1335 |
| DOI: | 10.1007/s00432-024-05964-3 |
| DOI: | 10.25673/117452 |
| Access URL: | https://pubmed.ncbi.nlm.nih.gov/39382778 https://repository.publisso.de/resource/frl:6493310 |
| Rights: | CC BY |
| Accession Number: | edsair.doi.dedup.....f545d1eb75b0f95b3be4041d8ed6804e |
| Database: | OpenAIRE |
| ISSN: | 14321335 |
|---|---|
| DOI: | 10.1007/s00432-024-05964-3 |