Academic Journal
ChatGPT Achieves Only Fair Agreement with ACFAS Expert Panelist Clinical Consensus Statements.
| Τίτλος: | ChatGPT Achieves Only Fair Agreement with ACFAS Expert Panelist Clinical Consensus Statements. |
|---|---|
| Συγγραφείς: | Casciato DJ; Orlando VA Medical Center, Orlando, Florida., Calhoun J; Orlando VA Medical Center, Orlando, Florida. |
| Πηγή: | Foot & ankle specialist [Foot Ankle Spec] 2026 Oct; Vol. 19 (5), pp. 492-497. Date of Electronic Publication: 2025 Feb 22. |
| Τύπος έκδοσης: | Journal Article |
| Γλώσσα: | English |
| Στοιχεία περιοδικού: | Publisher: Sage Publications Country of Publication: United States NLM ID: 101473598 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 1938-7636 (Electronic) Linking ISSN: 19386400 NLM ISO Abbreviation: Foot Ankle Spec Subsets: MEDLINE |
| Imprint Name(s): | Original Publication: Thousand Oaks, CA : Sage Publications |
| Ιατρικοί όροι (MeSH): | Consensus Statements as Topic* , Artificial Intelligence* , Clinical Decision-Making*, Humans ; Generative Artificial Intelligence ; Reproducibility of Results |
| Περίληψη: | IntroductionAs artificial intelligence (AI) becomes increasingly integrated into medicine and surgery, its applications are expanding rapidly-from aiding clinical documentation to providing patient information. However, its role in medical decision-making remains uncertain. This study evaluates an AI language model's alignment with clinical consensus statements in foot and ankle surgery.MethodsClinical consensus statements from the American College of Foot and Ankle Surgeons (ACFAS; 2015-2022) were collected and rated by ChatGPT-o1 as being inappropriate, neither appropriate nor inappropriate, and appropriate. Ten repetitions of the statements were entered into ChatGPT-o1 in a random order, and the model was prompted to assign a corresponding rating. The AI-generated scores were compared to the expert panel's ratings, and intra-rater analysis was performed.ResultsThe analysis of 9 clinical consensus documents and 129 statements revealed an overall Cohen's kappa of 0.29 (95% CI: 0.12, 0.46), indicating fair alignment between expert panelists and ChatGPT. Overall, ankle arthritis and heel pain showed the highest concordance at 100%, while flatfoot exhibited the lowest agreement at 25%, reflecting variability between ChatGPT and expert panelists. Among the ChatGPT ratings, Cohen's kappa values ranged from 0.41 to 0.92, highlighting variability in internal reliability across topics.ConclusionChatGPT achieved overall fair agreement and demonstrated variable consistency when repetitively rating ACFAS expert panel clinical practice guidelines representing a variety of topics. These data reflect the need for further study of the causes, impacts, and solutions for this disparity between intelligence and human intelligence.Level of Evidence:Level IV: Retrospective cohort study. |
| Competing Interests: | Declaration of Conflicting InterestsThe author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. |
| Contributed Indexing: | Keywords: artificial intelligence; machine learning; medical informatics; natural language processing; surgical decision-making |
| Entry Date(s): | Date Created: 20250222 Date Completed: 20260918 Latest Revision: 20260918 |
| Update Code: | 20260918 |
| DOI: | 10.1177/19386400251319567 |
| PMID: | 39985409 |
| Βάση Δεδομένων: | MEDLINE |
καταχωρήστε σχόλιο πρώτοι!