Academic Journal
View in EDS
Linked Full Text

ChatGPT Achieves Only Fair Agreement with ACFAS Expert Panelist Clinical Consensus Statements.

Bibliographic Details
Title: ChatGPT Achieves Only Fair Agreement with ACFAS Expert Panelist Clinical Consensus Statements.
Authors: Casciato DJ; Orlando VA Medical Center, Orlando, Florida., Calhoun J; Orlando VA Medical Center, Orlando, Florida.
Source: Foot & ankle specialist [Foot Ankle Spec] 2026 Oct; Vol. 19 (5), pp. 492-497. Date of Electronic Publication: 2025 Feb 22.
Publication Type: Journal Article
Language: English
Journal Info: Publisher: Sage Publications Country of Publication: United States NLM ID: 101473598 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 1938-7636 (Electronic) Linking ISSN: 19386400 NLM ISO Abbreviation: Foot Ankle Spec Subsets: MEDLINE
Imprint Name(s): Original Publication: Thousand Oaks, CA : Sage Publications
MeSH Terms: Consensus Statements as Topic* , Artificial Intelligence* , Clinical Decision-Making*, Humans ; Generative Artificial Intelligence ; Reproducibility of Results
Abstract: IntroductionAs artificial intelligence (AI) becomes increasingly integrated into medicine and surgery, its applications are expanding rapidly-from aiding clinical documentation to providing patient information. However, its role in medical decision-making remains uncertain. This study evaluates an AI language model's alignment with clinical consensus statements in foot and ankle surgery.MethodsClinical consensus statements from the American College of Foot and Ankle Surgeons (ACFAS; 2015-2022) were collected and rated by ChatGPT-o1 as being inappropriate, neither appropriate nor inappropriate, and appropriate. Ten repetitions of the statements were entered into ChatGPT-o1 in a random order, and the model was prompted to assign a corresponding rating. The AI-generated scores were compared to the expert panel's ratings, and intra-rater analysis was performed.ResultsThe analysis of 9 clinical consensus documents and 129 statements revealed an overall Cohen's kappa of 0.29 (95% CI: 0.12, 0.46), indicating fair alignment between expert panelists and ChatGPT. Overall, ankle arthritis and heel pain showed the highest concordance at 100%, while flatfoot exhibited the lowest agreement at 25%, reflecting variability between ChatGPT and expert panelists. Among the ChatGPT ratings, Cohen's kappa values ranged from 0.41 to 0.92, highlighting variability in internal reliability across topics.ConclusionChatGPT achieved overall fair agreement and demonstrated variable consistency when repetitively rating ACFAS expert panel clinical practice guidelines representing a variety of topics. These data reflect the need for further study of the causes, impacts, and solutions for this disparity between intelligence and human intelligence.Level of Evidence:Level IV: Retrospective cohort study.
Competing Interests: Declaration of Conflicting InterestsThe author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Contributed Indexing: Keywords: artificial intelligence; machine learning; medical informatics; natural language processing; surgical decision-making
Entry Date(s): Date Created: 20250222 Date Completed: 20260918 Latest Revision: 20260918
Update Code: 20260918
DOI: 10.1177/19386400251319567
PMID: 39985409
Database: MEDLINE
FullText Links:
  – Type: other
    Url: https://resolver.ebsco.com:443/public/rma-ftfapi/ejs/direct?AccessToken=4ADF88F2C8CB98D45154&Show=Object
Text:
  Availability: 0
Header DbId: cmedm
DbLabel: MEDLINE
An: 39985409
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: ChatGPT Achieves Only Fair Agreement with ACFAS Expert Panelist Clinical Consensus Statements.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AU" term="%22Casciato+DJ%22">Casciato DJ</searchLink>; Orlando VA Medical Center, Orlando, Florida.<br /><searchLink fieldCode="AU" term="%22Calhoun+J%22">Calhoun J</searchLink>; Orlando VA Medical Center, Orlando, Florida.
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%22101473598%22">Foot & ankle specialist</searchLink> [Foot Ankle Spec] 2026 Oct; Vol. 19 (5), pp. 492-497. <i>Date of Electronic Publication: </i>2025 Feb 22.
– Name: TypePub
  Label: Publication Type
  Group: TypPub
  Data: Journal Article
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: TitleSource
  Label: Journal Info
  Group: Src
  Data: <i>Publisher: </i><searchLink fieldCode="PB" term="%22Sage+Publications%22">Sage Publications </searchLink><i>Country of Publication: </i>United States <i>NLM ID: </i>101473598 <i>Publication Model: </i>Print-Electronic <i>Cited Medium: </i>Internet <i>ISSN: </i>1938-7636 (Electronic) <i>Linking ISSN: </i><searchLink fieldCode="IS" term="%2219386400%22">19386400 </searchLink><i>NLM ISO Abbreviation: </i>Foot Ankle Spec <i>Subsets: </i>MEDLINE
– Name: PublisherInfo
  Label: Imprint Name(s)
  Group: PubInfo
  Data: <i>Original Publication</i>: Thousand Oaks, CA : Sage Publications
– Name: SubjectMESH
  Label: MeSH Terms
  Group: Su
  Data: <searchLink fieldCode="MM" term="%22Consensus+Statements+as+Topic%22">Consensus Statements as Topic*</searchLink> <br /><searchLink fieldCode="MM" term="%22Artificial+Intelligence%22">Artificial Intelligence*</searchLink> <br /><searchLink fieldCode="MM" term="%22Clinical+Decision-Making%22">Clinical Decision-Making*</searchLink><br /><searchLink fieldCode="MH" term="%22Humans%22">Humans</searchLink> ; <searchLink fieldCode="MH" term="%22Generative+Artificial+Intelligence%22">Generative Artificial Intelligence</searchLink> ; <searchLink fieldCode="MH" term="%22Reproducibility+of+Results%22">Reproducibility of Results</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: IntroductionAs artificial intelligence (AI) becomes increasingly integrated into medicine and surgery, its applications are expanding rapidly-from aiding clinical documentation to providing patient information. However, its role in medical decision-making remains uncertain. This study evaluates an AI language model's alignment with clinical consensus statements in foot and ankle surgery.MethodsClinical consensus statements from the American College of Foot and Ankle Surgeons (ACFAS; 2015-2022) were collected and rated by ChatGPT-o1 as being inappropriate, neither appropriate nor inappropriate, and appropriate. Ten repetitions of the statements were entered into ChatGPT-o1 in a random order, and the model was prompted to assign a corresponding rating. The AI-generated scores were compared to the expert panel's ratings, and intra-rater analysis was performed.ResultsThe analysis of 9 clinical consensus documents and 129 statements revealed an overall Cohen's kappa of 0.29 (95% CI: 0.12, 0.46), indicating fair alignment between expert panelists and ChatGPT. Overall, ankle arthritis and heel pain showed the highest concordance at 100%, while flatfoot exhibited the lowest agreement at 25%, reflecting variability between ChatGPT and expert panelists. Among the ChatGPT ratings, Cohen's kappa values ranged from 0.41 to 0.92, highlighting variability in internal reliability across topics.ConclusionChatGPT achieved overall fair agreement and demonstrated variable consistency when repetitively rating ACFAS expert panel clinical practice guidelines representing a variety of topics. These data reflect the need for further study of the causes, impacts, and solutions for this disparity between intelligence and human intelligence.Level of Evidence:Level IV: Retrospective cohort study.
– Name: Abstract
  Label: Competing Interests
  Group: Ab
  Data: Declaration of Conflicting InterestsThe author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
– Name: SubjectMinor
  Label: Contributed Indexing
  Group:
  Data: <i>Keywords: </i>artificial intelligence; machine learning; medical informatics; natural language processing; surgical decision-making
– Name: DateEntry
  Label: Entry Date(s)
  Group: Date
  Data: <i>Date Created: </i>20250222 <i>Date Completed: </i>20260918 <i>Latest Revision: </i>20260918
– Name: DateUpdate
  Label: Update Code
  Group: Date
  Data: 20260918
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1177/19386400251319567
– Name: AN
  Label: PMID
  Group: ID
  Data: 39985409
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=cmedm&AN=39985409
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1177/19386400251319567
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        StartPage: 492
    Subjects:
      – SubjectFull: Humans
        Type: general
      – SubjectFull: Generative Artificial Intelligence
        Type: general
      – SubjectFull: Reproducibility of Results
        Type: general
      – SubjectFull: Consensus Statements as Topic
        Type: general
      – SubjectFull: Artificial Intelligence
        Type: general
      – SubjectFull: Clinical Decision-Making
        Type: general
    Titles:
      – TitleFull: ChatGPT Achieves Only Fair Agreement with ACFAS Expert Panelist Clinical Consensus Statements.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Casciato DJ
      – PersonEntity:
          Name:
            NameFull: Calhoun J
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 10
              Text: 2026 Oct
              Type: published
              Y: 2026
          Identifiers:
            – Type: issn-electronic
              Value: 1938-7636
          Numbering:
            – Type: volume
              Value: 19
            – Type: issue
              Value: 5
          Titles:
            – TitleFull: Foot & ankle specialist
              Type: main
ResultId 1