Academic Journal
Προβολή σε EDS
Συνδεδεμένο Πλήρες Κείμενο

Performance of large language models on the Japanese cardiovascular surgery board examination: a comparative analysis of eight contemporary AI models with educational implications.

Λεπτομέρειες βιβλιογραφικής εγγραφής
Τίτλος: Performance of large language models on the Japanese cardiovascular surgery board examination: a comparative analysis of eight contemporary AI models with educational implications.
Συγγραφείς: Kasa K; Division of Vascular Surgery, Department of Surgery, The Jikei University School of Medicine, Tokyo, Japan.; Division of Regenerative Medicine, Research Center for Medical Sciences, The Jikei University School of Medicine, Tokyo, Japan., Yoshimaru D; Division of Regenerative Medicine, Research Center for Medical Sciences, The Jikei University School of Medicine, Tokyo, Japan., Ohta H; Division of Regenerative Medicine, Research Center for Medical Sciences, The Jikei University School of Medicine, Tokyo, Japan. hiroojikei@gmail.com., Ohki T; Division of Vascular Surgery, Department of Surgery, The Jikei University School of Medicine, Tokyo, Japan., Okano HJ; Division of Regenerative Medicine, Research Center for Medical Sciences, The Jikei University School of Medicine, Tokyo, Japan. hjokano@jikei.ac.jp.
Πηγή: General thoracic and cardiovascular surgery [Gen Thorac Cardiovasc Surg] 2026 Oct; Vol. 74 (10), pp. 950-962. Date of Electronic Publication: 2026 Apr 27.
Τύπος έκδοσης: Journal Article; Comparative Study
Γλώσσα: English
Στοιχεία περιοδικού: Publisher: Springer Japan Country of Publication: Japan NLM ID: 101303952 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 1863-6713 (Electronic) Linking ISSN: 18636705 NLM ISO Abbreviation: Gen Thorac Cardiovasc Surg Subsets: MEDLINE
Imprint Name(s): Original Publication: Tokyo : Springer Japan
Ιατρικοί όροι (MeSH): Cardiovascular Surgical Procedures*/education , Large Language Models* , Specialty Boards* , Certification* , Educational Measurement* , Education, Medical, Graduate*, Humans ; Japan ; Generative Artificial Intelligence ; Clinical Competence ; East Asian People
Περίληψη: Objective: Large language models (LLMs) have shown strong performance on medical licensing and specialty examinations, but their utility in cardiovascular surgery certification and education remains unknown.
Methods: We evaluated eight LLMs (GPT-5, GPT-4o [OpenAI], Gemini-2.5Pro, Gemini-2.0, Gemma-3 [12B] [Google DeepMind], Claude-4, Claude-3 [Anthropic], and Llama-4 [Scout] [Meta AI]) using their official application programming interfaces as of September 2025. Examination items were obtained from the Japanese Cardiovascular Surgery Board (2021-2024; 523 questions). Texts were extracted from PDFs, images converted to JPEGs, and each question presented with a standardized Japanese prompt. Models produced three responses per item; final answers were determined by majority voting. Accuracy with 95% confidence intervals was calculated, and pairwise comparisons performed using McNemar's test.
Results: Across 523 items, GPT-5 achieved the highest accuracy (87.4%), followed by Gemini-2.5Pro (85.7%); their performance did not differ significantly. Claude-4 ranked third (80.3%), exceeding the passing threshold in some years. GPT-4o (65.6%) and Gemini-2.0 (58.5%) showed moderate accuracy, whereas Claude-3 (36.9%), Gemma-3 (40.9%), and Llama-4 (52.2%) scored lower. Pairwise testing confirmed a clear stratification: high (GPT-5, Gemini-2.5Pro), upper-intermediate (Claude-4), mid (GPT-4o, Gemini-2.0), and low (others). All models declined with image-based items, with top models reduced to around 70%. Accuracy remained stable across years, with GPT-5, Gemini-2.5Pro, and occasionally Claude-4, surpassing the pass threshold.
Conclusions: Successive model generations, particularly GPT-5 and Gemini-2.5Pro, consistently achieved passing-level accuracy. These findings highlight substantial gains through model evolution and underscore the potential of LLMs as supplementary tools for specialty education, despite persistent limitations in image-based reasoning.
(© 2026. The Author(s), under exclusive licence to The Japanese Association for Thoracic Surgery.)
Competing Interests: Declarations. Conflict of interest: The authors have no conflict of interest regarding this article. Ethical approval: This study utilized publicly available examination materials with explicit permission from the Japanese Board of Cardiovascular Surgery. The study did not involve human subjects or patient data; therefore, institutional review board approval was not required.
References: JMIR Med Educ. 2025 Jul 30;11:e69313. (PMID: 40737609)
JAMA. 2018 Apr 3;319(13):1317-1318. (PMID: 29532063)
Commun Med (Lond). 2023 Oct 10;3(1):141. (PMID: 37816837)
JAMA Netw Open. 2024 Jun 3;7(6):e2417641. (PMID: 38888919)
J Surg Educ. 2025 Apr;82(4):103442. (PMID: 39923296)
Lancet Digit Health. 2024 Oct;6(10):e767-e771. (PMID: 39214760)
BMC Med Educ. 2024 Nov 26;24(1):1372. (PMID: 39593041)
Nature. 2023 Aug;620(7972):172-180. (PMID: 37438534)
JMIR Med Educ. 2024 Dec 24;10:e63129. (PMID: 39718557)
JMIR Med Educ. 2023 Feb 8;9:e45312. (PMID: 36753318)
PLOS Digit Health. 2023 Feb 9;2(2):e0000198. (PMID: 36812645)
Jpn J Radiol. 2024 Aug;42(8):918-926. (PMID: 38733472)
Neurosurgery. 2023 Dec 1;93(6):1353-1365. (PMID: 37581444)
PLOS Digit Health. 2024 Jan 23;3(1):e0000433. (PMID: 38261580)
JAMA Netw Open. 2023 Dec 1;6(12):e2346721. (PMID: 38060223)
Mayo Clin Proc Digit Health. 2023 Jul 07;1(3):309-310. (PMID: 40206625)
Nat Med. 2025 Mar;31(3):943-950. (PMID: 39779926)
Cureus. 2024 Mar 18;16(3):e56402. (PMID: 38633935)
Grant Information: JP24K11980 JSPS KAKENHI
Contributed Indexing: Keywords: Artificial intelligence; Large language models; Vision language models
Entry Date(s): Date Created: 20260427 Date Completed: 20261002 Latest Revision: 20261008
Update Code: 20261008
DOI: 10.1007/s11748-026-02304-9
PMID: 42043642
Βάση Δεδομένων: MEDLINE
FullText Links:
  – Type: other
    Url: https://resolver.ebsco.com:443/public/rma-ftfapi/ejs/direct?AccessToken=495183E24C0DBCEA4191&Show=Object
Text:
  Availability: 0
CustomLinks:
  – Url: https://dx.doi.org/doi:10.1007/s11748-026-02304-9
    Name: EDS - Springer Nature Journals (s7799221)
    Category: fullText
    Text: View record at Springer
Header DbId: cmedm
DbLabel: MEDLINE
An: 42043642
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Performance of large language models on the Japanese cardiovascular surgery board examination: a comparative analysis of eight contemporary AI models with educational implications.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AU" term="%22Kasa+K%22">Kasa K</searchLink>; Division of Vascular Surgery, Department of Surgery, The Jikei University School of Medicine, Tokyo, Japan.; Division of Regenerative Medicine, Research Center for Medical Sciences, The Jikei University School of Medicine, Tokyo, Japan.<br /><searchLink fieldCode="AU" term="%22Yoshimaru+D%22">Yoshimaru D</searchLink>; Division of Regenerative Medicine, Research Center for Medical Sciences, The Jikei University School of Medicine, Tokyo, Japan.<br /><searchLink fieldCode="AU" term="%22Ohta+H%22">Ohta H</searchLink>; Division of Regenerative Medicine, Research Center for Medical Sciences, The Jikei University School of Medicine, Tokyo, Japan. hiroojikei@gmail.com.<br /><searchLink fieldCode="AU" term="%22Ohki+T%22">Ohki T</searchLink>; Division of Vascular Surgery, Department of Surgery, The Jikei University School of Medicine, Tokyo, Japan.<br /><searchLink fieldCode="AU" term="%22Okano+HJ%22">Okano HJ</searchLink>; Division of Regenerative Medicine, Research Center for Medical Sciences, The Jikei University School of Medicine, Tokyo, Japan. hjokano@jikei.ac.jp.
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%22101303952%22">General thoracic and cardiovascular surgery</searchLink> [Gen Thorac Cardiovasc Surg] 2026 Oct; Vol. 74 (10), pp. 950-962. <i>Date of Electronic Publication: </i>2026 Apr 27.
– Name: TypePub
  Label: Publication Type
  Group: TypPub
  Data: Journal Article; Comparative Study
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: TitleSource
  Label: Journal Info
  Group: Src
  Data: <i>Publisher: </i><searchLink fieldCode="PB" term="%22Springer+Japan%22">Springer Japan </searchLink><i>Country of Publication: </i>Japan <i>NLM ID: </i>101303952 <i>Publication Model: </i>Print-Electronic <i>Cited Medium: </i>Internet <i>ISSN: </i>1863-6713 (Electronic) <i>Linking ISSN: </i><searchLink fieldCode="IS" term="%2218636705%22">18636705 </searchLink><i>NLM ISO Abbreviation: </i>Gen Thorac Cardiovasc Surg <i>Subsets: </i>MEDLINE
– Name: PublisherInfo
  Label: Imprint Name(s)
  Group: PubInfo
  Data: <i>Original Publication</i>: Tokyo : Springer Japan
– Name: SubjectMESH
  Label: MeSH Terms
  Group: Su
  Data: <searchLink fieldCode="MM" term="%22Cardiovascular+Surgical+Procedures%22">Cardiovascular Surgical Procedures*</searchLink>/<searchLink fieldCode="MM" term="%22Cardiovascular+Surgical+Procedures+education%22">education</searchLink> <br /><searchLink fieldCode="MM" term="%22Large+Language+Models%22">Large Language Models*</searchLink> <br /><searchLink fieldCode="MM" term="%22Specialty+Boards%22">Specialty Boards*</searchLink> <br /><searchLink fieldCode="MM" term="%22Certification%22">Certification*</searchLink> <br /><searchLink fieldCode="MM" term="%22Educational+Measurement%22">Educational Measurement*</searchLink> <br /><searchLink fieldCode="MM" term="%22Education%2C+Medical%2C+Graduate%22">Education, Medical, Graduate*</searchLink><br /><searchLink fieldCode="MH" term="%22Humans%22">Humans</searchLink> ; <searchLink fieldCode="MH" term="%22Japan%22">Japan</searchLink> ; <searchLink fieldCode="MH" term="%22Generative+Artificial+Intelligence%22">Generative Artificial Intelligence</searchLink> ; <searchLink fieldCode="MH" term="%22Clinical+Competence%22">Clinical Competence</searchLink> ; <searchLink fieldCode="MH" term="%22East+Asian+People%22">East Asian People</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Objective: Large language models (LLMs) have shown strong performance on medical licensing and specialty examinations, but their utility in cardiovascular surgery certification and education remains unknown.<br />Methods: We evaluated eight LLMs (GPT-5, GPT-4o [OpenAI], Gemini-2.5Pro, Gemini-2.0, Gemma-3 [12B] [Google DeepMind], Claude-4, Claude-3 [Anthropic], and Llama-4 [Scout] [Meta AI]) using their official application programming interfaces as of September 2025. Examination items were obtained from the Japanese Cardiovascular Surgery Board (2021-2024; 523 questions). Texts were extracted from PDFs, images converted to JPEGs, and each question presented with a standardized Japanese prompt. Models produced three responses per item; final answers were determined by majority voting. Accuracy with 95% confidence intervals was calculated, and pairwise comparisons performed using McNemar's test.<br />Results: Across 523 items, GPT-5 achieved the highest accuracy (87.4%), followed by Gemini-2.5Pro (85.7%); their performance did not differ significantly. Claude-4 ranked third (80.3%), exceeding the passing threshold in some years. GPT-4o (65.6%) and Gemini-2.0 (58.5%) showed moderate accuracy, whereas Claude-3 (36.9%), Gemma-3 (40.9%), and Llama-4 (52.2%) scored lower. Pairwise testing confirmed a clear stratification: high (GPT-5, Gemini-2.5Pro), upper-intermediate (Claude-4), mid (GPT-4o, Gemini-2.0), and low (others). All models declined with image-based items, with top models reduced to around 70%. Accuracy remained stable across years, with GPT-5, Gemini-2.5Pro, and occasionally Claude-4, surpassing the pass threshold.<br />Conclusions: Successive model generations, particularly GPT-5 and Gemini-2.5Pro, consistently achieved passing-level accuracy. These findings highlight substantial gains through model evolution and underscore the potential of LLMs as supplementary tools for specialty education, despite persistent limitations in image-based reasoning.<br /> (© 2026. The Author(s), under exclusive licence to The Japanese Association for Thoracic Surgery.)
– Name: Abstract
  Label: Competing Interests
  Group: Ab
  Data: Declarations. Conflict of interest: The authors have no conflict of interest regarding this article. Ethical approval: This study utilized publicly available examination materials with explicit permission from the Japanese Board of Cardiovascular Surgery. The study did not involve human subjects or patient data; therefore, institutional review board approval was not required.
– Name: Ref
  Label: References
  Group: RefInfo
  Data: JMIR Med Educ. 2025 Jul 30;11:e69313. (PMID: <searchLink fieldCode="PM" term="%2240737609%22">40737609)</searchLink><br />JAMA. 2018 Apr 3;319(13):1317-1318. (PMID: <searchLink fieldCode="PM" term="%2229532063%22">29532063)</searchLink><br />Commun Med (Lond). 2023 Oct 10;3(1):141. (PMID: <searchLink fieldCode="PM" term="%2237816837%22">37816837)</searchLink><br />JAMA Netw Open. 2024 Jun 3;7(6):e2417641. (PMID: <searchLink fieldCode="PM" term="%2238888919%22">38888919)</searchLink><br />J Surg Educ. 2025 Apr;82(4):103442. (PMID: <searchLink fieldCode="PM" term="%2239923296%22">39923296)</searchLink><br />Lancet Digit Health. 2024 Oct;6(10):e767-e771. (PMID: <searchLink fieldCode="PM" term="%2239214760%22">39214760)</searchLink><br />BMC Med Educ. 2024 Nov 26;24(1):1372. (PMID: <searchLink fieldCode="PM" term="%2239593041%22">39593041)</searchLink><br />Nature. 2023 Aug;620(7972):172-180. (PMID: <searchLink fieldCode="PM" term="%2237438534%22">37438534)</searchLink><br />JMIR Med Educ. 2024 Dec 24;10:e63129. (PMID: <searchLink fieldCode="PM" term="%2239718557%22">39718557)</searchLink><br />JMIR Med Educ. 2023 Feb 8;9:e45312. (PMID: <searchLink fieldCode="PM" term="%2236753318%22">36753318)</searchLink><br />PLOS Digit Health. 2023 Feb 9;2(2):e0000198. (PMID: <searchLink fieldCode="PM" term="%2236812645%22">36812645)</searchLink><br />Jpn J Radiol. 2024 Aug;42(8):918-926. (PMID: <searchLink fieldCode="PM" term="%2238733472%22">38733472)</searchLink><br />Neurosurgery. 2023 Dec 1;93(6):1353-1365. (PMID: <searchLink fieldCode="PM" term="%2237581444%22">37581444)</searchLink><br />PLOS Digit Health. 2024 Jan 23;3(1):e0000433. (PMID: <searchLink fieldCode="PM" term="%2238261580%22">38261580)</searchLink><br />JAMA Netw Open. 2023 Dec 1;6(12):e2346721. (PMID: <searchLink fieldCode="PM" term="%2238060223%22">38060223)</searchLink><br />Mayo Clin Proc Digit Health. 2023 Jul 07;1(3):309-310. (PMID: <searchLink fieldCode="PM" term="%2240206625%22">40206625)</searchLink><br />Nat Med. 2025 Mar;31(3):943-950. (PMID: <searchLink fieldCode="PM" term="%2239779926%22">39779926)</searchLink><br />Cureus. 2024 Mar 18;16(3):e56402. (PMID: <searchLink fieldCode="PM" term="%2238633935%22">38633935)</searchLink>
– Name: GrantInfo
  Label: Grant Information
  Group: Grant
  Data: JP24K11980 JSPS KAKENHI
– Name: SubjectMinor
  Label: Contributed Indexing
  Group:
  Data: <i>Keywords: </i>Artificial intelligence; Large language models; Vision language models
– Name: DateEntry
  Label: Entry Date(s)
  Group: Date
  Data: <i>Date Created: </i>20260427 <i>Date Completed: </i>20261002 <i>Latest Revision: </i>20261008
– Name: DateUpdate
  Label: Update Code
  Group: Date
  Data: 20261008
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1007/s11748-026-02304-9
– Name: AN
  Label: PMID
  Group: ID
  Data: 42043642
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=cmedm&AN=42043642
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1007/s11748-026-02304-9
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        StartPage: 950
    Subjects:
      – SubjectFull: Humans
        Type: general
      – SubjectFull: Japan
        Type: general
      – SubjectFull: Generative Artificial Intelligence
        Type: general
      – SubjectFull: Clinical Competence
        Type: general
      – SubjectFull: East Asian People
        Type: general
      – SubjectFull: Cardiovascular Surgical Procedures education
        Type: general
      – SubjectFull: Large Language Models
        Type: general
      – SubjectFull: Specialty Boards
        Type: general
      – SubjectFull: Certification
        Type: general
      – SubjectFull: Educational Measurement
        Type: general
      – SubjectFull: Education, Medical, Graduate
        Type: general
    Titles:
      – TitleFull: Performance of large language models on the Japanese cardiovascular surgery board examination: a comparative analysis of eight contemporary AI models with educational implications.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Kasa K
      – PersonEntity:
          Name:
            NameFull: Yoshimaru D
      – PersonEntity:
          Name:
            NameFull: Ohta H
      – PersonEntity:
          Name:
            NameFull: Ohki T
      – PersonEntity:
          Name:
            NameFull: Okano HJ
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 10
              Text: 2026 Oct
              Type: published
              Y: 2026
          Identifiers:
            – Type: issn-electronic
              Value: 1863-6713
          Numbering:
            – Type: volume
              Value: 74
            – Type: issue
              Value: 10
          Titles:
            – TitleFull: General thoracic and cardiovascular surgery
              Type: main
ResultId 1