Academic Journal
Performance of large language models on the Japanese cardiovascular surgery board examination: a comparative analysis of eight contemporary AI models with educational implications.
| Τίτλος: | Performance of large language models on the Japanese cardiovascular surgery board examination: a comparative analysis of eight contemporary AI models with educational implications. |
|---|---|
| Συγγραφείς: | Kasa K; Division of Vascular Surgery, Department of Surgery, The Jikei University School of Medicine, Tokyo, Japan.; Division of Regenerative Medicine, Research Center for Medical Sciences, The Jikei University School of Medicine, Tokyo, Japan., Yoshimaru D; Division of Regenerative Medicine, Research Center for Medical Sciences, The Jikei University School of Medicine, Tokyo, Japan., Ohta H; Division of Regenerative Medicine, Research Center for Medical Sciences, The Jikei University School of Medicine, Tokyo, Japan. hiroojikei@gmail.com., Ohki T; Division of Vascular Surgery, Department of Surgery, The Jikei University School of Medicine, Tokyo, Japan., Okano HJ; Division of Regenerative Medicine, Research Center for Medical Sciences, The Jikei University School of Medicine, Tokyo, Japan. hjokano@jikei.ac.jp. |
| Πηγή: | General thoracic and cardiovascular surgery [Gen Thorac Cardiovasc Surg] 2026 Oct; Vol. 74 (10), pp. 950-962. Date of Electronic Publication: 2026 Apr 27. |
| Τύπος έκδοσης: | Journal Article; Comparative Study |
| Γλώσσα: | English |
| Στοιχεία περιοδικού: | Publisher: Springer Japan Country of Publication: Japan NLM ID: 101303952 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 1863-6713 (Electronic) Linking ISSN: 18636705 NLM ISO Abbreviation: Gen Thorac Cardiovasc Surg Subsets: MEDLINE |
| Imprint Name(s): | Original Publication: Tokyo : Springer Japan |
| Ιατρικοί όροι (MeSH): | Cardiovascular Surgical Procedures*/education , Large Language Models* , Specialty Boards* , Certification* , Educational Measurement* , Education, Medical, Graduate*, Humans ; Japan ; Generative Artificial Intelligence ; Clinical Competence ; East Asian People |
| Περίληψη: | Objective: Large language models (LLMs) have shown strong performance on medical licensing and specialty examinations, but their utility in cardiovascular surgery certification and education remains unknown. Methods: We evaluated eight LLMs (GPT-5, GPT-4o [OpenAI], Gemini-2.5Pro, Gemini-2.0, Gemma-3 [12B] [Google DeepMind], Claude-4, Claude-3 [Anthropic], and Llama-4 [Scout] [Meta AI]) using their official application programming interfaces as of September 2025. Examination items were obtained from the Japanese Cardiovascular Surgery Board (2021-2024; 523 questions). Texts were extracted from PDFs, images converted to JPEGs, and each question presented with a standardized Japanese prompt. Models produced three responses per item; final answers were determined by majority voting. Accuracy with 95% confidence intervals was calculated, and pairwise comparisons performed using McNemar's test. Results: Across 523 items, GPT-5 achieved the highest accuracy (87.4%), followed by Gemini-2.5Pro (85.7%); their performance did not differ significantly. Claude-4 ranked third (80.3%), exceeding the passing threshold in some years. GPT-4o (65.6%) and Gemini-2.0 (58.5%) showed moderate accuracy, whereas Claude-3 (36.9%), Gemma-3 (40.9%), and Llama-4 (52.2%) scored lower. Pairwise testing confirmed a clear stratification: high (GPT-5, Gemini-2.5Pro), upper-intermediate (Claude-4), mid (GPT-4o, Gemini-2.0), and low (others). All models declined with image-based items, with top models reduced to around 70%. Accuracy remained stable across years, with GPT-5, Gemini-2.5Pro, and occasionally Claude-4, surpassing the pass threshold. Conclusions: Successive model generations, particularly GPT-5 and Gemini-2.5Pro, consistently achieved passing-level accuracy. These findings highlight substantial gains through model evolution and underscore the potential of LLMs as supplementary tools for specialty education, despite persistent limitations in image-based reasoning. (© 2026. The Author(s), under exclusive licence to The Japanese Association for Thoracic Surgery.) |
| Competing Interests: | Declarations. Conflict of interest: The authors have no conflict of interest regarding this article. Ethical approval: This study utilized publicly available examination materials with explicit permission from the Japanese Board of Cardiovascular Surgery. The study did not involve human subjects or patient data; therefore, institutional review board approval was not required. |
| References: | JMIR Med Educ. 2025 Jul 30;11:e69313. (PMID: 40737609) JAMA. 2018 Apr 3;319(13):1317-1318. (PMID: 29532063) Commun Med (Lond). 2023 Oct 10;3(1):141. (PMID: 37816837) JAMA Netw Open. 2024 Jun 3;7(6):e2417641. (PMID: 38888919) J Surg Educ. 2025 Apr;82(4):103442. (PMID: 39923296) Lancet Digit Health. 2024 Oct;6(10):e767-e771. (PMID: 39214760) BMC Med Educ. 2024 Nov 26;24(1):1372. (PMID: 39593041) Nature. 2023 Aug;620(7972):172-180. (PMID: 37438534) JMIR Med Educ. 2024 Dec 24;10:e63129. (PMID: 39718557) JMIR Med Educ. 2023 Feb 8;9:e45312. (PMID: 36753318) PLOS Digit Health. 2023 Feb 9;2(2):e0000198. (PMID: 36812645) Jpn J Radiol. 2024 Aug;42(8):918-926. (PMID: 38733472) Neurosurgery. 2023 Dec 1;93(6):1353-1365. (PMID: 37581444) PLOS Digit Health. 2024 Jan 23;3(1):e0000433. (PMID: 38261580) JAMA Netw Open. 2023 Dec 1;6(12):e2346721. (PMID: 38060223) Mayo Clin Proc Digit Health. 2023 Jul 07;1(3):309-310. (PMID: 40206625) Nat Med. 2025 Mar;31(3):943-950. (PMID: 39779926) Cureus. 2024 Mar 18;16(3):e56402. (PMID: 38633935) |
| Grant Information: | JP24K11980 JSPS KAKENHI |
| Contributed Indexing: | Keywords: Artificial intelligence; Large language models; Vision language models |
| Entry Date(s): | Date Created: 20260427 Date Completed: 20261002 Latest Revision: 20261008 |
| Update Code: | 20261008 |
| DOI: | 10.1007/s11748-026-02304-9 |
| PMID: | 42043642 |
| Βάση Δεδομένων: | MEDLINE |
καταχωρήστε σχόλιο πρώτοι!