Academic Journal
Προβολή σε EDS

Performance of large language models on nursing licensure examinations: A systematic review and meta-analysis.

Λεπτομέρειες βιβλιογραφικής εγγραφής
Τίτλος: Performance of large language models on nursing licensure examinations: A systematic review and meta-analysis.
Συγγραφείς: Amankwaa I; School of Nursing, Auckland University of Technology, Auckland, New Zealand. Electronic address: Isaac.amankwaa@aut.ac.nz., Odoom A; Department of Medical Microbiology, University of Ghana Medical School, P. O. Box KB 4236, Korle Bu, Accra, Ghana., Kasim A; Nursing and Midwifery Training College, Techiman, Krobo, Ghana., Kobiah E; Department of Nursing, Kwame Nkrumah University of Science and Technology, Kumasi, Ghana., Diebieri M; Nursing and Midwifery Training College, Kpembe, Salaga, Ghana., Boateng EA; Department of Nursing, Kwame Nkrumah University of Science and Technology, Kumasi, Ghana., Gyamfi S; Faculty of Nursing, University of Windsor, Windsor, Ontario, Canada., Hales C; School of Health, Victoria University of Wellington, Wellington, New Zealand.
Πηγή: Nurse education today [Nurse Educ Today] 2026 Oct; Vol. 165, pp. 107154. Date of Electronic Publication: 2026 May 12.
Τύπος έκδοσης: Journal Article; Systematic Review; Meta-Analysis; Review
Γλώσσα: English
Στοιχεία περιοδικού: Publisher: Churchill Livingstone Country of Publication: Scotland NLM ID: 8511379 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 1532-2793 (Electronic) Linking ISSN: 02606917 NLM ISO Abbreviation: Nurse Educ Today Subsets: MEDLINE
Imprint Name(s): Original Publication: [Edinburgh, Scotland] : Churchill Livingstone
Ιατρικοί όροι (MeSH): Licensure, Nursing*/standards , Large Language Models*, Educational Measurement/methods ; Educational Measurement/standards ; Humans ; Internationality
Περίληψη: Objectives: This systematic review and meta-analysis assessed the performance of large language models (LLMs) in nursing licensure examinations. Despite the increasing use of LLMs in healthcare education, their capabilities in nursing licensure examinations remain uncertain. This study provides evidence on the accuracy and limitations of LLMs to help guide their integration into nursing education and licensure.
Design: The systematic review and meta-analysis adhered to PRISMA 2020 guidelines.
Data Sources: PubMed, CINAHL, PsycINFO, EMCARE, and ERIC were searched from April to June 2025.
Eligibility Criteria: Studies were eligible if they evaluated LLMs (e.g., GPT-4, ChatGPT, Qwen-2.5) using multiple-choice nursing licensure questions under exam-like conditions and reported quantitative accuracy. Open-ended items were excluded from the meta-analysis due to incompatible scoring methods, but were narratively synthesised.
Review Methods: Two reviewers independently screened, extracted data, and appraised the risk of bias. A random-effects meta-analysis estimated pooled accuracy; subgroup and meta-regression analyses explored heterogeneity.
Results: Twelve studies assessed 13,870 MCQs across seven exam systems and ten LLMs. Pooled accuracy was 69.6% (95% CI: 65.6-73.6%) with substantial heterogeneity (I2 = 98%). GPT-4 outperformed GPT-3.5 (77.2% vs. 60.4%); domain-customised and newer models reached 93.6%. LLMs excelled in general medicine and pharmacology but underperformed in ethics and psychosocial integrity. Accuracy did not differ significantly by exam system (p = 0.14), question difficulty (p = 0.90) or format (p = 0.96). In meta-regression, Custom GPT (p = 0.0006) and Qwen 2.5 (p = 0.026) were the only significant predictors of higher accuracy; no exam system, question format, or difficulty level reached significance. Methodological variability and underreporting of model parameters were common.
Conclusions: LLMs show promise for low-stakes educational applications, such as formative assessments within hybrid teaching models; however, they are unsuitable for unmoderated, high-stakes licensure decisions due to inconsistent performance. Regulatory guidelines, equitable access, and nursing-specific model development are needed to ensure fairness and validity. Research must prioritise standardised frameworks, error analysis, and broader geographic representation to address these limitations.
(Copyright © 2026 The Author(s). Published by Elsevier Ltd.. All rights reserved.)
Competing Interests: Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships, including affiliations with AI developers, that could have appeared to influence the work reported in this paper. No funding was received from AI-related organisations for this study.
Contributed Indexing: Keywords: Artificial Intelligence; Educational measurement; Large language models; Licensure; Nursing; Systematic review
Entry Date(s): Date Created: 20260518 Date Completed: 20260606 Latest Revision: 20260609
Update Code: 20260609
DOI: 10.1016/j.nedt.2026.107154
PMID: 42150508
Βάση Δεδομένων: MEDLINE
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://www.doi.org/10.1016/j.nedt.2026.107154?
    Name: ScienceDirect (all content) (s7799221)
    Category: fullText
    Text: View record from ScienceDirect
    MouseOverText: View record from ScienceDirect
  – Url: https://resolver.ebsco.com/c/fiv2js/result?sid=EBSCO:cmedm&genre=article&issn=15322793&ISBN=&volume=165&issue=&date=20261001&spage=107154&pages=107154&title=Nurse education today&atitle=Performance%20of%20large%20language%20models%20on%20nursing%20licensure%20examinations%3A%20A%20systematic%20review%20and%20meta-analysis.&aulast=Amankwaa%20I&id=DOI:10.1016/j.nedt.2026.107154
    Name: Full Text Finder (for New FTF UI) (ns324271)
    Category: fullText
    Text: Full Text Finder
    MouseOverText: Full Text Finder
Header DbId: cmedm
DbLabel: MEDLINE
An: 42150508
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Performance of large language models on nursing licensure examinations: A systematic review and meta-analysis.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AU" term="%22Amankwaa+I%22">Amankwaa I</searchLink>; School of Nursing, Auckland University of Technology, Auckland, New Zealand. Electronic address: Isaac.amankwaa@aut.ac.nz.<br /><searchLink fieldCode="AU" term="%22Odoom+A%22">Odoom A</searchLink>; Department of Medical Microbiology, University of Ghana Medical School, P. O. Box KB 4236, Korle Bu, Accra, Ghana.<br /><searchLink fieldCode="AU" term="%22Kasim+A%22">Kasim A</searchLink>; Nursing and Midwifery Training College, Techiman, Krobo, Ghana.<br /><searchLink fieldCode="AU" term="%22Kobiah+E%22">Kobiah E</searchLink>; Department of Nursing, Kwame Nkrumah University of Science and Technology, Kumasi, Ghana.<br /><searchLink fieldCode="AU" term="%22Diebieri+M%22">Diebieri M</searchLink>; Nursing and Midwifery Training College, Kpembe, Salaga, Ghana.<br /><searchLink fieldCode="AU" term="%22Boateng+EA%22">Boateng EA</searchLink>; Department of Nursing, Kwame Nkrumah University of Science and Technology, Kumasi, Ghana.<br /><searchLink fieldCode="AU" term="%22Gyamfi+S%22">Gyamfi S</searchLink>; Faculty of Nursing, University of Windsor, Windsor, Ontario, Canada.<br /><searchLink fieldCode="AU" term="%22Hales+C%22">Hales C</searchLink>; School of Health, Victoria University of Wellington, Wellington, New Zealand.
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%228511379%22">Nurse education today</searchLink> [Nurse Educ Today] 2026 Oct; Vol. 165, pp. 107154. <i>Date of Electronic Publication: </i>2026 May 12.
– Name: TypePub
  Label: Publication Type
  Group: TypPub
  Data: Journal Article; Systematic Review; Meta-Analysis; Review
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: TitleSource
  Label: Journal Info
  Group: Src
  Data: <i>Publisher: </i><searchLink fieldCode="PB" term="%22Churchill+Livingstone%22">Churchill Livingstone </searchLink><i>Country of Publication: </i>Scotland <i>NLM ID: </i>8511379 <i>Publication Model: </i>Print-Electronic <i>Cited Medium: </i>Internet <i>ISSN: </i>1532-2793 (Electronic) <i>Linking ISSN: </i><searchLink fieldCode="IS" term="%2202606917%22">02606917 </searchLink><i>NLM ISO Abbreviation: </i>Nurse Educ Today <i>Subsets: </i>MEDLINE
– Name: PublisherInfo
  Label: Imprint Name(s)
  Group: PubInfo
  Data: <i>Original Publication</i>: [Edinburgh, Scotland] : Churchill Livingstone
– Name: SubjectMESH
  Label: MeSH Terms
  Group: Su
  Data: <searchLink fieldCode="MM" term="%22Licensure%2C+Nursing%22">Licensure, Nursing*</searchLink>/<searchLink fieldCode="MM" term="%22Licensure%2C+Nursing+standards%22">standards</searchLink> <br /><searchLink fieldCode="MM" term="%22Large+Language+Models%22">Large Language Models*</searchLink><br /><searchLink fieldCode="MH" term="%22Educational+Measurement%22">Educational Measurement</searchLink>/<searchLink fieldCode="MH" term="%22Educational+Measurement+methods%22">methods</searchLink> ; <searchLink fieldCode="MH" term="%22Educational+Measurement%22">Educational Measurement</searchLink>/<searchLink fieldCode="MH" term="%22Educational+Measurement+standards%22">standards</searchLink> ; <searchLink fieldCode="MH" term="%22Humans%22">Humans</searchLink> ; <searchLink fieldCode="MH" term="%22Internationality%22">Internationality</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Objectives: This systematic review and meta-analysis assessed the performance of large language models (LLMs) in nursing licensure examinations. Despite the increasing use of LLMs in healthcare education, their capabilities in nursing licensure examinations remain uncertain. This study provides evidence on the accuracy and limitations of LLMs to help guide their integration into nursing education and licensure.<br />Design: The systematic review and meta-analysis adhered to PRISMA 2020 guidelines.<br />Data Sources: PubMed, CINAHL, PsycINFO, EMCARE, and ERIC were searched from April to June 2025.<br />Eligibility Criteria: Studies were eligible if they evaluated LLMs (e.g., GPT-4, ChatGPT, Qwen-2.5) using multiple-choice nursing licensure questions under exam-like conditions and reported quantitative accuracy. Open-ended items were excluded from the meta-analysis due to incompatible scoring methods, but were narratively synthesised.<br />Review Methods: Two reviewers independently screened, extracted data, and appraised the risk of bias. A random-effects meta-analysis estimated pooled accuracy; subgroup and meta-regression analyses explored heterogeneity.<br />Results: Twelve studies assessed 13,870 MCQs across seven exam systems and ten LLMs. Pooled accuracy was 69.6% (95% CI: 65.6-73.6%) with substantial heterogeneity (I<superscript>2</superscript> = 98%). GPT-4 outperformed GPT-3.5 (77.2% vs. 60.4%); domain-customised and newer models reached 93.6%. LLMs excelled in general medicine and pharmacology but underperformed in ethics and psychosocial integrity. Accuracy did not differ significantly by exam system (p = 0.14), question difficulty (p = 0.90) or format (p = 0.96). In meta-regression, Custom GPT (p = 0.0006) and Qwen 2.5 (p = 0.026) were the only significant predictors of higher accuracy; no exam system, question format, or difficulty level reached significance. Methodological variability and underreporting of model parameters were common.<br />Conclusions: LLMs show promise for low-stakes educational applications, such as formative assessments within hybrid teaching models; however, they are unsuitable for unmoderated, high-stakes licensure decisions due to inconsistent performance. Regulatory guidelines, equitable access, and nursing-specific model development are needed to ensure fairness and validity. Research must prioritise standardised frameworks, error analysis, and broader geographic representation to address these limitations.<br /> (Copyright © 2026 The Author(s). Published by Elsevier Ltd.. All rights reserved.)
– Name: Abstract
  Label: Competing Interests
  Group: Ab
  Data: Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships, including affiliations with AI developers, that could have appeared to influence the work reported in this paper. No funding was received from AI-related organisations for this study.
– Name: SubjectMinor
  Label: Contributed Indexing
  Group:
  Data: <i>Keywords: </i>Artificial Intelligence; Educational measurement; Large language models; Licensure; Nursing; Systematic review
– Name: DateEntry
  Label: Entry Date(s)
  Group: Date
  Data: <i>Date Created: </i>20260518 <i>Date Completed: </i>20260606 <i>Latest Revision: </i>20260609
– Name: DateUpdate
  Label: Update Code
  Group: Date
  Data: 20260609
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1016/j.nedt.2026.107154
– Name: AN
  Label: PMID
  Group: ID
  Data: 42150508
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=cmedm&AN=42150508
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1016/j.nedt.2026.107154
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        StartPage: 107154
    Subjects:
      – SubjectFull: Educational Measurement methods
        Type: general
      – SubjectFull: Educational Measurement standards
        Type: general
      – SubjectFull: Humans
        Type: general
      – SubjectFull: Internationality
        Type: general
      – SubjectFull: Licensure, Nursing standards
        Type: general
      – SubjectFull: Large Language Models
        Type: general
    Titles:
      – TitleFull: Performance of large language models on nursing licensure examinations: A systematic review and meta-analysis.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Amankwaa I
      – PersonEntity:
          Name:
            NameFull: Odoom A
      – PersonEntity:
          Name:
            NameFull: Kasim A
      – PersonEntity:
          Name:
            NameFull: Kobiah E
      – PersonEntity:
          Name:
            NameFull: Diebieri M
      – PersonEntity:
          Name:
            NameFull: Boateng EA
      – PersonEntity:
          Name:
            NameFull: Gyamfi S
      – PersonEntity:
          Name:
            NameFull: Hales C
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 10
              Text: 2026 Oct
              Type: published
              Y: 2026
          Identifiers:
            – Type: issn-electronic
              Value: 1532-2793
          Numbering:
            – Type: volume
              Value: 165
          Titles:
            – TitleFull: Nurse education today
              Type: main
ResultId 1