Academic Journal
View in EDS

Validation of a Natural Language Processing-Assisted Chart Review Method for Identifying Advance Care Planning Documentation in Pediatric Critical Illness.

Bibliographic Details
Title: Validation of a Natural Language Processing-Assisted Chart Review Method for Identifying Advance Care Planning Documentation in Pediatric Critical Illness.
Authors: Upham EJ; Division of Medical Critical Care, Department of Pediatrics, Boston Children's Hospital, Boston, MA.; Department of Supportive Oncology, Dana-Farber Cancer Institute, Boston, MA., Gouda SR; Division of Medical Critical Care, Department of Pediatrics, Boston Children's Hospital, Boston, MA.; Department of Supportive Oncology, Dana-Farber Cancer Institute, Boston, MA.; Harvard Medical School, Boston, MA., Lindvall C; Department of Supportive Oncology, Dana-Farber Cancer Institute, Boston, MA.; Harvard Medical School, Boston, MA., DeCourcey DD; Division of Medical Critical Care, Department of Pediatrics, Boston Children's Hospital, Boston, MA.; Harvard Medical School, Boston, MA.
Source: Critical care explorations [Crit Care Explor] 2026 Sep 25; Vol. 8 (10), pp. e1494. Date of Electronic Publication: 2026 Sep 25 (Print Publication: 2026).
Publication Type: Journal Article; Validation Study
Language: English
Journal Info: Publisher: Wolters Kluwer Health Country of Publication: United States NLM ID: 101746347 Publication Model: eCollection Cited Medium: Internet ISSN: 2639-8028 (Electronic) Linking ISSN: 26398028 NLM ISO Abbreviation: Crit Care Explor Subsets: MEDLINE
Imprint Name(s): Original Publication: Philadelphia, PA : Wolters Kluwer Health, [2019]-
MeSH Terms: Advance Care Planning*/standards , Advance Care Planning*/statistics & numerical data , Critical Illness*/therapy , Documentation*/methods , Documentation*/standards , Natural Language Processing*, Humans ; Retrospective Studies ; Electronic Health Records ; Child ; Female ; Male ; Adolescent ; Child, Preschool ; Intensive Care Units, Pediatric ; Infant
Abstract: Objectives: Reliable measurement of advance care planning (ACP) is essential for evaluating both communication practices and quality of care in pediatric critical illness. However, most ACP documentation is embedded in free-text clinical notes and cannot be captured using administrative data, while manual chart review is time-intensive and difficult to scale. We sought to validate a rule-based natural language processing (NLP) approach to identify ACP documentation in the electronic health record (EHR) compared with manual chart review.
Design: Retrospective cohort study.
Setting: Single-center, quaternary PICU.
Patients: Children younger than 21 years admitted to the PICU for greater than 24 hours following out-of-hospital cardiac arrest (OHCA) from 2012 to 2024.
Interventions: None.
Measurements and Main Results: A rule-based NLP approach incorporating semi-automated chart review was developed through iterative refinement of previously validated keyword libraries to identify ACP documentation across prespecified domains in the EHR. In the validation cohort (n = 95), NLP-assisted chart review showed complete agreement with manual chart review (F1 = 1.0) while substantially reducing abstraction time, enabling analysis of over 31,479 clinical notes in 3 days compared with 6 months for manual review of 43,179 notes. Among 125 children with OHCA, 65% had documented ACP discussions (n = 81). Goals-of-care conversations were most common (99%), followed by limitations of life-sustaining treatment (78%), subspecialty palliative care involvement (28%), preferred location of death (9%), and hospice discussions (7%).
Conclusions: NLP-assisted chart review can efficiently and accurately identify ACP documentation in pediatric critical illness for retrospective research and quality improvement efforts. This semi-automated workflow substantially reduced manual abstraction burden while maintaining excellent agreement with manual chart review, enabling effective abstraction of communication processes not captured in structured data.
(Copyright © 2026 The Authors. Published by Wolters Kluwer Health, LLC. on behalf of the Society of Critical Care Medicine.)
Competing Interests: The authors have disclosed that they do not have any potential conflicts of interest.
References: DeCourcey DD, Silverman M, Oladunjoye A, et al.: Advance care planning and parent-reported end-of-life outcomes in children, adolescents, and young adults with complex chronic conditions. Crit Care Med 2019; 47:101–108.
Canadian Paediatric Society Bioethics Committee: Advance care planning for Paediatric patients. Paediatr Child Health 2008; 13:791–796.
Lusney N, van Breemen C, Lim E, et al.: Pediatric advance care planning: A scoping review. Children (Basel) 2023; 10:1179.
Hirsh CD, Friebert S: Primary pediatric palliative care: Psychological and social support for children and families. Pediatr Rev 2014; 35:390–395.
Wu A, Huang RJ, Colón GR, et al.: Low rates of structured advance care planning documentation in electronic health records: Results of a single-center observational study. BMC Palliat Care 2022; 21:203.
Zupanc SN, Lakin JR, Volandes AE, et al.: Forms or free-text? Measuring advance care planning activity using electronic health records. J Pain Symptom Manage 2023; 66:e615–e624.
Lindvall C, Deng CY, Moseley E, et al.: Natural language processing to identify advance care planning documentation in a multisite pragmatic clinical trial. J Pain Symptom Manage 2022; 63:e29–e36.
Lee RY, Kross EK, Torrence J, et al.: Assessment of natural language processing of electronic health records to measure goals-of-care discussions as a clinical trial outcome. JAMA Netw Open 2023; 6:e231204.
Straub L, Gagne JJ, Maro JC, et al.: Evaluation of use of technologies to facilitate medical chart review. Drug Saf 2019; 42:1071–1080.
Lindsay ME, de Oliveira S, Sciacca K, et al.: Harnessing natural language processing to assess quality of end-of-life care for children with cancer. JCO Clin Cancer Inform 2024; 8:e2400134.
Agaronnik ND, Davis J, Manz CR, et al.: Large language models to identify advance care planning in patients with advanced cancer. J Pain Symptom Manage 2025; 69:243–250.e1.
Lindvall C, Lilley EJ, Zupanc SN, et al.: Natural language processing to assess end-of-life quality indicators in cancer patients receiving palliative surgery. J Palliat Med 2019; 22:183–187.
Feudtner C, Feinstein JA, Zhong W, et al.: Pediatric complex chronic conditions classification system version 2: Updated for ICD-10 and complex medical technology dependence and transplantation. BMC Pediatr 2014; 14:199.
Feudtner C, Friebert S, Jewell J, et al.; Section on Hospice and Palliative Medicine and Committee on Hospital Care: Pediatric palliative care and hospice care commitments, guidelines, and recommendations. Pediatrics 2013; 132:966–972.
Derrington S, Broden Arciprete EG, Lin MC, et al.: Society of Critical Care Medicine 2026 guidelines on the care and management of pediatric and neonatal intensive care patients at the end of life. Pediatr Crit Care Med 2026; 27:518–536.
Lindvall Lab: Clinical Regex [computer program]. Boston, MA, Dana-Farber Cancer Institute, 2025. Available at: https://lindvalllab.github.io/clinical-regex/. Accessed January 28, 2026.
Agaronnik ND, Davis J, Manz CR, et al.: Feasibility study for using large language models to identify goals-of-care documentation at scale in patients with advanced cancer. JCO Oncol Pract 2026; 22:294–305.
Tooten S, Hermens RPMG, Verhoeven M, et al.: Pediatric advance care planning: A mixed-methods evaluation of documentation and sharing in current practice. BMC Palliat Care 2026; 25:51.
Contributed Indexing: Keywords: advance care planning; critical care; natural language processing; out-of-hospital cardiac arrest; pediatrics
Entry Date(s): Date Created: 20260925 Date Completed: 20260925 Latest Revision: 20260925
Update Code: 20260926
DOI: 10.1097/CCE.0000000000001494
PMID: 42789538
Database: MEDLINE
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://resolver.ebsco.com/c/fiv2js/result?sid=EBSCO:cmedm&genre=article&issn=26398028&ISBN=&volume=8&issue=10&date=20260925&spage=e1494&pages=e1494&title=Critical care explorations&atitle=Validation%20of%20a%20Natural%20Language%20Processing-Assisted%20Chart%20Review%20Method%20for%20Identifying%20Advance%20Care%20Planning%20Documentation%20in%20Pediatric%20Critical%20Illness.&aulast=Upham%20EJ&id=DOI:10.1097/CCE.0000000000001494
    Name: Full Text Finder (for New FTF UI) (ns324271)
    Category: fullText
    Text: Full Text Finder
    MouseOverText: Full Text Finder
Header DbId: cmedm
DbLabel: MEDLINE
An: 42789538
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Validation of a Natural Language Processing-Assisted Chart Review Method for Identifying Advance Care Planning Documentation in Pediatric Critical Illness.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AU" term="%22Upham+EJ%22">Upham EJ</searchLink>; Division of Medical Critical Care, Department of Pediatrics, Boston Children's Hospital, Boston, MA.; Department of Supportive Oncology, Dana-Farber Cancer Institute, Boston, MA.<br /><searchLink fieldCode="AU" term="%22Gouda+SR%22">Gouda SR</searchLink>; Division of Medical Critical Care, Department of Pediatrics, Boston Children's Hospital, Boston, MA.; Department of Supportive Oncology, Dana-Farber Cancer Institute, Boston, MA.; Harvard Medical School, Boston, MA.<br /><searchLink fieldCode="AU" term="%22Lindvall+C%22">Lindvall C</searchLink>; Department of Supportive Oncology, Dana-Farber Cancer Institute, Boston, MA.; Harvard Medical School, Boston, MA.<br /><searchLink fieldCode="AU" term="%22DeCourcey+DD%22">DeCourcey DD</searchLink>; Division of Medical Critical Care, Department of Pediatrics, Boston Children's Hospital, Boston, MA.; Harvard Medical School, Boston, MA.
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%22101746347%22">Critical care explorations</searchLink> [Crit Care Explor] 2026 Sep 25; Vol. 8 (10), pp. e1494. <i>Date of Electronic Publication: </i>2026 Sep 25 (<i>Print Publication: </i>2026).
– Name: TypePub
  Label: Publication Type
  Group: TypPub
  Data: Journal Article; Validation Study
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: TitleSource
  Label: Journal Info
  Group: Src
  Data: <i>Publisher: </i><searchLink fieldCode="PB" term="%22Wolters+Kluwer+Health%22">Wolters Kluwer Health </searchLink><i>Country of Publication: </i>United States <i>NLM ID: </i>101746347 <i>Publication Model: </i>eCollection <i>Cited Medium: </i>Internet <i>ISSN: </i>2639-8028 (Electronic) <i>Linking ISSN: </i><searchLink fieldCode="IS" term="%2226398028%22">26398028 </searchLink><i>NLM ISO Abbreviation: </i>Crit Care Explor <i>Subsets: </i>MEDLINE
– Name: PublisherInfo
  Label: Imprint Name(s)
  Group: PubInfo
  Data: <i>Original Publication</i>: Philadelphia, PA : Wolters Kluwer Health, [2019]-
– Name: SubjectMESH
  Label: MeSH Terms
  Group: Su
  Data: <searchLink fieldCode="MM" term="%22Advance+Care+Planning%22">Advance Care Planning*</searchLink>/<searchLink fieldCode="MM" term="%22Advance+Care+Planning+standards%22">standards</searchLink> <br /><searchLink fieldCode="MM" term="%22Advance+Care+Planning%22">Advance Care Planning*</searchLink>/<searchLink fieldCode="MM" term="%22Advance+Care+Planning+statistics+%26+numerical+data%22">statistics & numerical data</searchLink> <br /><searchLink fieldCode="MM" term="%22Critical+Illness%22">Critical Illness*</searchLink>/<searchLink fieldCode="MM" term="%22Critical+Illness+therapy%22">therapy</searchLink> <br /><searchLink fieldCode="MM" term="%22Documentation%22">Documentation*</searchLink>/<searchLink fieldCode="MM" term="%22Documentation+methods%22">methods</searchLink> <br /><searchLink fieldCode="MM" term="%22Documentation%22">Documentation*</searchLink>/<searchLink fieldCode="MM" term="%22Documentation+standards%22">standards</searchLink> <br /><searchLink fieldCode="MM" term="%22Natural+Language+Processing%22">Natural Language Processing*</searchLink><br /><searchLink fieldCode="MH" term="%22Humans%22">Humans</searchLink> ; <searchLink fieldCode="MH" term="%22Retrospective+Studies%22">Retrospective Studies</searchLink> ; <searchLink fieldCode="MH" term="%22Electronic+Health+Records%22">Electronic Health Records</searchLink> ; <searchLink fieldCode="MH" term="%22Child%22">Child</searchLink> ; <searchLink fieldCode="MH" term="%22Female%22">Female</searchLink> ; <searchLink fieldCode="MH" term="%22Male%22">Male</searchLink> ; <searchLink fieldCode="MH" term="%22Adolescent%22">Adolescent</searchLink> ; <searchLink fieldCode="MH" term="%22Child%2C+Preschool%22">Child, Preschool</searchLink> ; <searchLink fieldCode="MH" term="%22Intensive+Care+Units%2C+Pediatric%22">Intensive Care Units, Pediatric</searchLink> ; <searchLink fieldCode="MH" term="%22Infant%22">Infant</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Objectives: Reliable measurement of advance care planning (ACP) is essential for evaluating both communication practices and quality of care in pediatric critical illness. However, most ACP documentation is embedded in free-text clinical notes and cannot be captured using administrative data, while manual chart review is time-intensive and difficult to scale. We sought to validate a rule-based natural language processing (NLP) approach to identify ACP documentation in the electronic health record (EHR) compared with manual chart review.<br />Design: Retrospective cohort study.<br />Setting: Single-center, quaternary PICU.<br />Patients: Children younger than 21 years admitted to the PICU for greater than 24 hours following out-of-hospital cardiac arrest (OHCA) from 2012 to 2024.<br />Interventions: None.<br />Measurements and Main Results: A rule-based NLP approach incorporating semi-automated chart review was developed through iterative refinement of previously validated keyword libraries to identify ACP documentation across prespecified domains in the EHR. In the validation cohort (n = 95), NLP-assisted chart review showed complete agreement with manual chart review (F1 = 1.0) while substantially reducing abstraction time, enabling analysis of over 31,479 clinical notes in 3 days compared with 6 months for manual review of 43,179 notes. Among 125 children with OHCA, 65% had documented ACP discussions (n = 81). Goals-of-care conversations were most common (99%), followed by limitations of life-sustaining treatment (78%), subspecialty palliative care involvement (28%), preferred location of death (9%), and hospice discussions (7%).<br />Conclusions: NLP-assisted chart review can efficiently and accurately identify ACP documentation in pediatric critical illness for retrospective research and quality improvement efforts. This semi-automated workflow substantially reduced manual abstraction burden while maintaining excellent agreement with manual chart review, enabling effective abstraction of communication processes not captured in structured data.<br /> (Copyright © 2026 The Authors. Published by Wolters Kluwer Health, LLC. on behalf of the Society of Critical Care Medicine.)
– Name: Abstract
  Label: Competing Interests
  Group: Ab
  Data: The authors have disclosed that they do not have any potential conflicts of interest.
– Name: Ref
  Label: References
  Group: RefInfo
  Data: DeCourcey DD, Silverman M, Oladunjoye A, et al.: Advance care planning and parent-reported end-of-life outcomes in children, adolescents, and young adults with complex chronic conditions. Crit Care Med 2019; 47:101–108.<br />Canadian Paediatric Society Bioethics Committee: Advance care planning for Paediatric patients. Paediatr Child Health 2008; 13:791–796.<br />Lusney N, van Breemen C, Lim E, et al.: Pediatric advance care planning: A scoping review. Children (Basel) 2023; 10:1179.<br />Hirsh CD, Friebert S: Primary pediatric palliative care: Psychological and social support for children and families. Pediatr Rev 2014; 35:390–395.<br />Wu A, Huang RJ, Colón GR, et al.: Low rates of structured advance care planning documentation in electronic health records: Results of a single-center observational study. BMC Palliat Care 2022; 21:203.<br />Zupanc SN, Lakin JR, Volandes AE, et al.: Forms or free-text? Measuring advance care planning activity using electronic health records. J Pain Symptom Manage 2023; 66:e615–e624.<br />Lindvall C, Deng CY, Moseley E, et al.: Natural language processing to identify advance care planning documentation in a multisite pragmatic clinical trial. J Pain Symptom Manage 2022; 63:e29–e36.<br />Lee RY, Kross EK, Torrence J, et al.: Assessment of natural language processing of electronic health records to measure goals-of-care discussions as a clinical trial outcome. JAMA Netw Open 2023; 6:e231204.<br />Straub L, Gagne JJ, Maro JC, et al.: Evaluation of use of technologies to facilitate medical chart review. Drug Saf 2019; 42:1071–1080.<br />Lindsay ME, de Oliveira S, Sciacca K, et al.: Harnessing natural language processing to assess quality of end-of-life care for children with cancer. JCO Clin Cancer Inform 2024; 8:e2400134.<br />Agaronnik ND, Davis J, Manz CR, et al.: Large language models to identify advance care planning in patients with advanced cancer. J Pain Symptom Manage 2025; 69:243–250.e1.<br />Lindvall C, Lilley EJ, Zupanc SN, et al.: Natural language processing to assess end-of-life quality indicators in cancer patients receiving palliative surgery. J Palliat Med 2019; 22:183–187.<br />Feudtner C, Feinstein JA, Zhong W, et al.: Pediatric complex chronic conditions classification system version 2: Updated for ICD-10 and complex medical technology dependence and transplantation. BMC Pediatr 2014; 14:199.<br />Feudtner C, Friebert S, Jewell J, et al.; Section on Hospice and Palliative Medicine and Committee on Hospital Care: Pediatric palliative care and hospice care commitments, guidelines, and recommendations. Pediatrics 2013; 132:966–972.<br />Derrington S, Broden Arciprete EG, Lin MC, et al.: Society of Critical Care Medicine 2026 guidelines on the care and management of pediatric and neonatal intensive care patients at the end of life. Pediatr Crit Care Med 2026; 27:518–536.<br />Lindvall Lab: Clinical Regex [computer program]. Boston, MA, Dana-Farber Cancer Institute, 2025. Available at: https://lindvalllab.github.io/clinical-regex/. Accessed January 28, 2026.<br />Agaronnik ND, Davis J, Manz CR, et al.: Feasibility study for using large language models to identify goals-of-care documentation at scale in patients with advanced cancer. JCO Oncol Pract 2026; 22:294–305.<br />Tooten S, Hermens RPMG, Verhoeven M, et al.: Pediatric advance care planning: A mixed-methods evaluation of documentation and sharing in current practice. BMC Palliat Care 2026; 25:51.
– Name: SubjectMinor
  Label: Contributed Indexing
  Group:
  Data: <i>Keywords: </i>advance care planning; critical care; natural language processing; out-of-hospital cardiac arrest; pediatrics
– Name: DateEntry
  Label: Entry Date(s)
  Group: Date
  Data: <i>Date Created: </i>20260925 <i>Date Completed: </i>20260925 <i>Latest Revision: </i>20260925
– Name: DateUpdate
  Label: Update Code
  Group: Date
  Data: 20260926
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1097/CCE.0000000000001494
– Name: AN
  Label: PMID
  Group: ID
  Data: 42789538
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=cmedm&AN=42789538
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1097/CCE.0000000000001494
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        StartPage: e1494
    Subjects:
      – SubjectFull: Humans
        Type: general
      – SubjectFull: Retrospective Studies
        Type: general
      – SubjectFull: Electronic Health Records
        Type: general
      – SubjectFull: Child
        Type: general
      – SubjectFull: Female
        Type: general
      – SubjectFull: Male
        Type: general
      – SubjectFull: Adolescent
        Type: general
      – SubjectFull: Child, Preschool
        Type: general
      – SubjectFull: Intensive Care Units, Pediatric
        Type: general
      – SubjectFull: Infant
        Type: general
      – SubjectFull: Advance Care Planning standards
        Type: general
      – SubjectFull: Advance Care Planning statistics & numerical data
        Type: general
      – SubjectFull: Critical Illness therapy
        Type: general
      – SubjectFull: Documentation methods
        Type: general
      – SubjectFull: Documentation standards
        Type: general
      – SubjectFull: Natural Language Processing
        Type: general
    Titles:
      – TitleFull: Validation of a Natural Language Processing-Assisted Chart Review Method for Identifying Advance Care Planning Documentation in Pediatric Critical Illness.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Upham EJ
      – PersonEntity:
          Name:
            NameFull: Gouda SR
      – PersonEntity:
          Name:
            NameFull: Lindvall C
      – PersonEntity:
          Name:
            NameFull: DeCourcey DD
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 25
              M: 09
              Text: 2026 Sep 25
              Type: published
              Y: 2026
          Identifiers:
            – Type: issn-electronic
              Value: 2639-8028
          Numbering:
            – Type: volume
              Value: 8
            – Type: issue
              Value: 10
          Titles:
            – TitleFull: Critical care explorations
              Type: main
ResultId 1