Academic Journal
Προβολή σε EDS

Practical Guide to Large Language Models for Information Extraction in Behavioral Health Notes: Tutorial.

Λεπτομέρειες βιβλιογραφικής εγγραφής
Τίτλος: Practical Guide to Large Language Models for Information Extraction in Behavioral Health Notes: Tutorial.
Συγγραφείς: Saha D; Department of Computer Science, University of California, Irvine, 6210 Donald Bren Hall, Irvine, CA, 92697, United States, 1 9498247427., Edgcomb JB; Center for Community Health, UCLA Mental Health Informatics & Data Science (MINDS) Hub, University of California, Los Angeles, Los Angeles, CA, United States.
Πηγή: JMIR mental health [JMIR Ment Health] 2026 Sep 24; Vol. 13, pp. e97498. Date of Electronic Publication: 2026 Sep 24.
Τύπος έκδοσης: Journal Article
Γλώσσα: English
Στοιχεία περιοδικού: Publisher: JMIR Publications Inc Country of Publication: Canada NLM ID: 101658926 Publication Model: Electronic Cited Medium: Internet ISSN: 2368-7959 (Electronic) Linking ISSN: 23687959 NLM ISO Abbreviation: JMIR Ment Health Subsets: MEDLINE
Imprint Name(s): Original Publication: Toronto : JMIR Publications Inc., [2014]-
Ιατρικοί όροι (MeSH): Information Storage and Retrieval*/methods , Large Language Models* , Electronic Health Records* , Natural Language Processing*, Humans
Περίληψη: Background: Mental health clinical notes contain decision-critical information often absent from structured electronic health record fields. Large language models (LLMs) can extract clinically relevant signals from narrative text; however, variability in output format, limited reproducibility, and inconsistent evaluation remain barriers to clinical deployment. Despite rapid advances in LLM-based information extraction, clear and reproducible guidance for interdisciplinary clinical teams is limited.
Objective: This tutorial aims to present a structured workflow for zero-shot information extraction from mental health clinical notes using locally deployed open-source LLMs. It aims to reduce barriers for clinicians and researchers with limited familiarity with natural language processing (NLP) or LLM-based pipelines. Each stage includes key decision points and examples. The workflow is illustrated on two tasks using synthetic notes: (1) detection of self-injurious thoughts and behaviors (SITB) in pediatric emergency department (ED) notes and (2) antipsychotic medication nonadherence detection in outpatient notes, using schema-constrained outputs and standardized evaluation.
Methods: We describe a five-stage zero-shot LLM pipeline: (1) infrastructure setup with local deployment via Ollama to prevent protected health information (PHI) transmission; (2) task definition specifying the clinical construct, output format, and evaluation; (3) dataset preparation using synthetic notes; (4) iterative prompt development using a hold-out development set with binary and Likert scale outputs constrained via JSON schemas; and (5) output parsing, normalization, and validation. We generated 300 synthetic notes per task using separate LLMs for generation and evaluation; 200 notes were used for evaluation, and 100 notes (50 positive and 50 negative) were used as a prompt-development set and excluded from final metrics. Evaluation used Large Language Model Meta AI (Llama) 3.2 and Llama 3.3 with deterministic decoding (temperature=0). Performance was assessed using accuracy, precision, recall, and F1-score; Likert thresholds were optimized using the Youden index with bootstrapped CIs.
Results: We demonstrated the pipeline's functionality using 2 example behavioral health detection tasks. Across both examples, the more capable model (Llama 3.3) performed better than the lighter model used earlier in development (Llama 3.2), and we described how the pipeline's evaluation and error-analysis steps work in practice. These examples also illustrated 2 useful design choices: requiring the model to output in a fixed format reduced errors, and using a graded rating scale, rather than a simple yes/no format, allowed the detection threshold to be adjusted based on clinical risk tolerance. These results are meant to show that the pipeline works as intended, not to serve as a benchmark of real-world accuracy.
Conclusions: A schema-driven, zero-shot LLM workflow can support reproducible extraction of clinically relevant information from narrative notes. Local deployment enables processing without transmitting PHI to external servers. This tutorial provides a transferable methodology for institutional adaptation and validation prior to clinical use. All prompts, code, and datasets are publicly available via Zenodo (European Organization for Nuclear Research [CERN]).
(© Diya Saha, Juliet B Edgcomb. Originally published in JMIR Mental Health (https://mental.jmir.org).)
References: Br J Psychiatry. 2024 Dec;225(6):532-537. (PMID: 39497458)
BMC Med Inform Decis Mak. 2025 Jan 13;25(1):20. (PMID: 39806393)
NPJ Digit Med. 2024 Sep 28;7(1):258. (PMID: 39333376)
J Med Internet Res. 2025 Jan 23;27:e63126. (PMID: 39847414)
Biom J. 2005 Aug;47(4):458-72. (PMID: 16161804)
Am J Psychiatry. 2007 Jul;164(7):1035-43. (PMID: 17606655)
NPJ Precis Oncol. 2025 Sep 17;9(1):313. (PMID: 40962856)
J Med Internet Res. 2022 May 4;24(5):e30405. (PMID: 35507393)
J Mood Anxiety Disord. 2025 Jan 31;10:100109. (PMID: 40657592)
Sci Rep. 2025 Nov 10;15(1):39231. (PMID: 41213985)
Sci Rep. 2025 Nov 24;15(1):45239. (PMID: 41286063)
J Am Med Inform Assoc. 2020 Mar 1;27(3):491-497. (PMID: 31682262)
JMIR Ment Health. 2026 Feb 04;13:e82371. (PMID: 41637484)
J Med Internet Res. 2025 May 5;27:e69284. (PMID: 40324177)
J Am Med Inform Assoc. 2025 Jan 1;32(1):218-226. (PMID: 39423850)
JMIR Ment Health. 2024 Jul 23;11:e57306. (PMID: 39042893)
Contributed Indexing: Keywords: clinical notes; electronic health records; information extraction; large language models; medication adherence; natural language processing; structured outputs; suicide risk; zero-shot learning
Entry Date(s): Date Created: 20260924 Date Completed: 20260924 Latest Revision: 20260927
Update Code: 20260927
PubMed Central ID: PMC13614219
DOI: 10.2196/97498
PMID: 42784808
Βάση Δεδομένων: MEDLINE
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://resolver.ebsco.com/c/fiv2js/result?sid=EBSCO:cmedm&genre=article&issn=23687959&ISBN=&volume=13&issue=&date=20260924&spage=e97498&pages=e97498&title=JMIR mental health&atitle=Practical%20Guide%20to%20Large%20Language%20Models%20for%20Information%20Extraction%20in%20Behavioral%20Health%20Notes%3A%20Tutorial.&aulast=Saha%20D&id=DOI:10.2196/97498
    Name: Full Text Finder (for New FTF UI) (ns324271)
    Category: fullText
    Text: Full Text Finder
    MouseOverText: Full Text Finder
Header DbId: cmedm
DbLabel: MEDLINE
An: 42784808
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Practical Guide to Large Language Models for Information Extraction in Behavioral Health Notes: Tutorial.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AU" term="%22Saha+D%22">Saha D</searchLink>; Department of Computer Science, University of California, Irvine, 6210 Donald Bren Hall, Irvine, CA, 92697, United States, 1 9498247427.<br /><searchLink fieldCode="AU" term="%22Edgcomb+JB%22">Edgcomb JB</searchLink>; Center for Community Health, UCLA Mental Health Informatics & Data Science (MINDS) Hub, University of California, Los Angeles, Los Angeles, CA, United States.
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%22101658926%22">JMIR mental health</searchLink> [JMIR Ment Health] 2026 Sep 24; Vol. 13, pp. e97498. <i>Date of Electronic Publication: </i>2026 Sep 24.
– Name: TypePub
  Label: Publication Type
  Group: TypPub
  Data: Journal Article
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: TitleSource
  Label: Journal Info
  Group: Src
  Data: <i>Publisher: </i><searchLink fieldCode="PB" term="%22JMIR+Publications+Inc%22">JMIR Publications Inc </searchLink><i>Country of Publication: </i>Canada <i>NLM ID: </i>101658926 <i>Publication Model: </i>Electronic <i>Cited Medium: </i>Internet <i>ISSN: </i>2368-7959 (Electronic) <i>Linking ISSN: </i><searchLink fieldCode="IS" term="%2223687959%22">23687959 </searchLink><i>NLM ISO Abbreviation: </i>JMIR Ment Health <i>Subsets: </i>MEDLINE
– Name: PublisherInfo
  Label: Imprint Name(s)
  Group: PubInfo
  Data: <i>Original Publication</i>: Toronto : JMIR Publications Inc., [2014]-
– Name: SubjectMESH
  Label: MeSH Terms
  Group: Su
  Data: <searchLink fieldCode="MM" term="%22Information+Storage+and+Retrieval%22">Information Storage and Retrieval*</searchLink>/<searchLink fieldCode="MM" term="%22Information+Storage+and+Retrieval+methods%22">methods</searchLink> <br /><searchLink fieldCode="MM" term="%22Large+Language+Models%22">Large Language Models*</searchLink> <br /><searchLink fieldCode="MM" term="%22Electronic+Health+Records%22">Electronic Health Records*</searchLink> <br /><searchLink fieldCode="MM" term="%22Natural+Language+Processing%22">Natural Language Processing*</searchLink><br /><searchLink fieldCode="MH" term="%22Humans%22">Humans</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Background: Mental health clinical notes contain decision-critical information often absent from structured electronic health record fields. Large language models (LLMs) can extract clinically relevant signals from narrative text; however, variability in output format, limited reproducibility, and inconsistent evaluation remain barriers to clinical deployment. Despite rapid advances in LLM-based information extraction, clear and reproducible guidance for interdisciplinary clinical teams is limited.<br />Objective: This tutorial aims to present a structured workflow for zero-shot information extraction from mental health clinical notes using locally deployed open-source LLMs. It aims to reduce barriers for clinicians and researchers with limited familiarity with natural language processing (NLP) or LLM-based pipelines. Each stage includes key decision points and examples. The workflow is illustrated on two tasks using synthetic notes: (1) detection of self-injurious thoughts and behaviors (SITB) in pediatric emergency department (ED) notes and (2) antipsychotic medication nonadherence detection in outpatient notes, using schema-constrained outputs and standardized evaluation.<br />Methods: We describe a five-stage zero-shot LLM pipeline: (1) infrastructure setup with local deployment via Ollama to prevent protected health information (PHI) transmission; (2) task definition specifying the clinical construct, output format, and evaluation; (3) dataset preparation using synthetic notes; (4) iterative prompt development using a hold-out development set with binary and Likert scale outputs constrained via JSON schemas; and (5) output parsing, normalization, and validation. We generated 300 synthetic notes per task using separate LLMs for generation and evaluation; 200 notes were used for evaluation, and 100 notes (50 positive and 50 negative) were used as a prompt-development set and excluded from final metrics. Evaluation used Large Language Model Meta AI (Llama) 3.2 and Llama 3.3 with deterministic decoding (temperature=0). Performance was assessed using accuracy, precision, recall, and F<subscript>1</subscript>-score; Likert thresholds were optimized using the Youden index with bootstrapped CIs.<br />Results: We demonstrated the pipeline's functionality using 2 example behavioral health detection tasks. Across both examples, the more capable model (Llama 3.3) performed better than the lighter model used earlier in development (Llama 3.2), and we described how the pipeline's evaluation and error-analysis steps work in practice. These examples also illustrated 2 useful design choices: requiring the model to output in a fixed format reduced errors, and using a graded rating scale, rather than a simple yes/no format, allowed the detection threshold to be adjusted based on clinical risk tolerance. These results are meant to show that the pipeline works as intended, not to serve as a benchmark of real-world accuracy.<br />Conclusions: A schema-driven, zero-shot LLM workflow can support reproducible extraction of clinically relevant information from narrative notes. Local deployment enables processing without transmitting PHI to external servers. This tutorial provides a transferable methodology for institutional adaptation and validation prior to clinical use. All prompts, code, and datasets are publicly available via Zenodo (European Organization for Nuclear Research [CERN]).<br /> (© Diya Saha, Juliet B Edgcomb. Originally published in JMIR Mental Health (https://mental.jmir.org).)
– Name: Ref
  Label: References
  Group: RefInfo
  Data: Br J Psychiatry. 2024 Dec;225(6):532-537. (PMID: <searchLink fieldCode="PM" term="%2239497458%22">39497458)</searchLink><br />BMC Med Inform Decis Mak. 2025 Jan 13;25(1):20. (PMID: <searchLink fieldCode="PM" term="%2239806393%22">39806393)</searchLink><br />NPJ Digit Med. 2024 Sep 28;7(1):258. (PMID: <searchLink fieldCode="PM" term="%2239333376%22">39333376)</searchLink><br />J Med Internet Res. 2025 Jan 23;27:e63126. (PMID: <searchLink fieldCode="PM" term="%2239847414%22">39847414)</searchLink><br />Biom J. 2005 Aug;47(4):458-72. (PMID: <searchLink fieldCode="PM" term="%2216161804%22">16161804)</searchLink><br />Am J Psychiatry. 2007 Jul;164(7):1035-43. (PMID: <searchLink fieldCode="PM" term="%2217606655%22">17606655)</searchLink><br />NPJ Precis Oncol. 2025 Sep 17;9(1):313. (PMID: <searchLink fieldCode="PM" term="%2240962856%22">40962856)</searchLink><br />J Med Internet Res. 2022 May 4;24(5):e30405. (PMID: <searchLink fieldCode="PM" term="%2235507393%22">35507393)</searchLink><br />J Mood Anxiety Disord. 2025 Jan 31;10:100109. (PMID: <searchLink fieldCode="PM" term="%2240657592%22">40657592)</searchLink><br />Sci Rep. 2025 Nov 10;15(1):39231. (PMID: <searchLink fieldCode="PM" term="%2241213985%22">41213985)</searchLink><br />Sci Rep. 2025 Nov 24;15(1):45239. (PMID: <searchLink fieldCode="PM" term="%2241286063%22">41286063)</searchLink><br />J Am Med Inform Assoc. 2020 Mar 1;27(3):491-497. (PMID: <searchLink fieldCode="PM" term="%2231682262%22">31682262)</searchLink><br />JMIR Ment Health. 2026 Feb 04;13:e82371. (PMID: <searchLink fieldCode="PM" term="%2241637484%22">41637484)</searchLink><br />J Med Internet Res. 2025 May 5;27:e69284. (PMID: <searchLink fieldCode="PM" term="%2240324177%22">40324177)</searchLink><br />J Am Med Inform Assoc. 2025 Jan 1;32(1):218-226. (PMID: <searchLink fieldCode="PM" term="%2239423850%22">39423850)</searchLink><br />JMIR Ment Health. 2024 Jul 23;11:e57306. (PMID: <searchLink fieldCode="PM" term="%2239042893%22">39042893)</searchLink>
– Name: SubjectMinor
  Label: Contributed Indexing
  Group:
  Data: <i>Keywords: </i>clinical notes; electronic health records; information extraction; large language models; medication adherence; natural language processing; structured outputs; suicide risk; zero-shot learning
– Name: DateEntry
  Label: Entry Date(s)
  Group: Date
  Data: <i>Date Created: </i>20260924 <i>Date Completed: </i>20260924 <i>Latest Revision: </i>20260927
– Name: DateUpdate
  Label: Update Code
  Group: Date
  Data: 20260927
– Name: PubmedCentralID
  Label: PubMed Central ID
  Group: ID
  Data: PMC13614219
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.2196/97498
– Name: AN
  Label: PMID
  Group: ID
  Data: 42784808
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=cmedm&AN=42784808
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.2196/97498
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        StartPage: e97498
    Subjects:
      – SubjectFull: Humans
        Type: general
      – SubjectFull: Information Storage and Retrieval methods
        Type: general
      – SubjectFull: Large Language Models
        Type: general
      – SubjectFull: Electronic Health Records
        Type: general
      – SubjectFull: Natural Language Processing
        Type: general
    Titles:
      – TitleFull: Practical Guide to Large Language Models for Information Extraction in Behavioral Health Notes: Tutorial.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Saha D
      – PersonEntity:
          Name:
            NameFull: Edgcomb JB
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 24
              M: 09
              Text: 2026 Sep 24
              Type: published
              Y: 2026
          Identifiers:
            – Type: issn-electronic
              Value: 2368-7959
          Numbering:
            – Type: volume
              Value: 13
          Titles:
            – TitleFull: JMIR mental health
              Type: main
ResultId 1