Academic Journal
Προβολή σε EDS

Embedding-Based Evaluation and Benchmarking Framework for Optimizing LLM Post-Processing of Medical Transcriptions from a Local Whisper System.

Λεπτομέρειες βιβλιογραφικής εγγραφής
Τίτλος: Embedding-Based Evaluation and Benchmarking Framework for Optimizing LLM Post-Processing of Medical Transcriptions from a Local Whisper System.
Συγγραφείς: Wattar A; Research Group (Bio-) Medical Data Science, Faculty of Medicine, Martin Luther University Halle-Wittenberg, Halle (Saale), Germany., Christoph J; Research Group (Bio-) Medical Data Science, Faculty of Medicine, Martin Luther University Halle-Wittenberg, Halle (Saale), Germany., Demus C; Research Group (Bio-) Medical Data Science, Faculty of Medicine, Martin Luther University Halle-Wittenberg, Halle (Saale), Germany., Liegel P; Research Group (Bio-) Medical Data Science, Faculty of Medicine, Martin Luther University Halle-Wittenberg, Halle (Saale), Germany., Jäger C; Research Group (Bio-) Medical Data Science, Faculty of Medicine, Martin Luther University Halle-Wittenberg, Halle (Saale), Germany.
Πηγή: Studies in health technology and informatics [Stud Health Technol Inform] 2026 Sep 17; Vol. 340, pp. 187-198.
Τύπος έκδοσης: Journal Article
Γλώσσα: English
Στοιχεία περιοδικού: Publisher: IOS Press Country of Publication: Netherlands NLM ID: 9214582 Publication Model: Print Cited Medium: Internet ISSN: 1879-8365 (Electronic) Linking ISSN: 09269630 NLM ISO Abbreviation: Stud Health Technol Inform Subsets: MEDLINE
Imprint Name(s): Original Publication: Amsterdam ; Washington, DC : IOS Press, 1991-
Ιατρικοί όροι (MeSH): Large Language Models* , Benchmarking* , Natural Language Processing* , Electronic Health Records*, Humans ; Semantics
Περίληψη: Introduction: Locally deployed speech-to-text systems such as Whisper enable privacy-preserving transcription of medical encounters. However, the resulting transcripts are often lengthy, noisy, and insufficiently structured for direct integration into clinical documentation workflows. Large language models (LLMs) can transform such transcripts into concise clinical summaries, yet evaluating their quality and reliability remains challenging, particularly in the absence of gold-standard reference summaries.
Materials and Methods: We propose a reference-free evaluation framework for benchmarking LLM-based post-processing of clinical transcripts generated by a locally deployed Whisper system. The framework combines embedding-based transcript-summary semantic similarity, stability analysis across repeated generations, and structured human evaluation. It was applied to 26 German physician-patient conversations. Four LLMs generated four summaries per transcript using standardized prompts and decoding parameters, resulting in 416 summaries. A subset of summaries was additionally assessed by three human raters using six predefined quality criteria.
Results: GPT-OSS-120B and MedGemma-27B achieved the highest transcript-summary similarity scores across most embedding models. Although absolute similarity values varied across embedding models, relative model rankings remained largely consistent, indicating robustness of the evaluation framework. Stability analysis showed high consistency across repeated runs, with cosine similarities typically exceeding 0.90, while higher sampling temperatures reduced semantic similarity. Human evaluation showed partial agreement between embedding-based similarity signals and human judgments of summary quality.
Conclusion: The proposed framework enables scalable, reference-free evaluation of LLM-based clinical summarization in privacy-sensitive settings. By combining semantic similarity, stability analysis, and human evaluation, it supports systematic model benchmarking, relative model comparison, and optimization without requiring reference summaries. These findings suggest that embedding-based metrics can provide useful signals for selecting and optimizing LLM-based post-processing models in local speech-to-text pipelines.
Contributed Indexing: Keywords: Artificial Intelligence; Benchmarking; Large Language Models; Medical Records; Natural Language Processing; Speech Recognition Software
Entry Date(s): Date Created: 20260917 Date Completed: 20260917 Latest Revision: 20260917
Update Code: 20260918
DOI: 10.3233/SHTI261001
PMID: 42752483
Βάση Δεδομένων: MEDLINE
FullText Text:
  Availability: 0
Header DbId: cmedm
DbLabel: MEDLINE
An: 42752483
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Embedding-Based Evaluation and Benchmarking Framework for Optimizing LLM Post-Processing of Medical Transcriptions from a Local Whisper System.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AU" term="%22Wattar+A%22">Wattar A</searchLink>; Research Group (Bio-) Medical Data Science, Faculty of Medicine, Martin Luther University Halle-Wittenberg, Halle (Saale), Germany.<br /><searchLink fieldCode="AU" term="%22Christoph+J%22">Christoph J</searchLink>; Research Group (Bio-) Medical Data Science, Faculty of Medicine, Martin Luther University Halle-Wittenberg, Halle (Saale), Germany.<br /><searchLink fieldCode="AU" term="%22Demus+C%22">Demus C</searchLink>; Research Group (Bio-) Medical Data Science, Faculty of Medicine, Martin Luther University Halle-Wittenberg, Halle (Saale), Germany.<br /><searchLink fieldCode="AU" term="%22Liegel+P%22">Liegel P</searchLink>; Research Group (Bio-) Medical Data Science, Faculty of Medicine, Martin Luther University Halle-Wittenberg, Halle (Saale), Germany.<br /><searchLink fieldCode="AU" term="%22Jäger+C%22">Jäger C</searchLink>; Research Group (Bio-) Medical Data Science, Faculty of Medicine, Martin Luther University Halle-Wittenberg, Halle (Saale), Germany.
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%229214582%22">Studies in health technology and informatics</searchLink> [Stud Health Technol Inform] 2026 Sep 17; Vol. 340, pp. 187-198.
– Name: TypePub
  Label: Publication Type
  Group: TypPub
  Data: Journal Article
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: TitleSource
  Label: Journal Info
  Group: Src
  Data: <i>Publisher: </i><searchLink fieldCode="PB" term="%22IOS+Press%22">IOS Press </searchLink><i>Country of Publication: </i>Netherlands <i>NLM ID: </i>9214582 <i>Publication Model: </i>Print <i>Cited Medium: </i>Internet <i>ISSN: </i>1879-8365 (Electronic) <i>Linking ISSN: </i><searchLink fieldCode="IS" term="%2209269630%22">09269630 </searchLink><i>NLM ISO Abbreviation: </i>Stud Health Technol Inform <i>Subsets: </i>MEDLINE
– Name: PublisherInfo
  Label: Imprint Name(s)
  Group: PubInfo
  Data: <i>Original Publication</i>: Amsterdam ; Washington, DC : IOS Press, 1991-
– Name: SubjectMESH
  Label: MeSH Terms
  Group: Su
  Data: <searchLink fieldCode="MM" term="%22Large+Language+Models%22">Large Language Models*</searchLink> <br /><searchLink fieldCode="MM" term="%22Benchmarking%22">Benchmarking*</searchLink> <br /><searchLink fieldCode="MM" term="%22Natural+Language+Processing%22">Natural Language Processing*</searchLink> <br /><searchLink fieldCode="MM" term="%22Electronic+Health+Records%22">Electronic Health Records*</searchLink><br /><searchLink fieldCode="MH" term="%22Humans%22">Humans</searchLink> ; <searchLink fieldCode="MH" term="%22Semantics%22">Semantics</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Introduction: Locally deployed speech-to-text systems such as Whisper enable privacy-preserving transcription of medical encounters. However, the resulting transcripts are often lengthy, noisy, and insufficiently structured for direct integration into clinical documentation workflows. Large language models (LLMs) can transform such transcripts into concise clinical summaries, yet evaluating their quality and reliability remains challenging, particularly in the absence of gold-standard reference summaries.<br />Materials and Methods: We propose a reference-free evaluation framework for benchmarking LLM-based post-processing of clinical transcripts generated by a locally deployed Whisper system. The framework combines embedding-based transcript-summary semantic similarity, stability analysis across repeated generations, and structured human evaluation. It was applied to 26 German physician-patient conversations. Four LLMs generated four summaries per transcript using standardized prompts and decoding parameters, resulting in 416 summaries. A subset of summaries was additionally assessed by three human raters using six predefined quality criteria.<br />Results: GPT-OSS-120B and MedGemma-27B achieved the highest transcript-summary similarity scores across most embedding models. Although absolute similarity values varied across embedding models, relative model rankings remained largely consistent, indicating robustness of the evaluation framework. Stability analysis showed high consistency across repeated runs, with cosine similarities typically exceeding 0.90, while higher sampling temperatures reduced semantic similarity. Human evaluation showed partial agreement between embedding-based similarity signals and human judgments of summary quality.<br />Conclusion: The proposed framework enables scalable, reference-free evaluation of LLM-based clinical summarization in privacy-sensitive settings. By combining semantic similarity, stability analysis, and human evaluation, it supports systematic model benchmarking, relative model comparison, and optimization without requiring reference summaries. These findings suggest that embedding-based metrics can provide useful signals for selecting and optimizing LLM-based post-processing models in local speech-to-text pipelines.
– Name: SubjectMinor
  Label: Contributed Indexing
  Group:
  Data: <i>Keywords: </i>Artificial Intelligence; Benchmarking; Large Language Models; Medical Records; Natural Language Processing; Speech Recognition Software
– Name: DateEntry
  Label: Entry Date(s)
  Group: Date
  Data: <i>Date Created: </i>20260917 <i>Date Completed: </i>20260917 <i>Latest Revision: </i>20260917
– Name: DateUpdate
  Label: Update Code
  Group: Date
  Data: 20260918
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.3233/SHTI261001
– Name: AN
  Label: PMID
  Group: ID
  Data: 42752483
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=cmedm&AN=42752483
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.3233/SHTI261001
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        StartPage: 187
    Subjects:
      – SubjectFull: Humans
        Type: general
      – SubjectFull: Semantics
        Type: general
      – SubjectFull: Large Language Models
        Type: general
      – SubjectFull: Benchmarking
        Type: general
      – SubjectFull: Natural Language Processing
        Type: general
      – SubjectFull: Electronic Health Records
        Type: general
    Titles:
      – TitleFull: Embedding-Based Evaluation and Benchmarking Framework for Optimizing LLM Post-Processing of Medical Transcriptions from a Local Whisper System.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Wattar A
      – PersonEntity:
          Name:
            NameFull: Christoph J
      – PersonEntity:
          Name:
            NameFull: Demus C
      – PersonEntity:
          Name:
            NameFull: Liegel P
      – PersonEntity:
          Name:
            NameFull: Jäger C
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 17
              M: 09
              Text: 2026 Sep 17
              Type: published
              Y: 2026
          Identifiers:
            – Type: issn-electronic
              Value: 1879-8365
          Numbering:
            – Type: volume
              Value: 340
          Titles:
            – TitleFull: Studies in health technology and informatics
              Type: main
ResultId 1