Academic Journal
Optimising screening efficiency in evidence synthesis on health Technology: A simulation study using ASReview.
| Title: | Optimising screening efficiency in evidence synthesis on health Technology: A simulation study using ASReview. |
|---|---|
| Authors: | Dias de Oliveira JM; Bridge Laboratory, Federal University of Santa Catarina, Florianópolis, Brazil; Graduate Program in Dentistry, Federal University of Santa Catarina, Florianópolis, Brazil. Electronic address: julia.meller@bridge.ufsc.br., Mello AT; Bridge Laboratory, Federal University of Santa Catarina, Florianópolis, Brazil; Postgraduate Program in Nutrition, Federal University of Santa Catarina (UFSC), Florianópolis, Brazil. Electronic address: arthur.mello@bridge.ufsc.br., Scandolara DH; Bridge Laboratory, Federal University of Santa Catarina, Florianópolis, Brazil; Postgraduate Program in Engineering, Management and Knowledge Media, Florianópolis, Brazil. Electronic address: daniel.scandolara@bridge.ufsc.br., Celuppi IC; Bridge Laboratory, Federal University of Santa Catarina, Florianópolis, Brazil. Electronic address: ianka@bridge.ufsc.br., Corrêa Rampinelli VP; Bridge Laboratory, Federal University of Santa Catarina, Florianópolis, Brazil; Department of Public Health, Federal University of Santa Catarina, Araranguá, Brazil. Electronic address: vanessa.correa@bridge.ufsc.br., Wazlawick RS; Bridge Laboratory, Federal University of Santa Catarina, Florianópolis, Brazil; Department of Informatics and Statistics, Federal University of Santa Catarina, Florianópolis, Brazil. Electronic address: raul@bridge.ufsc.br., Dalmarco EM; Bridge Laboratory, Federal University of Santa Catarina, Florianópolis, Brazil; Department of Clinical Analysis, Federal University of Santa Catarina, Florianópolis, Brazil. Electronic address: dalmarco@bridge.ufsc.br. |
| Source: | International journal of medical informatics [Int J Med Inform] 2026 Sep 15; Vol. 218, pp. 106516. Date of Electronic Publication: 2026 Jun 03. |
| Publication Type: | Journal Article; Evidence Synthesis |
| Language: | English |
| Journal Info: | Publisher: Elsevier Science Ireland Ltd Country of Publication: Ireland NLM ID: 9711057 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 1872-8243 (Electronic) Linking ISSN: 13865056 NLM ISO Abbreviation: Int J Med Inform Subsets: MEDLINE |
| Imprint Name(s): | Original Publication: Shannon, Co. Clare, Ireland : Elsevier Science Ireland Ltd., c1997- |
| MeSH Terms: | Computer Simulation* , Biomedical Technology*, Humans ; Retrospective Studies ; Support Vector Machine |
| Abstract: | Objective: This study evaluated model-configuration and stopping-rule decisions when using active learning-based title-and-abstract screening in health technology evidence syntheses. Methods: We conducted retrospective simulations using seven pre-labelled datasets from systematic, scoping, and overview reviews in health technology. Simulations were implemented with ASReview Makita and compared lightweight configurations based on one-hot encoding or term frequency-inverse document frequency with naive Bayes, logistic regression, random forest, and support vector machine classifiers. Performance was evaluated using normalised recall regret ("loss"), work saved over sampling at 95% (WSS@95) and 100% recall (WSS@100), early recall, and K%-consecutive-irrelevant stopping rules. Repeated simulations and exploratory dataset-level analyses were conducted for the highest-ranked configuration. Results: SVM + TF-IDF (with bigrams) had the most favourable overall performance, with an average loss of 0.08 (95% CI 0.06 to 0.09), WSS@95 of 0.70 (95% CI 0.59 to 0.79), and WSS@100 of 0.50 (95% CI 0.30 to 0.69). At a fixed 7% consecutive-irrelevant stopping rule, all datasets reached at least 95% recall in the main analysis, with mean recall of 98%. In repeated simulations, the fixed 7% rule achieved mean recall of 97%; however, one very low-prevalence dataset did not reach 95% recall until K = 33%. Exploratory analyses suggested that relevant-record prevalence, textual similarity among relevant records, and abstract completeness may help explain variation in model performance and stopping-rule reliability, although these analyses were hypothesis-generating. Conclusion: Active learning-based screening reduced workload in these health technology datasets, but its use requires explicit implementation choices. SVM + TF-IDF (with bigrams) was the most pragmatic initial configuration, and a 7% consecutive-irrelevant rule was a useful stopping heuristic. However, stopping decisions should depend on the review's tolerance for missed studies, dataset quality, topic heterogeneity, and available safeguards, rather than on a fixed threshold alone. (Copyright © 2026 The Authors. Published by Elsevier B.V. All rights reserved.) |
| Competing Interests: | Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. |
| Contributed Indexing: | Keywords: ASReview; Computational simulation; Evidence-based health science; Systematic review |
| Entry Date(s): | Date Created: 20260603 Date Completed: 20260628 Latest Revision: 20260628 |
| Update Code: | 20260628 |
| DOI: | 10.1016/j.ijmedinf.2026.106516 |
| PMID: | 42235438 |
| Database: | MEDLINE |
Be the first to leave a comment!