Academic Journal
Screenathon 2.0: human-AI collaborative screening applied to patient-generated health data.
| Τίτλος: | Screenathon 2.0: human-AI collaborative screening applied to patient-generated health data. |
|---|---|
| Συγγραφείς: | Bergmann J; Department of Methodology and Statistics, Faculty of Social and Behavioural Sciences, Utrecht University, Utrecht, The Netherlands., Azzi T; Department of Methodology and Statistics, Faculty of Social and Behavioural Sciences, Utrecht University, Utrecht, The Netherlands., Neeleman R; Department of Methodology and Statistics, Faculty of Social and Behavioural Sciences, Utrecht University, Utrecht, The Netherlands., Monschau K; Department of Methodology and Statistics, Faculty of Social and Behavioural Sciences, Utrecht University, Utrecht, The Netherlands., Yazan B; Department of Methodology and Statistics, Faculty of Social and Behavioural Sciences, Utrecht University, Utrecht, The Netherlands., Jalsovec E; Department of Methodology and Statistics, Faculty of Social and Behavioural Sciences, Utrecht University, Utrecht, The Netherlands., Westerbeek E; Department of Methodology and Statistics, Faculty of Social and Behavioural Sciences, Utrecht University, Utrecht, The Netherlands., Weijdema F; Utrecht University Library, Utrecht University, Utrecht, The Netherlands., de Bruin J; Department of Research and Data Management Services, Information Technology Services, Utrecht University, Utrecht, The Netherlands., Fang Q; Department of Methodology and Statistics, Faculty of Social and Behavioural Sciences, Utrecht University, Utrecht, The Netherlands., van de Schoot R; Department of Methodology and Statistics, Faculty of Social and Behavioural Sciences, Utrecht University, Utrecht, The Netherlands. a.g.j.vandeschoot@uu.nl. |
| Πηγή: | Scientific reports [Sci Rep] 2026 Mar 22; Vol. 16 (1). Date of Electronic Publication: 2026 Mar 22. |
| Τύπος έκδοσης: | Journal Article |
| Γλώσσα: | English |
| Στοιχεία περιοδικού: | Publisher: Nature Publishing Group Country of Publication: England NLM ID: 101563288 Publication Model: Electronic Cited Medium: Internet ISSN: 2045-2322 (Electronic) Linking ISSN: 20452322 NLM ISO Abbreviation: Sci Rep Subsets: MEDLINE |
| Imprint Name(s): | Original Publication: London : Nature Publishing Group, copyright 2011- |
| Ιατρικοί όροι (MeSH): | Patient Generated Health Data* , Artificial Intelligence*, Humans ; Machine Learning |
| Περίληψη: | Systematic reviews are essential for evidence-based research, yet the traditional screening process is time-consuming and difficult to scale. Human-only screening can introduce inconsistency, while fully automated approaches employing Large Language Models often lack the contextual judgement required for complex decisions. To address this, we introduce a crowd-based screening methodology that integrates human expertise with adaptive machine learning. The methods have been applied in the context of a large EU project where experts from 27 collaborating partners jointly screened 5842 papers across eleven disease topics related to patient-generated health data in a span of 2 days. Post-processing played a central role in ensuring data quality, including topic reallocation, targeted full-text verification, and noisy-label filtering. This Screenathon resulted in 487 records being labeled as relevant and 6,463 records as irrelevant. The number of records screened per participant ranged from 3 to 2496, with a mean of 216.4 records per screener (SE = 95.19). Exploratory analyses using survey results indicated increased trust in AI-assisted systematic reviewing after the event, along with generally positive evaluations of usability. The current Screenathon demonstrates that crowdsourced human–AI collaboration requires thoughtful training and calibration, together with strong post-processing safeguards. |
| Competing Interests: | Declarations. Competing interests: The authors declare no competing interests. Ethical approval and consent to participate: The Screenathon 2.0 protocol and data collection for the investigation were accepted by the Ethics Committee of the Faculty of Social and Behavioral Sciences of Utrecht University on 20 May 2025 (file no. 25–0241). All participants provided informed consent prior to participating in the study. We confirm that all experiments were performed in accordance with the relevant guidelines and regulations. Consent for publication: Not applicable. Generative AI usage: Large language models were not used to generate text for the present paper. They were, however, utilized during editing to correct grammar mistakes and improve the writing style. Furthermore, they were used to help write R code to create figures. After using these tools, the authors reviewed and edited the content as needed and take full responsibility for the content of the publication. |
| References: | PLoS One. 2020 Jan 14;15(1):e0227742. (PMID: 31935267) BMC Med Res Methodol. 2025 May 10;25(1):130. (PMID: 40349005) Sci Rep. 2024 Dec 28;14(1):31150. (PMID: 39730794) Syst Rev. 2023 Jun 20;12(1):100. (PMID: 37340494) Syst Rev. 2024 Feb 17;13(1):69. (PMID: 38368379) Syst Rev. 2015 Jan 01;4:1. (PMID: 25554246) BMC Med Res Methodol. 2020 Jan 13;20(1):7. (PMID: 31931747) Eur J Psychotraumatol. 2025 Dec;16(1):2546214. (PMID: 41090195) Sci Rep. 2022 Sep 02;12(1):14952. (PMID: 36056152) iScience. 2025 Sep 12;28(10):113559. (PMID: 41079628) Syst Rev. 2024 Jul 8;13(1):175. (PMID: 38978084) Nat Hum Behav. 2024 Dec;8(12):2293-2303. (PMID: 39468277) BMC Med Res Methodol. 2021 Apr 26;21(1):88. (PMID: 33906604) Patterns (N Y). 2025 Jul 03;6(7):101318. (PMID: 40926967) BMC Med Res Methodol. 2021 Dec 20;21(1):285. (PMID: 34930132) Sci Rep. 2026 Mar 22;16(1):. (PMID: 41865131) BMJ. 2021 Mar 29;372:n71. (PMID: 33782057) BMC Med Res Methodol. 2020 Oct 15;20(1):256. (PMID: 33059590) BMJ Open. 2017 Feb 27;7(2):e012545. (PMID: 28242767) J Eval Clin Pract. 2025 Apr;31(3):e70100. (PMID: 40290054) Expert Rev Pharmacoecon Outcomes Res. 2023 Jul-Dec;23(9):1049-1056. (PMID: 37573521) Comput Struct Biotechnol J. 2024 Oct 18;24:672-678. (PMID: 39534362) |
| Grant Information: | 101132847 Innovative Health Initative Joint Undertaking (IHI JU) |
| Contributed Indexing: | Keywords: Human-AI collaboration; Large-scale systematic review; Patient generated health data; Systematic literature screening |
| Entry Date(s): | Date Created: 20260322 Date Completed: 20260627 Latest Revision: 20260813 |
| Update Code: | 20260813 |
| PubMed Central ID: | PMC13149969 |
| DOI: | 10.1038/s41598-026-45385-5 |
| PMID: | 41865131 |
| Βάση Δεδομένων: | MEDLINE |
| ISSN: | 2045-2322 |
|---|---|
| DOI: | 10.1038/s41598-026-45385-5 |