Academic Journal
Προβολή σε EDS
Συνδεδεμένο Πλήρες Κείμενο

Can Large Language Models Be a Viable Tool for Consensus Working Groups? Experience of the Ventral Rectopexy Expert Consensus Group.

Λεπτομέρειες βιβλιογραφικής εγγραφής
Τίτλος: Can Large Language Models Be a Viable Tool for Consensus Working Groups? Experience of the Ventral Rectopexy Expert Consensus Group.
Συγγραφείς: Lee FG; Division of Colon and Rectal Surgery, Mayo Clinic, Rochester, Minnesota., Larson EL; Division of Colon and Rectal Surgery, Mayo Clinic, Rochester, Minnesota., Vermunt JV; Division of Colon and Rectal Surgery, Mayo Clinic, Rochester, Minnesota., Thiels CA; Division of Hepatobiliary and Pancreas Surgery, Mayo Clinic, Rochester, Minnesota., Christensen P; Department of Surgery, Aarhus University Hospital, Aarhus, Denmark., Collinson RJ; Colorectal Unit, Department of General Surgery, Te Toka Tumei Auckland City Hospital, Auckland, New Zealand., Cornish JA; Cardiff and Vale University Health Board, Cardiff, United Kingdom., D'Hoore A; Department of Abdominal Surgery, UZ Leuven, Leuven, Belgium., Gurland BH; Division of General Surgery, Stanford Medicine, Palo Alto, California., Mellgren A; Department of Surgical Oncology, King Faisal Specialist Hospital, Research Center, Riyadh, Saudi Arabia., Ratto C; Proctology and Pelvic Floor Surgery Unit, Catholic University, Ospedale Isola Tiberina - Gemelli Isola, Rome, Italy., Ris F; Division of Digestive Surgery, Hôpitaux Universitaires Genève, Geneva, Switzerland., Stevenson ARL; Colorectal Surgery Unit, Royal Brisbane and Women's Hospital, Brisbane, Queensland, Australia., Bordeianou LG; Division of Colorectal Surgery, Massachusetts General Hospital, Boston, Massachusetts., Perry WRG; Division of Colon and Rectal Surgery, Mayo Clinic, Rochester, Minnesota.
Συλλογικό Έργο: Ventral Rectopexy Consensus AI Group and the Ventral Rectopexy Working Group
Πηγή: Diseases of the colon and rectum [Dis Colon Rectum] 2026 May 01; Vol. 69 (5), pp. 836-846. Date of Electronic Publication: 2026 Jan 06.
Τύπος έκδοσης: Journal Article
Γλώσσα: English
Στοιχεία περιοδικού: Publisher: Lippincott Country of Publication: United States NLM ID: 0372764 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 1530-0358 (Electronic) Linking ISSN: 00123706 NLM ISO Abbreviation: Dis Colon Rectum Subsets: MEDLINE
Imprint Name(s): Original Publication: Philadelphia. Lippincott
Ιατρικοί όροι (MeSH): Rectal Prolapse*/surgery , Language*, Humans ; Consensus ; Large Language Models
Περίληψη: Background: The Ventral Rectopexy International Expert Panel recently published a consensus update on ventral rectopexy. The ability of large language models to synthesize the literature on ventral rectopexy without an explicit knowledge base was studied before the publication of the consensus.
Objective: To compare the responses and citations of different large language models on ventral rectopexy using the expert panel consensus as the reference standard.
Design: ChatGPT-4o, Gemini 1.5 Pro, and OpenEvidence were compared on content appropriateness (1 = inappropriate to 5 = appropriate), readability (Flesch reading ease), response length, citation fabrication, and citation quality per Oxford Levels of Evidence. The most content-appropriate chatbot response selected by the expert panel was de-identified and presented alongside the consensus text to 15 colorectal surgeons who attempted to identify the chatbot-generated text.
Settings: Questions were submitted from September 18 to 19, 2024. Analysis was performed before the online publication of the consensus on January 30, 2025.
Main Outcome Measures: The main outcome measures included content appropriateness, fabricated citation rate, citation quality, and the accuracy of identifying human versus chatbot text by colorectal surgeons.
Results: OpenEvidence ranked highest for content appropriateness (mean 3.5/5), above Gemini (3.0/5) and ChatGPT (2.8/5; p < 0.001). ChatGPT was most verbose with the highest readability scores ( p = 0.021). ChatGPT fabricated 53% of citations; Gemini fabricated 12%; OpenEvidence fabricated 0% ( p < 0.001). All OpenEvidence citations were peer reviewed, with 40 of 117 (34%) citing level I to III studies compared with 46 of 94 (49%) of the references cited in the consensus ( p = 0.043). Chatbot-generated responses were identified with 28 of 51 (55%) accuracy.
Limitations: Reproducibility may be affected due to the nature of large language models and the availability of the consensus after publication.
Conclusions: OpenEvidence outperformed Gemini 1.5 Pro and ChatGPT-4o in content appropriateness and peer-reviewed citation quantity and quality. Chatbot-generated text was indistinguishable from expert-authored consensus to subject matter experts. Large language models, as an early-stage research tool, may be viable for future consensus working groups, provided transparent disclosure and rigorous oversight. See Video Abstract .
Pueden Los Modelos Lingsticos De Gran Tamao Ser Una Herramienta Viable Para Los Grupos De Trabajo De Consenso La Experiencia Del Grupo De Consenso De Expertos En Rectopexia Ventral: ANTECEDENTES:El Panel Internacional de Expertos en Rectopexia Ventral publicó recientemente una actualización consensuada sobre la rectopexia ventral. Antes de la publicación del consenso, se estudió la capacidad de los grandes modelos lingüísticos para sintetizar la bibliografía sobre la rectopexia ventral sin una base de conocimientos explícita.Objetivo:Comparar las respuestas y citas de diferentes grandes modelos lingüísticos sobre la rectopexia ventral utilizando el consenso del panel de expertos como estándar de referencia.DISEÑO:Se compararon ChatGPT-4o, Gemini 1.5 Pro y OpenEvidence en cuanto a la idoneidad del contenido (1 = inadecuado a 5 = adecuado), la legibilidad (índice de legibilidad de Flesch), la longitud de la respuesta, la fabricación de citas y la calidad de las citas según los niveles de evidencia de Oxford. La respuesta del chatbot más adecuada en cuanto al contenido seleccionada por el panel de expertos se anonimizó y se presentó junto con el texto del consenso a 15 cirujanos colorrectales que intentaron identificar el texto generado por el chatbot.ENTORNO:Las preguntas se enviaron los días 18 y 19 de septiembre de 2024. El análisis se realizó antes de la publicación del consenso en línea el 30 de enero de 2025.PRINCIPALES MEDIDAS DE RESULTADOS:Adecuación del contenido, tasa de citas inventadas, calidad de las citas, precisión en la identificación del texto humano frente al del chatbot por parte de los cirujanos colorrectales.RESULTADOS:OpenEvidence obtuvo la puntuación más alta en cuanto a la adecuación del contenido (media de 3,5/5), por encima de Gemini (3,0/5) y ChatGPT (2,8/5) (p<0,001). ChatGPT fue el más prolijo, con las puntuaciones más altas en legibilidad (p = 0,021). ChatGPT inventó el 53% de las citas; Gemini inventó el 12 %; OpenEvidence inventó el 0 % (p < 0,001). Todas las citas de OpenEvidence fueron revisadas por pares, con 40/117 (34 %) que citaban estudios de nivel I-III frente a 46/94 (49 %) de las referencias citadas en el consenso (p = 0,043). Las respuestas generadas por el chatbot se identificaron con una precisión del 28/51 (55 %).LIMITACIONES:Reproducibilidad debido a la naturaleza de los modelos de lenguaje grandes y a la disponibilidad del consenso tras la publicación.CONCLUSIONES:OpenEvidence superó a Gemini 1.5 Pro y ChatGPT-4o en cuanto a la idoneidad del contenido y la cantidad y calidad de las citas revisadas por pares. El texto generado por el chatbot era indistinguible del consenso redactado por expertos para los expertos en la materia. Los modelos de lenguaje grandes como herramienta de investigación en fase inicial pueden ser viables para futuros grupos de trabajo de consenso, siempre que se garantice una divulgación transparente y una supervisión rigurosa. ( AI-generated translation ).
(Copyright © The ASCRS 2026.)
References: Tang L, Sun Z, Idnay B, et al. Evaluating large language models on medical evidence summarization. NPJ Digit Med. 2023;6:158.
Van Veen D, Van Uden C, Blankemeier L, et al. Adapted large language models can outperform medical experts in clinical text summarization. Nat Med. 2024;30:1134–1142.
Scherbakov D, Hubig N, Jansari V, Bakumenko A, Lenert LA. The emergence of large language models as tools in literature reviews: a large language model-assisted systematic review. J Am Med Inform Assoc. 2025;32:1071–1086.
Gao CA, Howard FM, Markov NS, et al. Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewers. NPJ Digit Med. 2023;6:75.
Liang W, Zhang Y, Cao H, et al. Can large language models provide useful feedback on research papers? A large-scale empirical analysis. NEJM AI. 2024;1:AIoa2400196.
Soroush A, Glicksberg BS, Zimlichman E, et al. Large language models are poor medical coders—benchmarking of medical code querying. NEJM AI. 2024;1:AIdbp2300040.
Nievas M, Basu A, Wang Y, Singh H. Distilling large language models for matching patients to clinical trials. J Am Med Inform Assoc. 2024;31:1953–1963.
Liu S, Wright AP, McCoy AB, Huang SS, Steitz B, Wright A. Detecting emergencies in patient portal messages using large language models and knowledge graph-based retrieval-augmented generation. J Am Med Inform Assoc. 2025;32:1032–1039.
Kung TH, Cheatham M, Medenilla A, et al. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLOS Digit Health. 2023;2:e0000198.
McGowan A, Gui Y, Dobbs M, et al. ChatGPT and Bard exhibit spontaneous citation fabrication during psychiatry literature search. Psychiatry Res. 2023;326:115334.
Perry WRG, Christensen P, Collinson RJ, et al.; Ventral Rectopexy Working Group. Ventral rectopexy: an international expert panel consensus and review of contemporary literature. Dis Colon Rectum. 2025;68:593–607.
Flesch RA. New readability yardstick. J Appl Psychol. 1948;32:221–233.
Oxford Centre for Evidence-Based Medicine. The Oxford levels of evidence 2.1. https://www.cebm.ox.ac.uk/resources/levels-of-evidence/ocebm-levels-of-evidence . 2011. Accessed June 3, 2025.
James LR, Demaree RG, Wolf G. rwg: an assessment of within-group interrater agreement. J Appl Psychol. 1993;78:306–309.
Fleiss JL. Measuring nominal scale agreement among many raters. Psychol Bull. 1971;76:378–382.
Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–174.
Liu S, McCoy AB, Wright A. Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines. J Am Med Inform Assoc. 2025;32:605–615.
Gargari OK, Habibi G. Enhancing medical AI with retrieval-augmented generation: a mini narrative review. Digit Health. 2025;11:20552076251337177.
Cohen F, Vallimont J, Gelfand AA. Caution regarding fabricated citations from artificial intelligence. Headache. 2024;64:3–4.
Borello-France DF, Handa VL, Brown MB, et al.; Pelvic Floor Disorders Network. Pelvic-floor muscle function in women with pelvic organ prolapse. Phys Ther. 2007;87:399–407.
Walters WH, Wilder EI. Fabrication and errors in the bibliographic citations generated by ChatGPT. Sci Rep. 2023;13:14045.
Chen A, Chen DO. Accuracy of chatbots in citing journal articles. JAMA Netw Open. 2023;6:e2327647.
Gravel J, D’Amours-Gravel M, Osmanlliu E. Learning to fake it: limited responses and fabricated references provided by ChatGPT for medical questions. Mayo Clin Proc Digit Health. 2023;1:226–234.
Hoffer EK. ChatGPT provides references that are real, inappropriate, or (most often) fake. J Vasc Interv Radiol. 2023;34:2240–2242.
Hueber AJ, Kleyer A. Quality of citation data using the natural language processing tool ChatGPT in rheumatology: creation of false references. RMD Open. 2023;9:e003248.
Wagner MW, Ertl-Wagner BB. Accuracy of information and references using ChatGPT-3 for retrieval of clinical radiological information. Can Assoc Radiol J. 2024;75:69–73.
Wong J, Kriegler C, Shrivastava A, Duimering A, Le C. Utility of chatbot literature search in radiation oncology. J Cancer Educ. 2025;40:589–593.
Moskatel LS, Zhang N. Comparative prevalence and characteristics of fabricated citations in large language models in headache medicine. Headache. 2024;64:93–95.
Institute of Medicine Committee on Standards for Developing Trustworthy Clinical Practice Guidelines. Clinical Practice Guidelines We Can Trust. Washington, DC: National Academies Press; 2011.
Shekar S, Pataranutaporn P, Sarabu C, Cecchi GA, Maes P. People overtrust AI-generated medical advice despite low accuracy. NEJM AI. 2025;2:AIoa2300015.
Pan ET, Florian-Rodriguez M. Human vs machine: identifying ChatGPT-generated abstracts in gynecology and urogynecology. Am J Obstet Gynecol. 2024;231:276.e1–276.e10.
Holland AM, Lorenz WR, Cavanagh JC, et al. Comparison of medical research abstracts written by surgical trainees and senior surgeons or generated by large language models. JAMA Netw Open. 2024;7:e2425373.
Helgeson SA, Johnson PW, Gopikrishnan N, et al. Human reviewers’ ability to differentiate human-authored or artificial intelligence-generated medical manuscripts: a randomized survey study. Mayo Clin Proc. 2025;100:622–633.
Kwon D. Scientists split on ethics of AI use. Nature. 2025;641:574–578.
Shool S, Adimi S, Saboori Amleshi R, Bitaraf E, Golpira R, Tara M. A systematic review of large language model (LLM) evaluations in clinical medicine. BMC Med Inform Decis Mak. 2025;25:117.
Gupta S, Tarapore R, Haislup B, Fillar A. Microsoft Copilot provides more accurate and reliable information about anterior cruciate ligament injury and repair than ChatGPT and Google Gemini; however, no resource was overall the best. Arthrosc Sports Med Rehabil. 2024;7:101043.
Davis J, Van Bulck L, Durieux BN, Lindvall C. The temperature feature of ChatGPT: modifying creativity for clinical research. JMIR Hum Factors. 2024;11:e53559.
Grant Information: T32 DK007198 United States DK NIDDK NIH HHS
Contributed Indexing: Investigator: M Adamina, Cantonal Hospital Winterthur, University of Basel, Basel, Switzerland.; S Ascanelli, University of Ferrara, Ferrara, Italy.; CIM Baeten, Groene Hart Hospital, Gouda, the Netherlands.; ECJ Consten, University Medical Centre Groningen, Groningen, Netherlands.; L Crepaz, San Camillo Hospital, Trento, Italy.; KJ Gorissen, Oxford University Hospitals, Oxford, United Kingdom.; W Lai, Derriford Hospital, Plymouth, United Kingdom.; AM McClure, St. Joseph Mercy Hospital, Ann Arbor, Michigan.; G Meurette, Geneva University Hospital, Geneva, Switzerland.; G Naldini, Cisanello University Hospital, Pisa, Italy.; S Rajaratnam, Te Whatu Ora - Health New Zealand Waitaha, North Shore, Auckland, New Zealand.; JR Speranza, University of Rochester Medical Center, Rochester, New York.; GP Thomas, St. Mark's Hospital, Harrow, United Kingdom.; AJ Thorsen, University of Minnesota, Minneapolis, Minnesota.; AM Warwick, University of Queensland, Brisbane, Queensland, Australia.; P Addison, Northwell Health, Staten Island, New York.; AA Bach-Nielsen, Aarhus University, Aarhus, Denmark.; LJ Convie, University of Queensland, Brisbane, Australia.; AA Marra, Catholic University, Rome, Italy.; Z Zhou, Stanford Medicine, Palo Alto, California.
Keywords: Artificial intelligence; Consensus; Knowledge discovery; Large language models; Rectal prolapse
Entry Date(s): Date Created: 20260106 Date Completed: 20260428 Latest Revision: 20260919
Update Code: 20260919
DOI: 10.1097/DCR.0000000000004058
PMID: 41492895
Βάση Δεδομένων: MEDLINE
Περιγραφή
ISSN:1530-0358
DOI:10.1097/DCR.0000000000004058