Academic Journal
View in EDS

High replicability of newly discovered social-behavioural findings is achievable.

Bibliographic Details
Title: High replicability of newly discovered social-behavioural findings is achievable.
Authors: Protzko J; Department of Psychological & Brain Sciences, University of California, Santa Barbara, Santa Barbara, CA, USA. protzko@gmail.com.; Department of Psychological Science, Central Connecticut State University, New Britain, CT, USA. protzko@gmail.com., Krosnick J; Institute for Research in the Social Sciences, Stanford University, Stanford, CA, USA., Nelson L; Haas School of Business, University of California, Berkeley, Berkeley, CA, USA., Nosek BA; Center for Open Science, Charlottesville, VA, USA.; Department of Psychology, University of Virginia, Charlottesville, VA, USA., Axt J; Department of Psychology, McGill University, Montreal, Quebec, Canada., Berent M; Matt Berent Consulting, Sharon, PA, USA., Buttrick N; Department of Psychology, University of Wisconsin-Madison, Madison, WI, USA., DeBell M; Institute for Research in the Social Sciences, Stanford University, Stanford, CA, USA., Ebersole CR; Department of Psychology, University of Virginia, Charlottesville, VA, USA., Lundmark S; SOM Institute, University of Gothenburg, Gothenburg, Sweden., MacInnis B; Institute for Research in the Social Sciences, Stanford University, Stanford, CA, USA., O'Donnell M; McDonough School of Business, Georgetown University, Washington, DC, USA., Perfecto H; Olin School of Business, Washington University in St. Louis, St. Louis, MO, USA., Pustejovsky JE; Educational Psychology Department, University of Wisconsin-Madison, Madison, WI, USA., Roeder SS; Darla Moore School of Business, University of South Carolina, Columbia, SC, USA., Walleczek J; Phenoscience Laboratories, Berlin, Germany., Schooler JW; Department of Psychological & Brain Sciences, University of California, Santa Barbara, Santa Barbara, CA, USA.
Source: Nature human behaviour [Nat Hum Behav] 2024 Feb; Vol. 8 (2), pp. 311-319. Date of Electronic Publication: 2023 Nov 09.
Publication Type: Journal Article; Retracted Publication
Language: English
Journal Info: Publisher: Springer Nature Publishing Country of Publication: England NLM ID: 101697750 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 2397-3374 (Electronic) Linking ISSN: 23973374 NLM ISO Abbreviation: Nat Hum Behav Subsets: MEDLINE
Imprint Name(s): Original Publication: [London] : Springer Nature Publishing, [2017]-
MeSH Terms: Research Design* , Social Behavior*, Humans ; Prospective Studies ; Reproducibility of Results ; Sample Size
Abstract: Failures to replicate evidence of new discoveries have forced scientists to ask whether this unreliability is due to suboptimal implementation of methods or whether presumptively optimal methods are not, in fact, optimal. This paper reports an investigation by four coordinated laboratories of the prospective replicability of 16 novel experimental findings using rigour-enhancing practices: confirmatory tests, large sample sizes, preregistration and methodological transparency. In contrast to past systematic replication efforts that reported replication rates averaging 50%, replication attempts here produced the expected effects with significance testing (P < 0.05) in 86% of attempts, slightly exceeding the maximum expected replicability based on observed effect sizes and sample sizes. When one lab attempted to replicate an effect discovered by another lab, the effect size in the replications was 97% that in the original study. This high replication rate justifies confidence in rigour-enhancing methods to increase the replicability of new discoveries.
(© 2023. The Author(s).)
Comments: Retraction in: Nat Hum Behav. 2024 Oct;8(10):2067. doi: 10.1038/s41562-024-01997-3.. (PMID: 39317794)
References: Putnam, H. Mathematics, Matter and Method: Volume 1, Philosophical Papers (CUP Archive, 1975).
Schmidt, S. Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Rev. Gen. Psychol. 13, 90–100 (2009). (PMID: 10.1037/a0015108)
Begley, C. G. & Ellis, L. M. Drug development: raise standards for preclinical cancer research. Nature 483, 531–533 (2012). (PMID: 10.1038/483531a22460880)
Camerer, C. F. et al. Evaluating replicability of laboratory experiments in economics. Science 351, 1433–1436 (2016). (PMID: 10.1126/science.aaf091826940865)
Camerer, C. F. et al. Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nat. Hum. Behav. 2, 637–644 (2018). (PMID: 10.1038/s41562-018-0399-z31346273)
Open Science Collaboration.Estimating the reproducibility of psychological science. Science 349, aac4716 (2015). (PMID: 10.1126/science.aac4716)
Errington, T. M. et al. Investigating the replicability of preclinical cancer biology. eLife 10, e71601 (2021). (PMID: 10.7554/eLife.71601348740058651293)
Klein, R. A. et al. Many Labs 2: investigating variation in replicability across samples and settings. Adv. Methods Pract. Psychol. Sci. 1, 443–490 (2018). (PMID: 10.1177/2515245918810225)
Ebersole, C. R. et al. Many Labs 3: evaluating participant pool quality across the academic semester via replication. J. Exp. Soc. Psychol. 67, 68–82 (2016). (PMID: 10.1016/j.jesp.2015.10.012)
Wagenmakers, E. J. et al. Registered replication report: Strack, Martin, & Stepper (1988). Perspect. Psychol. Sci. 11, 917–928 (2016). (PMID: 10.1177/174569161667445827784749)
O’Donnell, M. et al. Registered replication report: Dijksterhuis and van Knippenberg (1998). Perspect. Psychol. Sci. 13, 268–294 (2018). (PMID: 10.1177/174569161875570429463182)
Piovesan, M., Døssing, F. S. & Wengström, E. R. Registered replication report: Rand, Greene & Nowak (2012). Perspect. Psychol. Sci. 12, 527–542 (2017). (PMID: 10.1177/1745691617693624284754675453400)
Verschuere, B. et al. Registered replication report on Mazar, Amir, and Ariely (2008). Adv. Methods Pract. Psychol. Sci. 1, 299–317 (2018). (PMID: 10.1177/2515245918781032)
Gilbert, D. T., King, G., Pettigrew, S. & Wilson, T. D. Comment on ‘Estimating the reproducibility of psychological science’. Science 351, 1037 (2016). (PMID: 10.1126/science.aad724326941311)
Wilson, B. M., Harris, C. R. & Wixted, J. T. Science is not a signal detection problem. Proc. Natl Acad. Sci. USA 117, 5559–5567 (2020). (PMID: 10.1073/pnas.1914237117321274777084063)
Reich, E. S. Timing glitches dog neutrino claim. Nat. News 483, 17 (2012). (PMID: 10.1038/483017a)
Simmons, J. P., Nelson, L. D. & Simonsohn, U. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359–1366 (2011). (PMID: 10.1177/095679761141763222006061)
Van Bavel, J. J., Mende-Siedlecki, P., Brady, W. J. & Reinero, D. A. Contextual sensitivity in scientific reproducibility. Proc. Natl Acad. Sci. USA 113, 6454–6459 (2016). (PMID: 10.1073/pnas.1521897113272175564988618)
Hedges, L. V. & Schauer, J. M. Consistency of effects is important in replication: rejoinder to Mathur and VanderWeele (2019). Psychol. Methods 24, 576–577 (2019). (PMID: 10.1037/met000023731580142)
Glymour, C., Scheines, R. & Spirtes, P. Discovering Causal Structure: Artificial Intelligence, Philosophy of Science, and Statistical Modeling (Academic Press, 2014).
Protzko, J. Null-hacking, a lurking problem. Preprint at PsyArXiv https://doi.org/10.31234/osf.io/9y3mp (2018).
Bryan, C. J., Yeager, D. S. & O'Brien, J. Replicator degrees of freedom allow publication of misleading failures to replicate. Proc. Natl Acad. Sci USA 116, 25535–25545 (2019). (PMID: 10.1073/pnas.1910951116317677506925985)
Nosek, B. A. & Errington, T. M. What is replication? PLoS Biol. https://doi.org/10.1371/journal.pbio.3000691 (2020).
Schooler, J. Unpublished results hide the decline effect: some effects diminish when tests are repeated. Nature 470, 437–438 (2011). (PMID: 10.1038/470437a21350443)
Protzko, J. & Schooler, J. W. Decline effects: types, mechanisms, and personal reflections. In Psychological Science Under Scrutiny: Recent Challenges and Proposed Solutions (eds Lilienfeld, S. O. & Waldman, I. D.) 85–107 (Wiley Blackwell, 2017).
Yeager, D. S., Krosnick, J. A., Visser, P. S., Holbrook, A. L. & Tahk, A. M. Moderation of classic social psychological effects by demographics in the US adult population: new opportunities for theoretical advancement. J. Pers. Soc. Psychol. 117, e84–e99 (2019). (PMID: 10.1037/pspa0000171314644806918461)
Iso-Ahola, S. E. Reproducibility in psychological science: when do psychological phenomena exist? Front. Psychol. 8, 879 (2017). (PMID: 10.3389/fpsyg.2017.00879286264355454055)
Iso-Ahola, S. E. Replication and the establishment of scientific truth. Front. Psychol. 11, 2183 (2020). (PMID: 10.3389/fpsyg.2020.02183330418877525033)
Strack, F. & Stroebe, W. What have we learned? What can we learn? Behav. Brain Sci. 41, e151 (2018). (PMID: 10.1017/S0140525X1800087031064559)
Stroebe, W. & Strack, F. The alleged crisis and the illusion of exact replication. Perspect. Psychol. Sci. 9, 59–71 (2014). (PMID: 10.1177/174569161351445026173241)
Richard, F. D., Bond, C. F. Jr & Stokes-Zoota, J. J. One hundred years of social psychology quantitatively described. Rev. Gen. Psychol. 7, 331–363 (2003). (PMID: 10.1037/1089-2680.7.4.331)
Kvarven, A., Strømland, E. & Johannesson, M. Comparing meta-analyses and preregistered multiple-laboratory replication projects. Nat. Hum. Behav. https://doi.org/10.1038/s41562-019-0787-z (2019).
Schweinsberg, M. et al. The pipeline project: pre-publication independent replications of a single laboratory′s research pipeline. J. Exp. Soc. Psychol. 66, 55–67 (2016). (PMID: 10.1016/j.jesp.2015.10.001)
Schooler, J. W. Metascience could rescue the ‘replication crisis’. Nature 515, 9 (2014). (PMID: 10.1038/515009a25373639)
Protzko, J. & Schooler, J. W. No relationship between researcher impact and replication effect: an analysis of five studies with 100 replications. PeerJ 8, e8014 (2020). (PMID: 10.7717/peerj.8014322318687100597)
Wagenmakers, E. J., Wetzels, R., Borsboom, D., van der Maas, H. L. & Kievit, R. A. An agenda for purely confirmatory research. Perspect. Psychol. Sci. 7, 632–638 (2012). (PMID: 10.1177/174569161246307826168122)
Greenwald, A. G. Consequences of prejudice against the null hypothesis. Psychol. Bull. 82, 1–20 (1975). (PMID: 10.1037/h0076157)
Rosenthal, R. The file drawer problem and tolerance for null results. Psychol. Bull. 86, 638–641 (1979). (PMID: 10.1037/0033-2909.86.3.638)
Ernst, A. F., Hoekstra, R., Wagenmakers, E. J., Gelman, A. & van Ravenzwaaij, D. Do researchers anchor their beliefs on the outcome of an initial study? Testing the time-reversal heuristic. Exp. Psychol. 65, 158–169 (2018). (PMID: 10.1027/1618-3169/a00040229905114)
Tipton, E. & Pustejovsky, J. E. Small-sample adjustments for tests of moderators and model fit using robust variance estimation in meta-regression. J. Educ. Behav. Stat. 40, 604–634 (2015). (PMID: 10.3102/1076998615606099)
Konstantopoulos, S. Fixed effects and variance components estimation in three‐level meta‐analysis. Res. Synth. Methods 2, 61–76 (2011). (PMID: 10.1002/jrsm.3526061600)
Viechtbauer, W. Conducting meta-analyses in R with the metafor package. J. Stat. Softw. 36, 1–48 (2010). (PMID: 10.18637/jss.v036.i03)
R Core Team. R: a language and environment for statistical computing. R Foundation for Statistical Computing https://www.R-project.org/ (2022).
Protzko, J., Buttrick, N., Ebersole, C. R., Lundmark, S. & Schooler, J. Not all explanations are equal: only explanations invoking a change from the true self mitigate responsibility. Preprint at PsyArXiv https://doi.org/10.31234/osf.io/sx7cn (2020).
Protzko, J., Zedelius, C. M. & Schooler, J. W. Rushing to appear virtuous: time pressure increases socially desirable responding. Psychol. Sci. 30, 1584–1591 (2019). (PMID: 10.1177/095679761986793931603024)
Evers, E. R. K., O′Donnell, M. & Inbar, Y. Arbitrary fairness in reward and punishments. J. Exp. Psychol. Gen. 152, 956–967 (2023). (PMID: 10.1037/xge000130036442034)
Laukkonen, R. E. et al. Irrelevant insights make worldviews ring true. Sci. Rep. 12, 2075 (2022). (PMID: 10.1038/s41598-022-05923-3351361318826315)
Inbar, Y. & Evers, E. Worse is bad: divergent inferences from logically equivalent comparisons. J. Exp. Psychol. Gen. 151, 665–675 (2022). (PMID: 10.1037/xge000080434516200)
Entry Date(s): Date Created: 20231109 Date Completed: 20240228 Latest Revision: 20260727
Update Code: 20260727
PubMed Central ID: PMC10896719
DOI: 10.1038/s41562-023-01749-9
PMID: 37945809
Database: MEDLINE
Description
ISSN:2397-3374
DOI:10.1038/s41562-023-01749-9