Academic Journal
Conducting power analysis for meta-analysis with dependent effect sizes: Common guidelines and an introduction to the POMADE R package.
| Τίτλος: | Conducting power analysis for meta-analysis with dependent effect sizes: Common guidelines and an introduction to the POMADE R package. |
|---|---|
| Συγγραφείς: | Vembye MH; Department of Quantitative Methods, The Danish Center for Social Science Research, VIVE, Aarhus, Denmark., Pustejovsky JE; University of Wisconsin-Madison, Madison, Wisconsin, USA., Pigott TD; Georgia State University, Atlanta, Georgia, USA. |
| Πηγή: | Research synthesis methods [Res Synth Methods] 2024 Nov; Vol. 15 (6), pp. 1214-1230. Date of Electronic Publication: 2024 Sep 18. |
| Τύπος έκδοσης: | Journal Article |
| Γλώσσα: | English |
| Στοιχεία περιοδικού: | Publisher: Cambridge University Press Country of Publication: England NLM ID: 101543738 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 1759-2887 (Electronic) Linking ISSN: 17592879 NLM ISO Abbreviation: Res Synth Methods Subsets: MEDLINE |
| Imprint Name(s): | Publication: 2025- : Cambridge : Cambridge University Press Original Publication: Malden, MA : John Wiley & Sons, 2010- |
| Ιατρικοί όροι (MeSH): | Meta-Analysis as Topic* , Models, Statistical* , Research Design* , Software*, Social Sciences/methods ; Humans ; Data Interpretation, Statistical ; Guidelines as Topic ; Reproducibility of Results ; Sample Size |
| Περίληψη: | Sample size and statistical power are important factors to consider when planning a research synthesis. Power analysis methods have been developed for fixed effect or random effects models, but until recently these methods were limited to simple data structures with a single, independent effect per study. Recent work has provided power approximation formulas for meta-analyses involving studies with multiple, dependent effect size estimates, which are common in syntheses of social science research. Prior work focused on developing and validating the approximations but did not address the practice challenges encountered in applying them for purposes of planning a synthesis involving dependent effect sizes. We aim to facilitate the application of these recent developments by providing practical guidance on how to conduct power analysis for planning a meta-analysis of dependent effect sizes and by introducing a new R package, POMADE, designed for this purpose. We present a comprehensive overview of resources for finding information about the study design features and model parameters needed to conduct power analysis, along with detailed worked examples using the POMADE package. For presenting power analysis findings, we emphasize graphical tools that can depict power under a range of plausible assumptions and introduce a novel plot, the traffic light power plot, for conveying the degree of certainty in one's assumptions. (© 2024 The Author(s). Research Synthesis Methods published by John Wiley & Sons Ltd.) |
| References: | Hedges LV, Olkin I. Statistical Methods for Meta‐Analysis. Academic Press; 1985. Raudenbush SW, Becker BJ, Kalaian H. Modeling multivariate effect sizes. Psychol Bull. 1988;103(1):111‐120. doi:10.1037/0033‐2909.103.1.111. Tipton E, Pustejovsky JE, Ahmadi H. Current practices in meta‐regression in psychology, education, and medicine. Res Synth Methods. 2019;10(2):180‐194. doi:10.1002/jrsm.1339. Hedges LV, Tipton E, Johnson MC. Robust variance estimation in meta‐regression with dependent effect size estimates. Res Synth Methods. 2010;1(1):39‐65. doi:10.1002/jrsm.5. Van den Noortgate W, López‐López J, Marín‐Martínez F, Sánchez‐Meca J. Three‐level meta‐analysis of dependent effect sizes. Behav Res Methods. 2013;45(2):576‐594. doi:10.3758/s13428‐012‐0261‐6. Van den Noortgate W, López‐López JA, Marín‐Martínez F, Sánchez‐Meca J. Meta‐analysis of multiple outcomes: A multilevel approach. Behav Res Methods. 2014;47(4):1274‐1294. doi:10.3758/s13428‐014‐0527‐2. Pustejovsky JE, Tipton E. Meta‐analysis with robust variance estimation: Expanding the range of working models. Prev Sci. 2021;23(1):425‐438. doi:10.1007/s11121‐021‐01246‐3. Hedges LV, Pigott TD. The power of statistical tests in meta‐analysis. Psychol Methods. 2001;6(3):203‐217. doi:10.1037/1082‐989X.6.3.203. Hedges LV, Pigott TD. The power of statistical tests for moderators in meta‐analysis. Psychol Methods. 2004;9(4):426‐445. doi:10.1037/1082‐989X.9.4.426. Valentine JC, Pigott TD, Rothstein HR. How many studies do you need?: A primer on statistical power for meta‐analysis. J Educ Behav Stat. 2010;35(2):215‐247. doi:10.3102/1076998609346961. Jackson D, Turner R. Power analysis for random‐effects meta‐analysis. Res Synth Methods. 2017;8(3):290‐302. doi:10.1002/jrsm.1240. Vembye MH, Pustejovsky JE, Pigott TD. Power approximations for overall average effects in meta‐analysis with dependent effect sizes. J Educ Behav Stat. 2023;48(1):70‐102. doi:10.3102/10769986221127379. Zhang B, Konstantopoulos S. Statistical Power Analysis for Univariate Meta‐Analysis: A Three‐Level Model. J Res Educ Eff. 2024;1‐28. doi:10.1080/19345747.2023.2290544. Vembye MH, Pustejovsky JE. POMADE: Power for Meta‐Analysis of Dependent Effects (R package version 0.2.0). CRAN. 2024. https://doi.org/10.32614/CRAN.package.POMADE. Fernández‐Castilla B, Aloe AM, Declercq L, et al. Estimating outcome‐specific effects in meta‐analyses of multiple outcomes: A simulation study. Behav Res Methods. 2020;53(1):702‐717. doi:10.3758/s13428‐020‐01459‐4. Satterthwaite FE. An approximate distribution of estimates of variance components. Biometrics. 1946;2(6):110‐114. Hedges LV. Distribution theory for Glass's estimator of effect size and related estimators. J Educ Stat. 1981;6(2):107‐128. doi:10.2307/1164588. Tipton E. Small sample adjustments for robust variance estimation with meta‐regression. Psychol Methods. 2015;20(3):375‐393. https://doi.org/10.1037/met0000011. Konstantopoulos S. Fixed effects and variance components estimation in three‐level meta‐analysis. Res Synth Methods. 2011;2(1):61‐76. doi:10.1002/jrsm.35. Lakens D, Adolfi FG, Albers CJ, et al. Justify your alpha. Nat Hum Behav. 2018;2(3):168‐171. doi:10.1038/s41562‐018‐0311‐x. Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Routledge; 1988. doi:10.4324/9780203771587. Hattie J. Visible Learning: A Synthesis of over 800 Meta‐Analysis Relating to Achievement. Routledge; 2009. Lipsey MW, Puzio K, Yun C, et al. Translating the statistical representation of the effects of education interventions into more readily interpretable forms. Natl Cent Spec Educ Res. 2012;1‐46. https://files.eric.ed.gov/fulltext/ED537446.pdf. Kraft MA. Interpreting effect sizes of education interventions. Educ Res. 2020;49(4):241‐253. doi:10.3102/0013189X20912798. Schäfer T, Schwarz MA. The meaningfulness of effect sizes in psychological research: Differences between sub‐disciplines and the impact of potential biases. Front Psychol. 2019;10(813):1‐13. doi:10.3389/fpsyg.2019.00813. Ahn S, Ames AJ, Myers ND. A review of meta‐analyses in education: Methodological strengths and weaknesses. Rev Educ Res. 2012;82(4):436‐476. doi:10.3102/0034654312458162. Davey J, Turner RM, Clarke MJ, Higgins JPT. Characteristics of meta‐analyses and their component studies in the Cochrane Database of Systematic Reviews: a cross‐sectional, descriptive analysis. BMC Med Res Methodol. 2011;11(1):160. doi:10.1186/1471‐2288‐11‐160. White T, Noble D, Senior A, Hamilton KW, Viechtbauer W. metadat: Meta‐analysis datasets (R package version 1.0‐0). 2021. https://doi.org/10.32614/CRAN.package.metadat. Polanin JR, Espelage DL, Grotpeter JK, et al. Locating unregistered and unreported data for use in a social science systematic review and meta‐analysis. Syst Rev. 2020;9(1):116. doi:10.1186/s13643‐020‐01376‐9. Borenstein M, Hedges LV, Higgins JPT, Rothstein HR. Introduction to Meta‐Analysis. 1st ed. John Wiley & Sons; 2009. Raudenbush SW, Bryk AS. Hierarchical Linear Models: Applications and Data Analysis Methods. Vol 1. 2nd ed. Sage; 2002. Higgins JPT, Eldridge S, Li T. Including variants on randomized trials. In: Higgins JPT, Thomas J, Chandler J, et al., eds. Cochrane Handbook for Systematic Reviews of Interventions. 2nd ed. Wiley; 2019:569‐593. doi:10.1002/9781119536604. Hedges LV. Effect sizes in cluster‐randomized designs. J Educ Behav Stat. 2007;32(4):341‐370. doi:10.3102/1076998606298043. Hedges LV. Effect sizes in three‐level cluster‐randomized experiments. J Educ Behav Stat. 2011;36(3):346‐380. doi:10.3102/1076998610376617. Taylor JA, Pigott TD, Williams R. Promoting knowledge accumulation about intervention effects: Exploring strategies for standardizing statistical approaches and effect size reporting. Educ Res. 2021;51(1):72‐80. doi:10.3102/0013189X211051319. Hedges LV, Hedberg EC. Intraclass correlation values for planning group‐randomized trials in education. Educ Eval Policy Anal. 2007;29(1):60‐87. doi:10.3102/0162373707299706. Gulliford MC, Ukoumunne OC, Chinn S. Components of variance and intraclass correlations for the design of community‐based surveys and intervention studies: Data from the Health Survey for England 1994. Am J Epidemiol. 1999;149(9):876‐883. doi:10.1093/oxfordjournals.aje.a009904. Verma V, Lee T. An analysis of sampling errors for the demographic and health surveys. Int Stat Rev. 1996;64(3):265‐294. doi:10.2307/1403786. Murray DM, Blitstein JL. Methods to reduce the impact of intraclass correlation in group‐randomized trials. Eval Rev. 2003;27(1):79‐103. doi:10.1177/0193841X02239019. Tanner‐Smith EE, Lipsey MW. Brief alcohol interventions for adolescents and young adults: A systematic review and meta‐analysis. J Subst Abuse Treat. 2015;51(1):1‐18. doi:10.1016/j.jsat.2014.09.001. Dietrichson J, Bøg M, Filges T, Klint Jørgensen AM. Academic interventions for elementary and middle school students with low socioeconomic status: A systematic review and meta‐analysis. Rev Educ Res. 2017;87(2):243‐282. doi:10.3102/0034654316687036. Linden AH, Hönekopp J. Heterogeneity of research results: A new perspective from which to assess and promote progress in psychological science. Perspect Psychol Sci. 2021;16(2):358‐376. Turner RM, Jackson D, Wei Y, Thompson SG, Higgins JPT. Predictive distributions for between‐study heterogeneity and simple methods for their application in Bayesian meta‐analysis. Stat Med. 2015;34(6):984‐998. doi:10.1002/sim.6381. Pigott TD. Advances in Meta‐Analysis. Springer; 2012. Fraley RC, Vazire S. The N‐Pact factor: Evaluating the quality of empirical journals with respect to sample size and statistical power. PLoS One. 2014;9(10):e109019. doi:10.1371/journal.pone.0109019. Vembye MH, Weiss F, Bhat BH. The effects of co‐teaching and related collaborative models of instruction on student achievement: A systematic review and meta‐analysis. Rev Educ Res. 2023;1‐47. doi:10.3102/0034654323118658. Kirkham JJ, Riley RD, Williamson PR. A multivariate meta‐analysis approach for reducing the impact of outcome reporting bias in systematic reviews. Stat Med. 2012;31(20):2179‐2195. doi:10.1002/sim.5356. Wickham H. ggplot2: Elegant graphics for data analysis (R package version 3.5.1). CRAN. 2016. https://doi.org/10.32614/CRAN.package.ggplot2. Filges T, Sonne‐Schmidt CS, Nielsen BCV. Small class sizes for improving student achievement in primary and secondary schools: A systematic review. Campbell Syst Rev. 2018;14(1):1‐107. doi:10.4073/csr.2018.10. Pustejovsky JE, Chen M. Equivalences between ad hoc strategies and meta‐analytic models for dependent effect sizes. J Educ Behav Stat. 2023;1‐31. doi:10.3102/10769986241232524. Valentine JC, Wilson SJ, Rindskopf D, et al. Synthesizing evidence in public policy contexts: The challenge of synthesis when there are only a few studies. Eval Rev. 2017;41(1):3‐26. doi:10.1177/0193841X16674421. McKenzie JE, Brennan SE. Synthesizing and presenting findings using other methods. In: Higgins JPT, Thomas J, Chandler J, et al., eds. Cochrane Handbook for Systematic Reviews of Interventions. 2nd ed. Wiley; 2019:321‐347. Rothman KJ, Greenland S. Planning study size based on precision rather than power. Epidemiology. 2018;29(5):599‐603. doi:10.1097/EDE.0000000000000876. Moreau D, Gamble B. Conducting a meta‐analysis in the age of open science: Tools, tips, and practical recommendations. Psychol Methods. 2020;27:426‐432. doi:10.1037/met0000351. |
| Contributed Indexing: | Keywords: dependent effect sizes; meta‐analysis; power; robust variance estimation; traffic light power plot |
| Entry Date(s): | Date Created: 20240918 Date Completed: 20241107 Latest Revision: 20241123 |
| Update Code: | 20260130 |
| DOI: | 10.1002/jrsm.1752 |
| PMID: | 39293999 |
| Βάση Δεδομένων: | MEDLINE |
καταχωρήστε σχόλιο πρώτοι!