Adjusting for Study Redundancy in Second-Order Meta-Analysis

Title:

Adjusting for study redundancy in second-order meta-analysis.

 Abstract:

Introduction: As we pursue an increasingly interdisciplinary understanding of prevention science, new forms of synthesis reviews are being developed. Among these are synthesis projects that combine results across multiple published meta-analyses, sometimes referred to as “second-order meta-analyses” (Hunter & Schmidt, 2004).

An important methodological challenge that is essential to address in second-order meta-analyses is redundancy, or repeated use, of trials across multiple meta-analyses. As the review of a topic area is updated with a new meta-analysis, for example, for prevention of depression, new trials are combined with many previously included trials, while other previously included trials are dropped because of different inclusion-exclusion criteria. Thus, we cannot rely completely on just the most recent meta-analyses. This redundancy results in non-independent data for the second-order meta-analysis and threatens to introduce bias into the estimated overall effect size and the associated standard error.

Methods/ Results: To date, we are unaware of any methodological work that has been done to determine the amount of trial redundancy that is likely to present a bias in second-order mean effect sizes, nor are we aware of any methodological work to determine how best to handle trial redundancy when calculating second-order mean effect sizes or their standard errors.  Working with a sample of meta-analyses and trials focused on the prevention of adolescent depression, we will examine the level of bias and distortion of second order effect size estimates, as well as their standard errors. We propose developing a general purpose methodology that ultimately combines published data from meta-analyses with individual level effect sizes from analyses of pertinent outcomes from individual level trials.  To make this procedure useful, we will account for the level of effort required to search through trials and their outcome analyses, and calibrate this against the degree of bias and precision that can be obtained. Simulation of more and less extreme scenarios will also provide further variability.

Conclusions: If we are able to identify a way to correct for bias introduced by trial redundancy in second-order meta-analyses, trials that may have been excluded in the past due to procedures where some primary meta-analyses were dropped could be included.  This would result in better estimates of overall effect sizes in these second order studies, and ultimately provide a means to identify which programs should be moved to scale in wider public health efforts.