Associations Among Effect Size Indices in Multilevel Models

Intuitive metrics for conveying the real world impact of a statistical effect are integral for effectively communicating and disseminating prevention science to scientists, providers, and community stakeholders alike. To extend the reach of our findings and to allow readers to make qualitative decisions about program effectiveness, an increasing number of scientific journals are requiring effect size measures to be included in manuscripts submitted for publication. Additionally, publishing these statistics enables researchers to conduct more accurate power analyses when preparing grants for submission and to synthesize results across the literature through meta-analysis. Although the push for more thorough statistical reporting standards will undoubtedly increase transparency and research consumption, confusion may arise when computing effect sizes using certain modern statistical methods—in particular, Multilevel Models (MLMs).

MLMs are well-suited for analyzing data in prevention studies where observations can be correlated as a result of an intervention delivered in a group setting or from repeated measurements of the same people. These models account for depended-ness among observations by estimating separate error terms within and between units and, thus, correcting for downwardly biased standard errors and inflated alpha values that occur when independence assumptions are violated. Because MLMs partition variability into multiple terms, the composition of the reference variance for effect size computations is unclear.  

To alleviate complications in effect size calculation, a number of R² and d-type effect size measures have been proposed for MLMs; however, there is little consensus as to which offers the most intuitive interpretation, is most efficient, and is most robust. To date there is only one comparative study (i.e., LaHuis et al. 2014) evaluating the performance of several R² measures. As effect size reporting becomes the expectation rather than the exception, it is necessary that we develop a clear understanding of how the plethora of effect size measures are related, what they mean, and when to use them.

The current study was designed to evaluate the performance of and correlation among several effect size measures, clarify their interpretation, and make recommendations for their use in prevention research. Simulations were conducted varying type of predictor (continuous or dichotomous), level 1 sample size, level 2 sample size, “true” effect size, and intraclass correlation. Correlations between effect size measures were evaluated with special interest given to the relationship between global (whole model) and local (single variable) R² measures. Recommendations for their use in prevention science will be discussed.