Unbalanced Cluster Sizes and Variation in Sampling in Cluster Randomized Controlled Trials: What’s the Impact on Power?

Introduction: Cluster randomized trials (CRTs; Murray, 1998) are an increasingly common design used to test the impact of preventive interventions to determine the impact on groups of individuals that are nested within a hierarchical setting, such as teachers or students nested within schools. While multilevel approaches for analyzing these nested data structures have become the standard in the field, issues related to statistical power in CRTs remain an area of interest for many prevention scientists. In these real-world settings various challenges often occur, such as unbalanced cluster sizes; yet previous formula- and simulation-based approaches have failed to account for the effects of sample size variation and sampling coverage on power. In this paper, we examine the effects of unbalanced cluster sizes as well as assumptions behind that variability (e.g., missing data, sampling decisions) on power and parameter estimation using Monte Carlo simulation studies.

Method: The first aim of this paper is to understand how Level-1 sample size variation is related to power for two-level designs. Formula based approaches (e.g., Optimal Design) only consider the average number of i units per j cluster, while this simulation explores the effects of variation in Level-1 sample size on statistical power. The second aim of this paper explores the proportion of Level-1 units sampled in two-level designs. Again, considering only the average number of i units per j cluster fails to capture the amount of sampling coverage/noncoverage of units within a cluster. Monte Carlo simulation studies were generated and analyzed using Mplus. Both 95% coverage (the proportion of replications for which the 95% confidence interval contains the true parameter value) and statistical power (the probability to correctly reject the null hypothesis when it is indeed false) were used to assess each model.

Results: With regard to both 95% coverage and power, simulation results suggested that variability in Level-1 sample size has limited impact on the power estimates in two-level models. However, models with varying degrees of sampling coverage of Level-1 units produced mixed results. As the proportion of Level-1 units sampled increased, 95% coverage values increased as well. However, statistical power remained relatively constant between models, regardless of sampling coverage.

Conclusions: Prior literature has neglected the issues of variability in Level-1 sample sizes, as well as sampling coverage/proportion of Level-1 units sampled, and their effects on statistical power. The results from this paper help to inform sampling considerations for future researchers using CRTs.