Method: The first aim of this paper is to understand how Level-1 sample size variation is related to power for two-level designs. Formula based approaches (e.g., Optimal Design) only consider the average number of i units per j cluster, while this simulation explores the effects of variation in Level-1 sample size on statistical power. The second aim of this paper explores the proportion of Level-1 units sampled in two-level designs. Again, considering only the average number of i units per j cluster fails to capture the amount of sampling coverage/noncoverage of units within a cluster. Monte Carlo simulation studies were generated and analyzed using Mplus. Both 95% coverage (the proportion of replications for which the 95% confidence interval contains the true parameter value) and statistical power (the probability to correctly reject the null hypothesis when it is indeed false) were used to assess each model.
Results: With regard to both 95% coverage and power, simulation results suggested that variability in Level-1 sample size has limited impact on the power estimates in two-level models. However, models with varying degrees of sampling coverage of Level-1 units produced mixed results. As the proportion of Level-1 units sampled increased, 95% coverage values increased as well. However, statistical power remained relatively constant between models, regardless of sampling coverage.
Conclusions: Prior literature has neglected the issues of variability in Level-1 sample sizes, as well as sampling coverage/proportion of Level-1 units sampled, and their effects on statistical power. The results from this paper help to inform sampling considerations for future researchers using CRTs.