Violations of Invariance in a Mediation Model: Impact on Relative Bias, Type I Error Rates, Statistical Power and Coverage

Introduction: In the evaluation of interventions, researchers compare the outcome variable between a group of individuals that received the treatment condition and a control group. Frequently, it is of interest to test if a third variable mediates the effects between the intervention and the outcome variable. In a conventional mediation model this is analyzed by examining the relationship between a categorical independent variable X that indicates membership to the treatment or control group, a mediator variable M and an independent variable Y. For the purposes of the present study, M and Y represent latent variables measured by several indicators. It is assumed that the instrument used to measure M is invariant across the groups in X. That is, the relationships between the indicators and the latent variable M are independent of group membership. Although some studies have examined this problem (Guenole & Brown, 2014; Williams, Jones, Pemberton et al. 2010), the consequences of violations of invariance in mediation models are largely unknown. The purpose of the present study was to systematically examine the impact of violations of invariance on the Type I error rates, statistical power, coverage and bias in the parameter estimates of the mediation model. 

Method: A simulation study in which measurement invariance in the latent mediating variable was simulated by either violating the invariance in item loadings or intercepts was conducted. Results were compared to conditions under measurement invariance. The variables manipulated were the proportion of noninvariant items, the magnitude of the violations, sample size, and the effect size of mediation path coefficients. 

Results: The results indicate that in general, the mediation model was robust to violations of invariance in loadings. While the Type I error rates were not affected by the presence of noninvariant loadings, the statistical power, the coverage and the relative bias of the parameter estimates were affected in the conditions with large violations of invariance and 2/3 ratio of noninvariant items. In contrast, under noninvariant intercepts the path coefficients were severely biased, and the coverage rates were below 0.95 in most conditions with medium and large violations of invariance and sample sizes of 200 and 500. 

Conclusion: The study shows that mediation results are prone to bias, especially when there is measurement non-invariance in intercepts. An explanation for the different patterns of results will be offered and the implications will be discussed.