Testing the Direction of Mediation Effects in Cluster-Randomized Intervention Studies: Do Prosocial Skills Mediate the Relation between Effective Classroom Management and Academic Competence?

In the standards for evidence in research on preventive interventions, the Society of Prevention Research emphasizes the importance of evaluating and testing the causal mechanism through which an intervention is expected to have an effect on an outcome. Mediation analysis is commonly applied to study such causal processes. However, these tools are limited in their potential to fully understand the role of theorized mediators. For example, in a design where the treatment x is randomized and the mediator (m) and the outcome (y) are measured cross-sectionally, the direction of the hypothesized mediator-outcome relation is not uniquely identified. That is, both mediation models, x → m → y and x → y → m, may be plausible candidates to describe the underlying intervention theory. As a third explanation, unobserved confounders can still be responsible for the mediator-outcome association. The present study introduces Direction Dependence Analysis (DDA; Wiedermann & von Eye, 2015) which can be used to test the plausibility of the competing explanatory models. We show that DDA can be used to uniquely identify the direction of a mediator-outcome relation. DDA is applied using data from the Incredible Years Teacher Classroom Management Program (IYTCM; Reinke, Herman & Dong, 2017). Participants included 105 K to 3^rd grade teachers (22% African American, 75% White) and their students (N = 1817; 76% African American, 22% White) of 9 elementary schools from one large urban school district. The intervention focuses on increasing teachers’ use of effective classroom management and improving students’ behavior, and academic achievement. Multilevel mediation models were used to test the hypothesis that students’ behavior (disruptive behaviors, concentration problems, prosocial behavior, and emotional regulation) mediates the effect of the IYTCM on students’ academic competence (AC). A significant cross-level mediation effect was observed for prosocial behavior (PB; est.=0.06, 95% CI=[0.01;0.11]; cluster level mediation: est.=0.03; 95% CI=[-0.01;0.07]). No direct effect was observed (b=0.02, p=.683) suggesting that PB completely mediates the effect of the IYTCM on AC. No significant mediation effects were observed for the other student behavior variables. DDA was used to test whether PB affected AC (i.e., IYTCM → PB → AC) or vice versa (IYTCM → AC → PB), or whether a third unconsidered factor confounds the PB-AC relation. Significance tests compatible with DDA suggested that IYTCM → PB → AC best approximates the underlying intervention mechanism. Data requirements of DDA for best-practice applications are discussed and software implementations in R and SPSS are provided.