How Do Teachers Reports of Adherence, Quality and Engagement Relate to Implementation Strength?

Background and Purpose: School-based social emotional learning (SEL) interventions are intended to nurture social and emotional capacities in children, disrupt trajectories toward problem occurrence, and strengthen prospects for school and life success. Although SEL interventions have been tested and demonstrated to be effective for supporting positive youth development, there is an underdeveloped science to inform their widespread delivery (Wandersman et al., 2008).

TOOLBOXis a universal SEL curriculum for grades K-6. The program guides school and family support for their children’s social and emotional development through the explicit cultivation of “tools” that students can use to manage their emotions and relationships. This study seeks to understand which implementation characteristics (i.e., adherence, dosage, and/or participant responsiveness) related to overall implementation strength.

Methods: Data are from a district-wide, online survey administered in 11 elementary schools where TOOLBOX was implemented during the 2014-2015 year.  The survey yielded a 73% response rate. Multivariate regression was used to estimate the effects of specific aspects of implementation (e.g., adherence, dosage, participant responsiveness) on overall implementation strength (i.e., How fully did you implement Toolbox this year?). Adherence was measured by asking teachers how closely they followed the approach outlined in the curriculum guide while teaching a sample lesson. Dosage was measured by the number of lessons taught from the curriculum guide during the academic year. Participant responsiveness was assessed through teacher ratings of student engagement on a sample lesson.

Results:    Teachers assessed their implementation from 0 (low) to 10 (high) (M=6.02, SD=2.06). Adherence was moderate (1-4 scale; M=2.5, SD=6.8), meaning that teachers typically made more than minor changes to the lessons.  Dosage was also moderate for both grade levels; grades K-3 (M=8.79 of 14 lessons, SD=3.6) and grades 4-6 (M=8.38 of 16 lessons, SD=3.3). Participant responsiveness was high (1-5 scale; M=4.03, SD=.81).  Results indicated that teacher adherence and teacher dosage, but not student responsiveness, significantly predicted teacher assessment their implementation strength. Dosage explained a much larger amount (34% for K-3, 28% for 4-6) of the variance in implementation strength than adherence (4%).

Conclusions and Implications: Findings suggest that adherence and dosage contribute to teacher assessment of the strength of their program delivery.  Teachers seem to appreciate dosage more than adherence as essential to implementation success. Implications for implementation monitoring and strengthening will be discussed.