Intervention Format, Implementation Leadership, and Dosage As Related to Toolbox Youth Outcomes

Introduction: TOOLBOX (Collin, 2015) is a program that promotes children’s social-emotional development through the instruction and reinforcement of 12 tools (e.g., the Breathing Tool, the Garbage Can Tool). TOOLBOX has been widely implemented, but has only been explored in 2 studies (e.g., pre/post studies without comparison group; DeLong-Cotty, 2011). The current study is a quasi-experimental study of TOOLBOX that examines implementation contexts, behaviors, and outcomes.

Methods: Students’ social-emotional competence was measured using the Devereux Student Strengths Assessment Mini (DESSA-Mini; Naglieri et al., 2009), a teacher-completed behavior rating scale. The sample includes 1,625 K-2 students nested within 79 classes. TOOLBOX Standard (lesson-emphasis) was administered to 554 students, Primer (strategies-emphasis) to 496 students, and 575 students received no TOOLBOX. There were no baseline differences in initial DESSA scores by condition. All analyses controlled for baseline DESSA scores (group-mean centered), and specified random intercepts and fixed slopes in a multi-level framework. Implementation leadership was measured through the teacher-reported Implementation Leadership Scales. TOOLBOX dosage was measured by teacher report at 3 time points. Dosage measurement included 3 lesson-based indicators (e.g., number of lessons taught) and 20 strategy-based indicators (e.g., asking students what tools they could use in the moment).

Results: Controlling for baseline scores, TOOLBOX students had higher DESSA scores at mid-year (p<.05) and year-end (p<.001) relative to comparison students. Within TOOLBOX conditions, there were no differences in student outcomes between Standard and Primer conditions. Controlling for class-level variations in DESSA scores, several implementation leadership and dosage variables were significantly related to student outcomes. When teachers perceived implementation leadership as more supportive (β=2.43) and perseverant (β=2.55), their students were more likely to have positive outcomes at year-end (p<.05). The teacher-reported dosage also related to student outcomes. For example, when teachers taught more lessons (β=.44), reinforced TOOLBOX more outside of lessons (β=1.41), and used TOOLBOX more frequently to improve their own lives (β=2.17) at mid-year, their students had higher DESSA scores at year-end (p<.05).

Implications: Through a dose-response relationship, these results provide evidence of TOOLBOX effectiveness. Directives to emphasize either manualized lessons or strategies does not appear to make a significant difference in youth outcomes, but implementation leadership and dosage indicators explain 8-22% of between-class variations in social-emotional competence across conditions.