Going Beyond the Checklist to Measure Intervention Fidelity: Adherence and Competence

Introduction: The feasibility of implementation and sustainability of evidence-based programs (EBPs) delivered in schools and classrooms has been a concern of prevention researchers and practitioners. While there is substantial evidence that school-based EBPs can have positive effects on students’ academic and behavioral functioning, the ability of teachers and other school personnel responsible for implementing these programs to sustain high-quality implementation remains an open and vexing question, due in part to limitations in the measurement of fidelity of implementation of EBPs. As teachers are often the intervention agents in school-based EBPs, assessing both their adherence to a protocol as well as the competence with which they deliver the intervention is critical in determining both efficacy as well as issues related to sustainability.

Methods: Trained observers measured treatment fidelity to the classroom meeting component of the Olweus Bullying Prevention Program. Both adherence and competence were rated on two dimensions (Teacher Instructional Behavior, Protocol Fidelity) using a 3-point Likert-type scale. Observers also rated student engagement during the class meeting using two items also rated on a 3-point Likert-type scale. Observations of teachers (n = 22) occurred at several time points over the school year for a total of 111 independent observations.

Results: Greater adherent and competent teacher instructional behavior were related to higher student engagement (r = .60 and .59, respectively). Further results indicated that student engagement was higher when teacher instructional behavior was rated as highly adherent and competent, compared to low adherence and competence on teacher instructional behavior (d = 1.11). Similarly, higher adherence and competence in protocol fidelity were related to student engagement (r = .33 and .49, respectively). More specifically, student engagement was higher for teacher behavior rated as more competent in protocol fidelity, regardless of their level of adherence to protocol fidelity.

Conclusions:  Two dimensions of treatment fidelity, adherence and competence, were measured and for this sample of observations appear to have some differential effects on student engagement. These findings have implications for future research and potentially the training of teachers who implement EBPs in schools.