Scaling Early Intervention By Increasing Fidelity Monitors: A Training Program for Off-Site Community Supervisors

Monitoring fidelity to evidence-based interventions is a critical component of successful dissemination (Durlak & DuPre, 2008), however, implementing a fidelity monitoring system that is feasible, and which yields valid data, is challenging for many community-based providers (Schoenwald, Garland, Chapman, Frazier, Sheidow, & Southam-Gerow, 2011).

[Intervention name removed for blind review] is an evidence-based early intervention that has received a rating of 1 on the California Evidence-Based Clearinghouse for Child Welfare scientific rating scale, and has been disseminated in community settings across the United States. It is a home-visiting based program that uses in vivo feedback (via in-the-moment comments) to improve parents’ sensitive and synchronous behaviors toward their infants. Recently, larger-scale efforts have been established as interest in the intervention has grown. The current study describes the process of disseminating the intervention with fidelity while also increasing the reach of the program to providers across the country and internationally. Our goal is to ensure fidelity even as the model gets further from direct supervision by the developer: Part of meeting this goal involves training off-site fidelity supervisors at community agencies to improve reach and sustainability.

To enable ongoing fidelity monitoring by off-site community supervisors, we have developed a fidelity coding “bootcamp” to train supervisors at local agencies where clinicians are practicing the intervention. The fidelity system codes for parent and clinician behaviors: first, intervention-targeted parent behaviors (e.g., nurturance, following the lead, delight, frightening behavior) are coded, and second, clinician responses (or lack of responses) to these behaviors are coded. In our curriculum, supervisors code several practice videos, then review their coding with a master coder.

Reliability with the coding system was then measured by the percent agreement between the supervisor and a “master coder” on multiple parent behavior and parent coach commenting codes across 10 videos. To be considered reliable in the coding system, supervisors are expected to agree with the master coder at least 70% on each type of code. The process is quickly growing, with 6 off-site parent coaches currently trained in this system, and 5-10 more to be trained over the next few months. For the 6 off-site parent coaches who have been currently trained, agreement with the master coder ranged from 69.4-89.6% on the parent behavior codes, and 74.9-90.0% on the coach commenting codes. These scores indicate an ability to meet reliability criteria through this training system.

Once off-site supervisors are trained to reliability in the coding system, they receive training in how to perform weekly supervisions with clinicians, including learning how to tailor feedback relative to each clinicians’ progress in commenting and coding and the needs of the family. We have also developed a system that provides ongoing consultation with the intervention developer’s team to the off-site supervisors. Because of the quick growth, this is a dynamic process that we are continuing to refine; lessons learned will be discussed.