Examining Fidelity of Implementation in the Context of an Executive Function Intervention

In testing early childhood interventions, measuring fidelity of implementation is critical to drawing conclusions regarding program efficacy. Further, once an intervention has demonstrated empirical efficacy, measuring fidelity is critical when scaling the intervention to ensure the program is being delivered in a way that works. The current paper describes the development and utilization of several fidelity measures implemented in concert with a pilot study of the Ready? Set. Go! (RSG) intervention, designed to improve preschoolers’ executive function (EF). RSG targets preschoolers’ EF through a brief but intensive program including classroom curriculum, parent education, and child-based individual support.

The fidelity tools developed by our team measure both the quality and quantity of the classroom and individual support activities, including self-report of the number and duration of activities as well as objective observers’ ratings of activity quality. Data were collected during 3 separate iterations of RSG implemented between June 2013 – February 2014 in an early childhood education program located in an emergency homeless shelter as well as a community preschool.

Teachers reported a slight increase in the average number of core EF activities (designed by our team) implemented daily over the course of the intervention, from 1.6 activities per day to 1.8. The average duration of those activities decreased from 16.1 to 13.5 minutes. In addition to core EF activities, teachers also reported an increase in the number of times they adapted regular classroom activities to have an EF focus, with only 41.7% reporting more than 10 instances of EF-adaption in the first week, and 65.7% in the final week.

Objective observers rated the quality of teachers’ administration core EF activities, indicating whether the teacher gave adequate instruction and used the correct props for each activity. The quality of administration measured in this way was perfect: adequate activity instructions and props were used 100% of the time.

The quality of EF-related instruction on both EF and non-EF activities was also measured. Observers reported that teachers in intervention classrooms had higher quality instruction than those in control classrooms on 5 of the 9 dimensions considered.

While the individual support component was to be administered 4 times per week, the actual administration of this component happened less frequently, with the required 4 sessions per week occurring only 36.4% of the time. However, when the sessions did occur, the majority, 79.2%, lasted for at least 10 minutes. The challenges to administering the required number of sessions appeared to be largely a logistic issue, as only 11.8% of sessions were not completed due to child behavior.