Comparison of Video and Audio Recording Modalities for Assessment of Provider Fidelity to a Family-Centered Drug Abuse Prevention Program

Objective: RCTs demonstrate that evidence-based parenting programs (EBPs) have the potential to improve parenting practices and, as a result, enhance child health and wellbeing. Fidelity to family-based interventions predicts client engagement, improvements in parenting practices, and positive child outcomes. Consequently, as the EBPs are implemented in service delivery settings, it is essential to monitor fidelity in order to prevent drift and maintain positive effects. The COACH is a validated observational system used to rate fidelity in the Family Check-Up (FCU), a family-based EBP that has positive effects on parenting and a wide array of child outcomes. Historically, the COACH has been applied to video-recordings of intervention sessions. However, because of privacy and logistical concerns, community agencies have indicated that requirements to video-record sessions are a barrier to implementation. This study seeks to determine whether the COACH is valid and reliable with audio-recordings.

Method: Using data from an existing RCT, 79 families were selected for inclusion based on elevated child problem behavior at baseline and engaging in the FCU. All sessions were rated for fidelity and client engagement using video recordings; these dimensions will also be coded using audio recordings to compare the methods. Reliability is assessed with a randomly selected 20% of recordings via ICCs. We will conduct t-tests to determine differences between the coding modalities for fidelity and engagement. Predictive validity is assessed using a path model (i.e., fidelity to engagement to parenting to problem behavior) with a chi-square difference test for the two-group (i.e., video vs. audio ratings) stacked model.

Results: For the video coding, ICCs are .74 for fidelity and .87 for engagement. Video-based fidelity was associated with client engagement (Β=.41***) and client engagement predicted improvements in parenting (Β=.28**). Indirect effects from fidelity to problem behavior were significant (CI: -.66;-.02). ICCs, t-tests, and validity results will be presented for the audio coding.

Conclusions: We expect that inter-rater reliability based on audio recordings will exceed guidelines (i.e., will be in the “good” or “excellent” range, at or above .60) and that we will find evidence of predictive validity. If these hypotheses are confirmed, it would validate the use of audio-based implementation monitoring. Given that agencies are reluctant to use video recording, due to concerns regarding privacy and the associated expense of equipment and storage, offering these agencies the use of audio recording as an alternative would have the potential to increase adoption of the FCU and broaden the reach of families served.