Reliability and Validity of a Fidelity of Implementation Rating System for the Family Check-up: An Experimental Study

Fidelity of implementation has critical implications for the validity of findings from randomized trials and for the translation of evidence-based interventions into community service delivery settings. A breakdown in treatment fidelity inhibits the ability to draw valid inferences regarding the nature of a putative treatment effect. In this study, we conducted a randomized experiment to understand the conditions under which rating of fidelity to the Family Check-Up (FCU) could be improved. We also sought to validate the client engagement item in our fidelity rating system, which has been found to be an intervening variable between fidelity to the FCU and clinical change (Smith et al., in press). A subsample of 46 families from the intervention group who participated in the FCU when the child was 5 years old was selected from a multisite randomized prevention trial of 731 families. Raters were trained and experienced in the delivery of the FCU. Half of the cases were rated for fidelity with the family’s age 5 assessment results available to the coder; previously, this information was not provided. Reliability statistics of this coding team, compared with those of previous studies, revealed greater consistency in single-item and overall ratings of fidelity. The overall fidelity score obtained from the assessment data condition had statistically superior reliability. Support was found for the validity of the client engagement item in predicting change in oppositional behavior above and beyond similar constructs, such as working alliance, based on incremental validity analyses conducted using Bayesian estimation in SEM. Implications of these findings for prevention and implementation sciences are discussed.