Identification of Feasible Indicators of Program Implementation As Programs Move from Effectiveness Trials to Sustained Community Implementation

Introduction: Prevention programs have developed a sufficient body of evidence to justify wide-scale implementation in community settings. There is a great deal of concern, however, as to the ability of service agencies to deliver these programs with enough rigor to match the effects found in efficacy trials. While monitoring the delivery of these programs is critical, doing so can be resource intensive, and there is no guidance as to what factors are most important to monitor. This study uses data from a large-scale effectiveness trial to identify indicators that could inexpensively be used to monitor quality of program implementation as a community-based service.

Methods: To assess the contribution of theoretically based implementation domains (Berkel et al., 2011), we use data from the New Beginning Program (NBP) for divorcing parents, including fidelity (amount of curriculum delivered), quality (rapport building, presentation quality, feedback on skills), and participant responsiveness (attendance, quality of home practice). Multilevel analyses were conducted in Mplus. Missing data were addressed using FIML. All implementation variables and pretest scores were simultaneously entered into the regression model. Warm parenting was modeled as a latent construct.

Results: Fit indices indicated good fit for all models. In predicting improvements in warm parenting, parent reported efficacy in doing home practice efficacy was the best predictor (β = .25, p ≤ .001). In predicting improvements in child behavioral health symptoms, there was a trend of an association between home practice efficacy and improvements in internalizing (β = -.10, p ≤ .1). The best predictor of internalizing, however, was the program facilitators’ use of feedback for participants’ skill practice. In contrast with hypotheses, better feedback was associated with increases in internalizing (β = .13, p ≤ .05). It was also associated with decreases in parenting (β = -.13, p ≤ .05). For improvements in child externalizing symptoms, facilitators’ use of rapport building was the best predictor (β = -.16, p ≤ .05).

Conclusions: These results support the importance of collecting data on participants’ efficacy in doing home practice skills, which is a relatively feasible indicator to collect. Two indicators of process quality were important predictors of program outcomes, rapport building and feedback. Feedback functioned in the opposite direction as hypothesized, which may be due to a greater need for feedback in groups where participants struggled more. Measurement of process quality involves observation and substantial amounts of training that might not be available in community settings. We will discuss implications for decision making, which should take into account the evidence, funding requirements, and resources available.