A Multidimensional Perspective on the “Voltage Drop” between Efficacy Trials and Community-Based Delivery of an Evidence-Based Parenting Program

There is substantial concern about “voltage drop” in the effectiveness of evidence-based programs (EBPs) when transported from tightly controlled efficacy studies to systems that can support wide scale delivery. Relative to community settings, researchers in efficacy trials are able to maintain tight control over hiring, training, and supervision. Efficacy studies also have homogeneous samples that may be more motivated to participate, whereas community agencies often have difficulty recruiting and retaining program participants. This dilution of effects is thought to be due to declines in program fidelity. It is also important to consider elements of participant responsiveness, such as attendance and satisfaction, however, there has been limited research in this area.

We will compare multiple dimensions of implementation in the efficacy and effectiveness trials of the New Beginnings Program (NBP), an evidence-based parenting program for families going through divorce. Because the family court system serves all families going through divorce, it is a clear partner for scale-up of NBP. For the effectiveness trial, 4 county-level family courts used their standard RFP process to identify service providers to deliver NBP or a low-dose control. NBP was delivered by 20 providers delivered to 477 parents in 50 groups. The participants were substantially more diverse (40% racial/ethnic minority, compared to 12% in the efficacy trial) and included fathers for the first time (42% of participants).

In the efficacy trial, independent observers rated the extent to which activities were delivered on a scale of 1 (not at all completed) -3 (completed). Mean degree of session completion was 2.86, for a completion rate of 95%. Using the same methods in the effectiveness trial, we found that mean degree of session completion was 2.80, or 93%. In terms of participant responsiveness, parents in the efficacy trial attended 79% of sessions, whereas in the effectiveness trial, parents attended 50%. Parents also provided satisfaction ratings with respect to how useful they perceived each skill taught by the program to be. The average rating was 4.4 in the efficacy trial and 4.5 in the effectiveness trial. We will also present differences between the trials with respect to demographics and baseline parenting and child mental health scores.

Results demonstrated that the main difference in implementation between the trials was in terms of attendance, rather than fidelity to the curriculum or the appropriateness of the skills for the intended audience. We will discuss the implication of various levels of measurement for assessing fidelity. We will also discuss the potential impact of differences in baseline characteristics on attendance, and implications for program outcomes.