Implementation Across Diverse Contexts: Implications for Measurement

Introduction: Despite the increased availability of evidence-based programs, there are growing concerns about poor implementation quality across multiple settings. These concerns are particularly great in the context of scale-up efforts. As a result, there is a great need for additional research to improve the measurement. This presentation will discuss the importance of context in efforts to scale up preventive interventions under non-research conditions, with particular attention to issues of measurement in the assessment of dissemination and implementation across diverse settings.

Method: Relevant findings from two studies will be used to demonstrate the significance of context for implementation measurement. We will first present findings from a randomized trial examining the implementation of a school-based drug prevention curriculum delivered by teachers and police officers in diverse middle schools. The study used both implementer self-reports of implementation fidelity and quality as well as videotaped observations coded by nationally-certified trainers. The second study to be presented describes a 14-year initiative involving nearly 200 replications of a diverse menu of evidence-based programs across one state. Five waves of annual assessments of dissemination and implementation examined the facilitators and barriers to adoption, implementation fidelity and quality, adaptation, and sustainability of these programs under natural, non-research conditions. 

Results: Findings from the randomized study showed considerable intra-individual variation as well as significant differences by measurement type (i.e., self-report vs. videotaped observation), and highlighted the limitations of conventional “fidelity checklist” measures. This study demonstrated the importance of rich data in assessing the wide variability of implementation across practitioners and settings. In the second study examining a dissemination effort, programs were grouped by the context of implementation, including: school-based, community-based, family-focused, and therapeutic interventions. Analyses revealed that there were significant differences by program type/implementation context for every domain measured, including adoption, implementation, fidelity, adaptation, and sustainability.

Discussion: These studies highlight the importance of context in translational research, and validate the difficulty of developing dissemination and implementation measures that are robust across diverse settings. Recommendations are provided for future research on implementation science with an emphasis on sensitivity to context and burden on program managers. Implications for program implementers are also considered.