Methods: Based on the RE-AIM framework, implementation outcomes are assessed with measures of program adoption (e.g., teacher participation, belief in program efficacy, ease/clarity of program) and delivery (quality, fidelity) gathered through surveys and classroom observations of teachers. Qualitative data were also collected. Analyses consisted of descriptive statistics, chi-square tests, and t-tests. We examined differences in implementation outcomes between EBI 1 and EBI 2. We then conducted analyses for each school separately to determine how implementation outcomes between EBIs differed across schools. Content analysis was used for qualitative data.
Results: Adoption and program delivery measures improved after the switch to EBI 2 compared to EBI 1. Data from School 1 primarily drove these improvements. School 2 had relatively good adoption and delivery outcomes for both EBI 1 and 2; while School 1 had relatively poor adoption and delivery outcomes associated with Program 1, and substantially better outcomes associated with Program 2. Qualitative data from teachers confirmed findings from the quantitative analysis.
Conclusions: Results underscore the importance of measuring and evaluating implementation fidelity of youth-focused EBIs in real-world settings. This process evaluation allowed us to document how well EBIs translate to real-world settings of two metro middle schools, where our initial EBI choice was not an ideal match for teachers and students. The new EBI led to significantly better adoption and delivery outcomes. Implementation assessment was key to understanding how processes were operating, which led to adjustments and better implementation outcomes for the trial.