Evaluation of a Youth Substance Use Prevention System of Care

Introduction: South Florida Behavioral Health Network (SFBHN), a Florida Department of Children and Families regional managing entity, funded BSRI to conduct an evaluation of the Prevention System of Care across Miami-Dade and Monroe counties to address the need for a results-based Prevention System. Previously, SFBHN relied on traditional methods of assessing intervention effectiveness for youth substance use prevention such as process data or program-specific data with little standardization across the system. The current evaluation addresses the need for organizations to show that services provided are linked to positive outcomes. This is more critical than ever as funders demand evidence that their dollars are making a significant difference. The evaluation is designed to increase the capacity of SFBHN, the State, and providers to monitor, improve, and plan what prevention activities to sustain.

Methods: The evaluation incorporates SAMHSA’s Strategic Prevention Framework (SPF) for assessment, capacity building, planning, implementation, and evaluation across providers and funding sources.  First, the evaluation team identified and addressed gaps in county-level goals by conducting a needs assessment using epidemiological data and community feedback. A collaborative effort led to development of local goals that feed into county-level goals and encompass prevention providers within the system of care. Evidence-based tools were used to gather baseline data on providers’ capacity and motivation to implement evaluation processes to achieve prevention goals and contribute to the system of care. As the project progresses, process and outcome data is consistently used to inform planning and technical assistance. 

Results: Results of the evaluative process yielded both a Strategic Prevention Plan and Evaluation Plan.  BSRI used prevention research to tie EBP assessment items to risk and protective factors (Hawkins & Catalano) and developmental assets (The Search Institute) which assesses individual and group differences in responses and/or outcomes based on those factors. These outcomes are linked to broader regional county and state level outcomes as well. Capacity building plans have also been developed for each provider and are used to monitor their processes and progress towards implementing the evaluation framework.  Outcome data is generated each month and aggregated across providers and EBP strategies.

Conclusions: A cadre of tools was developed to facilitate the implementation of a new comprehensive evaluation strategy including prevention provider report cards, individual capacity building plans, training and technical assistance surveys, and outcome reports to providers and funders.  Garnering buy-in across individual providers, coalitions, and administration has proved critical. Strategies to minimize data burden in the wake of administrative challenges such as lack of billing codes for data entry and required direct service/administrative time breakdowns had to be developed uniquely for each provider. Building capacity using CBOs as peer learning coaches proved effective in increasing communication between providers and strengthening trust and cohesion within the prevention system of care.