Using Data Harmonization, Qualitative Comparative Analysis, and Other Analytic Techniques to Evaluation Samhsa's Partnership for Success

Introduction:

The Program Evaluation for Prevention Contract’s (PEP-C) cross-site evaluation of SAMHSA's Partnership for Success (PFS) combines several analytic innovations to answer the program's main evaluation questions.  These techniques address a number of challenges arising from the evaluation design, including: 1) data on multiple interventions are nested in subrecipient communities which are nested within state, jurisdiction or tribal grantees; 2) nonrandomized comparison communities within grantees; 3) nonrandom selection of intervention types that often occur in combination; 4) cross-grantee variation in measurement of underage drinking and prescription drug misuse outcomes; 5) small sample sizes at the grantee level, and 6) the importance of intervention costs.  This presentation describes the challenges and provides examples of how innovative data analytic solutions address the challenges posed by the evaluation of the PFS program.

The PEP-C team uses relevant baseline census, archival, and survey estimates to select comparison communities for PFS subrecipients.  Propensity score weighting of outcomes analyses then helps to produce less biased estimates of program effects, despite the non-random assignment to program or comparison groups.  The PFS cross-site evaluation uses data harmonization (or integration) to maximize the use of available data (i.e. estimates originating from different measures) without compromising data quality or integrity.  To assess small sample outcomes, meta-regression uses effect sizes instead of raw data (similar to meta-analysis).  Qualitative comparative analysis (QCA) similarly addresses small sample size, but uses logic and truth tables to assess which factors---alone or in combination---identify causal pathways leading to an outcome. QCA has special utility in examining which combinations of interventions and intervention types most often are associated with positive outcomes. Cost analyses for PFS focus on both paid and in-kind resources (labor, space, equipment, materials, etc.) that support intervention implementation and specifically address questions of cost effectiveness as well as cost-benefit. The PFS evaluation represents one of the first attempts to incorporate cost analyses in a real world, large scale assessment of substance use prevention interventions.  Overall this presentation describes and highlights a comprehensive and innovative evaluation approach designed to address challenges associated with conducting similar efforts.   It also will encourage discussion and suggestions from prevention researchers on the potential use of these other innovative techniques within the evaluation of PFS.