Documenting and Learning from Years of Technical Assistance to Community Coalitions and Evidence-Based Programs

Introduction. The Research-Based Delinquency and Violence Prevention Initiative within Pennsylvania has been implemented for 20 years. In that time, the technical assistance, coaching resources and structure available to program implementers within Pennsylvania has grown meaningfully. As the availability and TA resource pool has grown, so has the documentation of all technical assistance and coaching interactions. The current study analyzes almost 10,000 reported coaching/TA interactions between TA professionals and program implementers. Questions about feasibility and challenges with coding 10,000 archival instances of TA-implementer interactions are explored.

Methods. Technical assistance providers now provide support for 17 evidence-based programs in Pennsylvania. Starting in 2008 and continuing through 2017, every TA provider (n=14) noted each instance of TA with each grantee/program that they had. This tracking mechanism was solely for performance measures reporting to the funder of program grantees. TA providers entered their staff ID, date, type of TA and additional notes around TA provision. In the end, 10,954 instances of TA were provided and reported on over the 10 study years. A team of three prevention science researchers with expertise in implementation science worked together to develop initial TA strategy codes. Then, two research assistants were hired and trained to code the data; high levels of reliability were achieved.

Results. Overall, 10,309 TA instances across nine programs remained in the dataset after excluding programs with less than 300 reported TA instances. A primary TA strategy (agreement = 89%) and whether or not it was a multiple TA strategy (agreement = 87%) was coded; in the cases of multiple strategies, a secondary strategy was also coded (agreement = 78%). An average agreement of 88% (Min = 75, Max = 94) was achieved across all codes. Six different TA strategies were uniquely identified: coaching, consultation, coordination-logistics, coordination-networking, monitoring, and resource distribution. Some interesting findings are surfacing. Certain TA strategies seem to be more related to some programs than others. In addition, the frequency of each code at different time points vary. Additional analyses are investigating whether there are systematic predictable differences between programs that affect the type of TA delivered and program outcomes.

Discussion. A “big data” qualitative data source, collecting information about technical assistance interactions for the purposes of program accountability funding, has successfully been used to understand more about the TA process. Along the way there were multiple challenges. This presentation will highlight some early results and then focus on these challenges.