Creating an Evidence-Based System-Wide Supervision Model: Impact of Co-Creation

Introduction

R³, a supervisor-targeted implementation approach, was born out of a request by the NYC CWS to train their workforce in the use of evidence-based principles in their every day interactions with families. The system had a desire to implement widely, and turned to researchers to find a solution. Through a collaborative, iterative, and evaluative process, R³ was developed including training, remote coaching, fidelity monitoring, and certification.

R³ draws from 3 areas of reinforcement: effort, relationships and roles, and small steps toward goal achievement. The aim is to expose families to these 3 Rs in each interaction with their caseworkers to bolster positive progress toward completing CWS treatment plans, create a positive culture to sustain evidence-based principles, and to yield a significant public health impact by improving system-level outcomes. A supervisor targeted implementation strategy is used to maximize the potential reach across the system.

Methods

R³ originally was implemented across 5 NYC CWS agencies (n = 281 caseworkers,114 supervisors, 23 agency leaders). To meet another system-initiated request it was replicated across 12 counties in Tennessee (n = 99 supervisors, 364 caseworkers, 39 regional administrators). Currently 6 Oregon counties are in the pre-implementation phase of a replication emphasizing system needs related to addiction. Training and materials were modified for relevance to each system and the state policies under which they operate. Modifications were made in collaboration with systems and are being rigorously evaluated in a hybrid 2 trial evaluating both the implementation and intervention effectiveness of R³.

R³ supervision sessions are video recorded and uploaded to a secure fidelity monitoring system for review by an R³ expert coach. Coaches provide monthly observational monitoring and fidelity rating, written feedback, and virtual consultation. Supervisors are coached to use R³ in their interactions with caseworkers and to support the caseworkers to use R³ with their families.

Results

Outcomes suggest a successful implementation in varying systems. Although the original model was created specifically for the NYC workforce, a collaborative approach to modification led to successful replication. Some supervisors achieved consistent delivery of R³ with fidelity within a year of initial training (44%), the majority by 18 months (80%), and nearly all by 24 months (93%).

Conclusions

Lessons learned regarding system collaboration and large-scale rollout will be shared, including workforce considerations (e.g., inability to take entire workforce offline for training) and the need for flexibility to address system contexts. The impact of system leader turnover will be highlighted.