Can Observation be a Pragmatic Measure of Fidelity?

Introduction: Observation of client -therapist interaction is considered the “gold standard” for measuring fidelity however, it is often argued that observation is not a practical or pragmatic method that can be used in “real world” non-research settings such as in scale up of evidence based practices in child and family service systems. The FIDO (FIDelity Observation) system was designed to code the content and process occurring in parent groups implementing the KEEP model. FIDO data are coded by expert coaches and regular on-site clinical supervisors during weekly routine reviews of intervention sessions. A HIPAA secure platform was developed to upload video recordings of parent group sessions. During initial implementation, expert coaches use the FIDO system to code weekly observed group leader/parent interactions, track attendance and other critical implementation tasks including ratings of the occurrence of child behavioral problems and of parent involvement. During the sustainability phase of KEEP implementation, expert coaches train on- the- ground supervisors to complete the session coding as part of their routine case monitoring.

Methods: Data on observed fidelity scores were collected as part of a real world implementation of the KEEP intervention intended to keep foster and kinship parents trained and supported in the New York City and Tennessee Child Welfare Systems. Session fidelity scores and rates of parent reported child behavioral problems were fed back to parent group leaders and to child welfare system leadership in monthly reports. Fidelity was classified as being in one of three categories for each group session; red (unacceptable), yellow (needs work) or green (acceptable). These scores were used in weekly coaching/supervision sessions between KEEP experts and and group leaders and later between on-site supervisors and KEEP group leaders. The FIDO system was implemented by private agency personnel (in NYC) and by state case workers in Tennessee. 

Results: As hypothesized, the observed fidelity scores of group leaders implementing KEEP increased over time as group leaders became more expert in conducting the model. This demonstrates that the expert coaching by model developers had the intended effect. Additionally, parent reports of child behavior and emotional problems decreased over time as the fidelity of implementation increased.

Conclusion: Observational methods, considered as the gold standard method for measuring fidelity can be adapted to be pragmatic and acceptable for use in real world implementations of evidence-based practices in real world scale ups. Expert coaches can train front line supervisors to implement observational methods to sustain the fidelity of EBPs in private agency and state child welfare settings.