Computer-Based Fidelity Measurements in Behavioral Interventions

A typical behavioral intervention requires recording the audio or video of the session lead by an intervention agent (facilitator). These recordings offer valuable information for training, supervision, and analysis of intervention impact. Yet, they are costly and resource intensive. During efficacy and effectiveness trials, typically more than 90% of the sessions recordings are not coded for fidelity. During implementation of prevention programs, even less sessions are coded for fidelity, and often the required procedures for recording the session and assessing fidelity becomes unsustainable due to cost and time. We propose a way to use computational methods to analyze fidelity automatically. These methods provide promise that enables local organizations to better implement carefully tested evidence-based programs.

Our method analyzes the communication between of the facilitator and at least one individual who is the target of the intervention. We demonstrate how these method can used in behavioral interventions such as New Beginnings and Familias Unidas, both of which are based on parent training to avoid externalizing behavior in either children or adolescents/young adults. Familias Unidas is our first test case. This parent-centered preventive intervention for Hispanic adolescents has been found to be efficacious in reducing drug use and HIV-risk behaviors. It has been tested with adolescents showing varying levels of risk based upon externalizing behavior problems.

In our first test case, we show that an important dimension of joining/engagement can be operationalized by a machine rater with results comparable to those of human raters. While most fidelity assessments are based on a global score for the entire session, our method is similar to micro-coding in that each utterance that belongs to the facilitator is rated on the joining/engagement prescribed linguistic behavior. This utterance-level coding provides information that in aggregate can be exploited to learn the engagement effects across sessions.

Our approach is based on a two level code which marks if an utterance has evidence of high/acceptable joining behaviors or if the utterance is not related to joining. The recognition is based on the search of linguistic patterns developed by a process of knowledge engineering guided by expert raters. These patterns include whether or not a binary question was asked by the facilitator, an open-ended question or a statement. We demonstrate that these utterance-level coding can be reliably coded with a kappa of 0.8 and machine rated with an accuracy of 66%. We will show what are the next steps to operationalize other dimensions of fidelity that can be assessed with verbal as well as non-verbal cues (humor, emotional states, etc.).