Using Computational Linguistics and Knowledge Engineering towards an Automatic Fidelity and Monitoring System for Familiasunidas

Introduction: Fidelity monitoring and feedback are critical components for implementing effective preventive interventions. However, these are also resource intensive for the research team even in efficacy/effectiveness trials, and generally too expensive to maintain when the program is  implemented fully within the host institution or community.  A reasonable goal is to reduce the cost of obtaining reliable and valid fidelity ratings for behavioral intervention programs by an order of magnitude or more.  We present a proof of concept of a computational method that measures fidelity in the Familias Unidas intervention. Familias Unidas is a family-focused prevention intervention targeting externalizing behaviors among Hispanic youth. It is delivered by school counselors in Spanish to parents and in the home in English and Spanish. Our work uses speech analysis, knowledge engineering, and computational linguistics to measure fidelity.

Methods/ Results: The existing method for rating fidelity to Familias Unidas requires expert raters to code videos of recorded sessions. We present a novel semi-automatic method to rate fidelity in these sessions. This method uses a) speech recognition and b) machine learning to rate utterances spoken by the facilitator. The algorithm searches for linguistic patterns associated with high or low fidelity. These patterns were developed by a process of knowledge engineering with experts on fidelity to Familias Unidas. We demonstrate the first step of such algorithm on an initial set of Familias Unidas intervention sessions. We tested our system and obtained (step a) 72% accuracy from automatic transcription as compared to human generated transcripts, and (step b) 66% accuracy from automatic fidelity ratings against the human expert ratings. Examples of the output will be presented. Future work is discussed to develop a more complete system of automated fidelity ratings for Familias Unidas.

Conclusions: This work represents an initial step towards reducing the high cost and potentially improving the quality of fidelity ratings of evidence based interventions.