Developing Computer-Based Methods for Assessing Quality of Implementation in Parent-Training Behavioral Interventions

Introduction: Traditional methods for monitoring implementation of evidence-based programs require labor-intensive quality assessments. These assessments generally involve human observation, hence becoming a major bottleneck in the monitoring of implementation. In this talk, we present the development of computer-based methods for implementation quality measurement. Firstly, we describe seven principles that help us identify a critical dimension of implementation for automatization, the quality with which the program is delivered. Secondly, we present linguistic constructs that map to these dimensions which can be recognized by machine. Our goal is to develop tools that reduce the burden of quality monitoring and in order to provide timely feedback for local agencies.

Methods: Based on human-based ratings of 470 sessions of the New Beginning Program (NBP) for divorcing parents, we identified a subset of quality dimensions that a) had the highest inter-rater reliability and b) predicted the most variance of certain outcomes of relevance such as parent’s retention and positive engagement in home practice. The measure has three subscales: Positive Engagement, Skillful Presentation, and Skillful Feedback. We use computational linguistics to extract linguistic cues from session transcripts as evidence of high and low quality dimensions. These linguistic cues are used for monitoring implementation constructs.

Results: We began analyses with items from the Positive Engagement subscale. Overall positive engagement is explained by four dimensions (r > .75): Validation and normalization, Reinforced sharing, Warmth and support, and Body language. We present linguistic cues that are associated with these dimensions. Quantitative assessments of reliability against human coding will be provided for each item. We present to what extent the machine scoring replicates the psychometric properties and predictive validity of the scale of retention and positive engagement.

Conclusions: The development of computer-based ratings of delivery quality will facilitate supervision of implementation in community agencies, ensuring high quality delivery and positive outcomes for participants. This presentation will provide evidence for the feasibility of computer-based coding and will present principles that can be used to select items for ratings in other preventive interventions.