A Test of Quality: An Examination of the Hi-Q Measure of Quality of Implementation in the New Beginnings Program

Success of evidence-based programs depends on multiple dimensions of implementation. Measurement of these dimensions has typically been limited to dosage counts or fidelity checklists. One dimension of primary importance, but that has rarely been assessed is quality, or the teaching and clinical skill with which a facilitator delivers a program. Our theoretical model proposes that quality increases participant responsiveness, or participants’ retention in the program and engagement with home practice, which in turn will lead to improvements in program outcomes. We hypothesize that the nature of quality may depend on the type of activity being delivered within a program. For example, many programs contain didactic activities, where the facilitator is presenting new information to the group. In these types of activities, facilitators must be engaging, clear, and show how the material is relevant to program participants. In other activities where group members have the opportunity to practice or get feedback on their use of a skill, facilitators again must be engaging but they also need to provide constructive feedback concerning skill use, including ideas about how to use the skill more effectively.

The New Beginnings Program (NBP) for divorcing families uses these types of activities to help parents develop skills to protect children from the potential negative outcomes that can come with divorce. We are currently piloting a newly developed measure of quality (Hi-Q) across three activity types: didactic presentations (DP), home practice review discussions (HPR), and skills practice role plays (SP). The measure has three subscales which we hypothesize will break down accordingly across activity types: Positive Engagement (DP, HPR, SP), Skillful Presentation (DP only), and Skillful Feedback (HPR and SP). Items for each subscale were based on theory and qualitative review of tapes of program delivery.

For each of the 10 NBP sessions, two trained coders observed one activity, which was chosen based on primary importance to NBP’s theoretical model and representation of the activity types, DP (3), HPR (4), and SP (3). Interrater reliability ≤ .85 for the 186 activities coded across the 20 groups in Cohorts 1-2. Cronbach’s α for the first two cohorts ≤ .80 for across activity types. The 30 groups of Cohorts 3-4 will be coded during the 2013-14 academic year. Confirmatory Factor Analysis will be conducted in Mplus, modeling the three subscales of the Hi-Q measure for each activity type. Then, we will test the relation between quality and participant responsiveness indicators of retention and home practice completion. Finally, multiple group analyses will be used to examine possible differences across parent gender and ethnicity.