Propensity Score Approach to Evaluating the Effectiveness of the New Beginnings Program

Introduction: Community-based collaboration and ethics often necessitate an active rather than inactive control condition such that all participants receive some form of treatment. Because an active control may have a positive effect, researchers may not find treatment effects relative to an active control condition (Merry et al., 2011), although the program or the control condition might be superior to what would have occurred if the participants had received no intervention. Another issue that frequently arises is that many participants drop out following randomization but prior to the intervention. Because intent-to-treat analyses classify participants with no exposure to the intervention with active participants assigned to the same condition, treatment effect estimates may be biased. We discuss an alternative to the “analyze them as you randomize them” (Boruch, 1997, p. 17) approach that simultaneously addresses these issues. We form a synthetic inactive control condition comprised of participants who were randomized to conditions but dropped out prior to the intervention and use a propensity score approach to evaluate the effectiveness of the New Beginnings (NB) Program and active control condition as compared to no intervention.

Method: We classify families into three groups: parents randomized to the ten-session NB condition who attended at least one session (N = 337, 40.6% of sample); parents randomized to the two-session active control condition who attended at least one session (N = 320, 38.6%); and parents who never attended a session regardless of randomization (N = 173, 20.8%). We achieve balance based on a large set of baseline covariates and use the resulting propensity score weights to estimate the average treatment effects of the ten-session NB condition and two-session active control condition relative to the synthetic inactive control condition.

Results: At posttest, the ten-session NB condition promotes effective parenting (e.g., family routines, discipline strategies) and reduces parents’ demoralization relative to the synthetic inactive control condition. We also investigate whether the ten-session NB condition and two-session active control condition improve child outcomes at ten-month follow-up.

Conclusions: The propensity score approach (1) provides a synthetic inactive control condition that could not have existed otherwise due the necessity of providing a credible intervention to all families enrolled through the courts and child service agencies and (2) addresses treatment noncompliance. We describe how a propensity score approach can supplement the intent-to-treat analyses used in prevention research and discuss the implications for causal inference.