Quasi-Experimental Designs for Community-Level Public Health Violence Reduction Interventions: A Case Study in the Challenges of Selecting the Counterfactual

Introduction: Public health and criminal justice scholars have been vocal about the difficulties inherent in rigorously evaluating large community-level prevention and intervention programs and strategies. Randomized controlled trials (RCTs) often are not possible in community-level evaluation for political, practical, operational, and technical reasons. Rigorous quasi-experimental designs (QXDs) can replicate some of the strengths of an RCT by minimizing unobserved heterogeneity through statistical modeling that balances the treatment and comparison groups. However, for QXDs to have rigor, the selection of the comparison group must provide a valid estimate of the counterfactual outcome for the treatment.

Objectives: We highlight the importance of documenting the step-by-step processes used for the selection of comparison areas when evaluating a community-level violence intervention that targets a large-scale community.

Methods: We demonstrate the proposed method using a propensity score matching framework for an impact analysis of the Cure Violence Public Health model in Philadelphia. To select comparison communities, propensity score models are run using different levels of aggregation to define the intervention site. We discuss the trade-offs made.

Results: We find wide variation in documentation and explanation in the extant literature of the methods used to select comparison communities. The size of the unit of analysis at which a community is measured complicates the decision processes, and in turn, can affect the validity of the counterfactual.

Conclusions: It is important to carefully consider the unit of analysis for measurement of comparison communities. Assessing the geographic clustering of matched communities to mirror that of the treated community holds conceptual appeal and represents a strategy to consider when evaluating community-level interventions taking place at a large scale. Regardless of the final decisions made in the selection of the counterfactual, the field could benefit from more systematic diagnostic tools that document and guide the steps and decisions along the way, and ask: “could there have been another way of doing each step, and what difference would this have made?” Overall, across community-level evaluations that utilize QXDs, documentation of the counterfactual selection process will provide a more fine-grained understanding of causal inference.