Invited Presentation: Some Quasi-Experimental Designs That Have Robustly Recreated Experimental Estimates and Others That Are Promising but Have Been Less Often Tested

This presentation seeks to identify some classes of quasi-experiment that are worth supporting because they robustly reproduce causal estimates from randomized experiments sharing the same treatment group. The presentation is in 4 parts. (1) A critical analysis is offered of within study comparisons (aka design replications). These seek to compare causal estimates from designs where the treatment group is held constant but the comparison group is formed either at random or systematically. At issue is testing the extent to which a specific combination of sampling, measurement, structural design and data-analytic procedures is successful in creating a quasi-experiment whose results are not different from when a randomly formed control group is used.  (2) A review is offered of within-study comparisons featuring regression discontinuity designs (RD). Special emphasis is placed on RD designs with a comparison discontinuity function. Here the issue is: Does this last function permit unbiased tests, not just at the RD treatment cutoff, but also throughout all the treatment distribution and so facilitates more general causal conclusions than RD normally allows? (3) A review of within-study comparisons featuring interrupted time series designs, especially comparative interrupted time series with a treatment group and  a comparison group that does not receive treatment but is observed over time. And (4) a review of within-study comparisons featuring simpler non-equivalent comparison group design features like (I) explicit analyses and measurement of the selection process into treatment, (ii) matching on a "rich" set of covariates when the researcher holds only weak cards because the selection process is under-explicated or opaque; (iii) sequential matching across multiple levels, (iv) hybrid matching that privileges local intact group matching but uses other procedures when local matches are shown to be inadequate, and (v) matching on pretest measures of the main study outcome. We close with some strong and some tentative conclusions about the role of quasi-experimentation in evidence-based research.