Reducing Bias and Increasing Precision By Adding Either a Pretest Measure of the Study Outcome or a Nonequivalent Comparison Group to the Basic Regression Discontinuity Design

Introduction: Regression Discontinuity (RD) design has become a standard method to produce causal estimates. However, it suffers from three weaknesses: dependency on the functional form estimation, limited ability to generalize away from the cutoff and low power (i.e., precision). To mitigate these three weaknesses, we supplement a basic RD with a comparison function and create Comparative Regression Discontinuity (CRD) design, following Wing and Cook (2013). The comparison function can be either a pretest measure of the study outcome, called CRD-Pre, or a nonequivalent comparison group, called CRD-CG. 

Methods: To assess the performance of CRD-Pre and CRD-CG, we conducted six within study comparisons based on three outcomes and two assignment variables using data from the Head Start impact study. Our research answers the following three questions: first, to what extent do CRD-Pre and CRD-CG reduce bias and increase precision compared to RD? Second, which is better for reducing bias and increasing precision - CRD-Pre or CRD-CG? Third, to what extent do CRD-Pre and CRD-CG produce unbiased and precise estimates compared to RCT? 

Results: We conclude that (I) both RD and CRD designs produce unbiased estimates at the cutoff compared to RCT benchmarks, but find CRD designs have greater power at the cutoff. When certain conditions are met, CRD designs produce unbiased estimates above the cutoff and are at least as powerful as RCT above the cutoff. In contrast, RD cannot generalize the treatment effect away from the cutoff. (II) Both CRD-Pre and CRD-CG are unbiased. However, the power advantage for each design depends on the parameter values - the pretest-posttest correlation in CRD-Pre and the proportion of comparison cases in CRD-CG. Researchers may construct either CRD-Pre or CRD-CG depending on what data is accessible. (III) Although CRD designs can generalize away from the cutoff, they still estimate local effects. To estimate average treatment effect, researchers should use RCT. However, when the RCT sample size is too small to produce reliable estimates, or when the treatment effect is heterogeneous across populations, we suggest researchers consider constructing CRD designs and estimating local effects. 

Conclusions: Both CRD-CG and CRD-Pre designs mitigate the three weaknesses of a basic RD. Thus CRD designs are strongly recommended to replace a basic RD design whenever possible.