Investigation of Propensity Score Matching Paradox in Longitudinal Data

Introduction: Propensity score matching (PSM) has been used to approximate a completely randomized controlled trial design from both experimental and non-experimental designs to draw causal inferences. In many applications, PSM has been thought to result in an observational sample that can approximate such an experimental sample through the reduction in imbalance and biases in parameter estimates. However, King and Nielson (2016) questioned the efficiency of PSM, stating that the expected reductions in imbalance, model dependence, and bias do not occur. Rather, PSM often increases imbalance, model dependence, and bias; a phenomenon called the propensity score matching (PSM) paradox. When comparing other matching methods such as Mahalanobis Distance Matching (MDM) and Coarsened Exact Matching (CEM), the PSM results increase model dependence and bias as the raw imbalance grows while MDM and CEM reduce both model dependence and bias regardless to the raw imbalance.

Method: This paper explores the PSM paradox in longitudinal data analysis using data from a state-wide scale-up of a tiered intervention framework in schools, called Positive Behavior Intervention and Supports (PBIS; Horner & Sugai, 2002). One limitation in King and Nielson (2016) study was the use of one-to-one matching in their simulation, although they showed about 80% PSM literature utilizing one-to-one matching. In our longitudinal study, we applied two propensity score methods, 1-to-n matching and propensity score weighting on the longitudinal PBIS data, to see if the paradox does not emerge with these other approaches. A simulation study will be conducted along with the comparison.

Results: In the current dataset, there are 54 schools implementing PBIS and 604 control schools in one (i.e., the 2005-6) school year. Among 16 variables (15 covariates and 1 propensity score distance), 11 variables indicate positive improvement with 1-to-4 matching while the average number of variables improved via PSM is six variables. That is, a final matched sample of 270 (54+216) reduced selection bias and is recommended for post-PSM analysis. Although the degree to which distance in the select baseline measures was reduced differed in the two PSA methods, 1-to-n matching and propensity score weighting both reduced a significant amount of selection biases. Results from MDM and CEM analyses will also be presented.

Conclusions: We will discuss the improvement of imbalance and bias of the two PSA methods compared with MDM and CEM and provide a recommendation for which PSA methods to eliminate the PSM paradox. Our study related to PSM paradox contributes to the PSM literature in longitudinal and multilevel data.