Efficiency of Propensity Score Methods in Longitudinal and Multilevel Data Via Monte Carlo Simulation

Introduction: Propensity score analysis (PSA) has been widely used in social and behavioral research (Pan & Bai, 2015). The use of PSA spans multiple data structures, including longitudinal and multilevel data. However, the history of using PSA with such data is relatively shorter than that of single level and/or cross-sectional data analysis, which may be due to complexities including within-subject variance in longitudinal data analysis and the violation of independent sampling in multilevel data analysis (Leite, 2017; Guo & Fraser, 2015). Further, extant literature on the relative efficiency of different PSA methods available for structured data analysis is lacking. The purpose of this paper is to conduct an efficiency study, comparing longitudinal PSA methods including the inverse probability of treatment weighting (IPTW: Robins et al., 2000) including marginal structural models (Hernan et al., 2000) and using generalized estimating equations (Liang & Zeger, 1986); regression estimation with cluster-robust standard errors (Heeringa et al., 2010); fixed-effects models (Allison, 2009), and mixed-effects models (Snijders & Bosker, 2012). In addition, we also explore multilevel PSA methods using fixed-effects model and regression models with cluster-robust standard errors.

Method: In this efficiency study, we disentangle the complexities of utilizing PSA with structured data using a simulation study. Utilizing observed data from 875 elementary and elementary/middle combined schools in Maryland implementing Positive Behavioral Interventions and Supports (PBIS) from 2006 to 2011, we examined the effect of PBIS on student suspension rates from 2006 to 2012. As a preliminary analysis, we fit marginal structural models. In the analyses, we used time-varying covariates such as enrollment, truancy, % of special education, % of free and reduced meals, % of student mobility, student-teacher ratio, % of Asian, % of Hispanic, % of African American, % of White, and proficiency of reading.

Results: To address any sensitivity of selection of propensity score models, we fit both logistic regression and generalized boosted method and found consistent results. Both PSAs reduced significant amounts of selection biases on variables. In addition, both results showed that suspension rates reduced over years from 2006 to 2011 ( and ) while the PBIS variable was positively associated with suspension rate at each year ( and ).

Conclusions: Preliminary results of this study showed that a marginal structured model fits to this longitudinal study. In our presentation, we will show the results from Monte Carlo simulations so that researchers understand better how (dis-)similar PSA models are in structured data analysis.