Selecting Covariates for Propensity Scores: A Demonstration of Data Mining with an Adolescent Recovery Example

Background. Studies of complex behavioral health issues often utilize randomized controlled trials (RCTs) to enable causal inferences about treatment effects on outcomes; yet, an RCT may be inappropriate for ethical or practical reasons. Thus, nonrandomized quasi-experimental designs (QEDs) are one study design substitute. In the absence of randomization, propensity scores can be used to balance groups on pretreatment differences, thus enabling causal inferences about the effectiveness of the treatment. There are many methodological considerations in propensity score estimation (PSE) beginning with selecting the proper covariates that influence both selection into treatment and resulting outcomes: selection of the correct covariates and the creation of propensity scores should result in covariates that are balanced between the treatment and control groups (e.g., as would be seen if randomization to separate groups had occurred). Although covariate selection is one of the most important steps in PSE, there is no gold-standard approach for selecting covariates and different methods produce different propensity scores and varying levels of covariate balance. Traditional methods used for PSE, i.e., logistic regression, can be burdensome and often ignore considerations such as non-linearity and interactions between variables. Exploratory methods, such as data mining approaches like classification trees, can provide important details for PSE and may reduce some of the burden in covariate selection; however, data mining approaches for PSE are rarely used in social science (Thoemmes & Kim, 2011).

Methods. Using a case example from an observational study of adolescents in recovery from substance use disorders (N = 260), this study demonstrates the utility of combining a theory-driven approach with exploratory data mining methods, such as classification trees, for PSE. It compared logistic regression to two data mining approaches, classification trees and random forests, for covariate selection and balance among 35 potential covariates.

Results and Implications. Although the logistic regression method produced the best balance on included covariates (34/35), the highest number of observations were dropped (resulting N = 207). The random forest method retained the entire sample (N = 260) and identified key interactions for proper balance, however, balanced fewer covariates (26/35). Single classification trees dropped more observations (resulting N = 239) and balanced the fewest number of covariates (23/35). Thus, the random forest method may be most useful for initially identifying potential covariate interactions to include in PSE models to properly balance groups prior to estimating the treatment effect, while the classification trees may be less precise and more unreliable. Challenges encountered during the PSE process are discussed and practical recommendations for those wishing to use these PSE analytic methods for estimating treatment effects from quasi-experimental designs are presented.