Examining Construct Validity of School Climate Survey in Middle Schools: Application of Multilevel Multidimensional Item Response Theory

Introduction:  Positive school climate has been consistently associated with many desirable student outcomes (Thapa et al., 2013). The U.S. Department of Education (USDOE) conceptualized a three-factor model for school climate consisting of safety, engagement, and environment. This three-factor model has been validated using the MDS3 climate survey in high school samples within classical testing theory (CTT) framework (Bradshaw et al., 2014). School climate surveys often consist of polytomous items (for e.g., Bradshaw et al., 2014; Cornell, Shukla, & Konold, 2015) which are well suited for the application of item response theory (IRT) in order to obtain unbiased estimates (Embretson & Reise, 2013; Rhemtulla, Brosseau-Liard, & Savalei; 2012).  Moreover, school climate is often treated as a school characteristic with researchers aggregating all variables to the school-level without any psychometric examination. Accordingly, the present study applied multidimensional multilevel IRT to the middle school version of the MDS3 survey to examine the construct validity of the three-factor model at both the student and school-levels.

Methods:  Participants included 15,099 students from 27 middle schools. We employed multidimensional graded response models to accommodate polytomous data and sub-factors of the three broad factors of school climate (engagement, environment, and safety). All analyses were conducted in Mplus 7.3 using WLSMV estimator with theta parameterization. Given that the students were nested within their schools, we employed multilevel analyses with specification of TYPE= TWOLEVEL.

Results:  Consistent with the high school findings, Engagement consisted of six factors: academic engagement, teacher, student, and whole-school connectedness, culture of equity, and parental engagement at student and school-levels. Model fit was good (CFI = .98, RMSEA = 0.02, SRMR = 0.046 at student-level & 0.057 at school-level). Correlations between factors were high and statistically significant (>.55 and >.8 at student and school level). The hypothesized four factor model for environment also fitted the data well (CFI = .97, RMSEA = 0.02, SRMR = 0.056 at student-level & 0.10 at school-level). Associations between its sub-factors (rules and consequences, physical comfort, support, and less-disorder) were significant and positive (>.44 and >.74 at student and school level). Unlike the 3 factor Safety factor that was found for high schoolers, the middle school Safety factor consisted of two factors: bullying climate and peer deviance (correlation = 0.53 at student and .65 at school-level). Model fit was acceptable; CFI = .94, RMSEA = 0.037, SRMR = 0.10 at student-level & 0.09 at school-level.

Conclusion:  This study presents evidence for the construct and convergent validity of the three-factor model for middle schools which, for the most part, was similar to what was found for high schools. In addition, these multilevel models provide evidence for creating and interpreting the school-level scale scores.