Multiple Calibrations in Integrative Data Analysis: When Needed, How Many, and How to Combine?

The recently proposed Integrative Data Analysis (IDA; Curran & Hussong, 2009) has provided a new approach in the regime of research synthesis. IDA allows researchers to pull multiple studies into one and analyze the data integratively. Although this type of research requires tremendous collaboration efforts among researchers from different sites so data sharing could be possible, it has also been accompanied by many methodological challenges. For example, the complexity of IDA involving longitudinal data structure exceeds the capacity of many existing software packages. Frequently, analyses have to be separated into two steps, measurement harmonization and advanced statistical modeling. Bauer and his colleagues (Bauer & Hussong, 2010) suggested a practical calibration sample approach, of which a random sample, based on  one observation per individual, is drawn from the complete sample. Hence, measurement harmonization could be achieved by using the calibration sample without worrying about within subject correlation. Factor scores could then be estimated for next step of statistical modeling. There have been mixed reports on how well calibration sample method has performed. There are no established criteria on when calibration method may provide sufficient precision for further statistical modeling. The present study's  simulation-based analysis attempts to provide guidelines on how calibration method works with multiple factors being considered (i.e., sample size, growth characteristics, number of measurement occasions, and correlation structure of the measurements). Computational restrictions of simulation study preclude consideration of factors such as heterogeneity of studies, missing data pattern, different time trends, and spacing of the measurement..

For illustration purposes, this paper applies the method to an ongoing project that currently have data from six randomized control trials that tested the effects of Multidimensional Family Therapy (Liddle, 2002) on adolescent drug use among adolescents who were seeking drug treatment. Overall, approximately half of the participants were randomly assigned to MDFT or treatment as usual (TAU)  with participants aged 12 to 17 years.  Data were collected at treatment intake, 6 weeks, 2 months, 3 months, 6 months, 9 months, 12 months, and 24 months.  Substance use was measured by a substance use latent variable with indicators of positive urine analysis, 30-day usage from the TLFB instrument, POSIT score and the Personal Experience Inventory score.