Reliability of Intensive, Repeated Measures in Context-Dependent, Multi-Language Setting

Introduction: In time-intensive research designs, such as ecological momentary assessment (EMA), it is important to closely examine different aspects of measurement properties of intensively repeated data. The primary goal of such measures is to capture meaningful within-subject variability of a construct and to reliably assess dynamic, micro-level processes. In this paper, we consider different sources of variance that represent the extent to which level-specific reliability is assessed. The current study provides evidence for the extent to which 1) dynamic and time-varying nature of the constructs of interest can be reliably assessed, 2) the psychometric properties are similar across different contexts (i.e., smoking vs. not smoking), and 3) whether these psychometric properties are preserved across languages used (English vs. Korean). Methods: Our sample included 78 Korean American emerging adults (18-25 yo). The majority of the participants completed the EMA surveys in English (n=61, 78%). Participants were instructed to respond to signal-contingent (random; 5 times/day) and event-contingent (smoking) prompts. We selected 3 items with the highest loadings from 4 scales: Negative and Positive Affect (NA, PA), Cigarette Craving (CC) and Anhedonia (ANH). VARCOMP procedures in SAS were conducted to partition variance and level-specific reliability indices were calculated. Results: Between-participant reliability (R_1F) were found to be moderate to high for all 4 scales for both languages (ranging from .65 to .86), except for NA assessed in Korean language, especially when participants completed the measures while smoking (R_1F=.35). Reliability for within-participant level (R_C) for all measures completed in English was moderate to good (.72 to .83) and was comparable across the two contexts. For measures completed in Korean, in contrast, higher within-participant reliability was observed when smoking, for PA, CC, and ANH.  The NA measure completed in Korean showed poor performance in differentiating within-participant fluctuation over time for both contexts (.39 to .41). Notably, within-participant reliability for cigarette craving was higher for random prompts for English measures, but the opposite pattern was observed for Korean measures. Discussion: Our findings highlight time-varying contexts may play a role in how consistently a scale performs in between-person and within-person level. In multi-cultural and multi-language settings, it is also recommended that future investigators consider the possibility that language preference might influence multi-level measurement characteristics.