Measuring Engagement in Active Involvement Interventions to Reduce Substance Use Among Adolescents

Introduction: Narrative Engagement Theory (NET) and the Theory of Active Involvement (TAI) are theories that posit behavior change through user engagement. Both NET and TAI suggest that a program’s ultimate impact on substance use depends on how involved and engaged participants are in a given program, with more engaged youth likely to report better outcomes. In practice, measurement of engagement is a challenge and has relied primarily on self-report measures. Fortunately, the emergence of e-learning delivery platforms offers another avenue for assessment of user engagement in real time via program analytics captured for each individual user. However, it is unclear to what extent these data are useful indicators of engagement and how they relate to self-report measures. The current study sought to address these questions via a pilot-study of REAL media, a web-based drug prevention intervention.

Methods: Participants were 31 4-H youth (54.8% Female, 45.2% White) ages 13-17 who participated in a pilot trial of REAL media in New Jersey during Fall 2015. The pilot session lasted two hours during which youth navigated the program and rated their engagement in each module, or level. Participants were compensated with a $30 gift card. Program analytic data included whether participants participated in optional content and responses to questions posed during the program.

Results: Average self-report ratings of engagement, including realism (M = 4.43, SD = .42), interest (M = 4.07, SD = .62), and identification (M = 3.85, SD = .60) subscales, were all in a positive direction on a 5-point scale from Strongly Disagree (1) to Strongly Agree (5). Program analytic data indicated that users engaged in an average of 4.32 optional segments (out of 10). Further, participants offered thoughtful responses to open-ended questions that demonstrated their understanding of challenging curriculum concepts. Optional depth user data were significantly correlated with the realism subscale of the self-report engagement data (p = .01), but not the interest and identification subscales.

Conclusions: Both program analytic data collected in real time as well as self-report data suggested high levels of engagement by participating youth. Youth who engaged in more optional segments reported stronger agreement that program content was believable, but there was no relationship with other self-report indicators. Program analytics may offer a unique and important indicator of engagement and should be included in future studies that include outcome data.