Using SMS and Mobile Web-Based Survey Technology to Measure Knowledge Gain

Introduction

To further “promote the well-being of National Guard and Reserve members, their families and communities, by connecting them with resources throughout the deployment cycle,” the Yellow Ribbon Reintegration Program (YRRP) partnered with our organization. As an organization, we focus on evidence-informed decision making to guide program improvement efforts, utilizing the basic and prevention science literatures along with data collected from YRRP stakeholders.

One policy relevant question regarding YRRP impact is whether a focus on building awareness of resources is warranted (i.e., is there a lack of awareness on the part of attendees about the resources available to them?). To determine if attendees demonstrate gains in knowledge of resources over the course of an event, we implemented a pre/post-test data collection effort. This survey was distributed via SMS, mobile web, and hard copy.

Methods

Possible technology platforms were explored for a number of implementation considerations (e.g., cost, flexibility, ease of data extraction) and ProTexting.com was identified as best suited to the needs of this data collection. The company built a custom survey in which participants could scan a QR code or text a keyword to a shortcode to receive a SMS prompt (including a link to an online survey and instructions for completing via SMS). Upon completion of the pre-test, we scheduled an outgoing text invitation to participate in the post-test. This technology allowed us to track participants over time, and thus to compare pre- and post-test scores.

Results

This technology platform was pilot-tested at a YRRP event in September 2013. After data cleaning, the response rate was 37% at pre-test (n=129 out of 350 estimated eligible attendees) and 49% at post-test (n=63 out of the 129 pre-completers), rates comparable to if not higher than those with various distribution methods and similar populations (e.g., Baruch & Holtom, 2008). Of all included participants, 69% answered by web, 20% by SMS, and 11% by paper.  Through this technology we were able to identify modest gains in knowledge of resources available to attendees across the event.

Conclusions

SMS and mobile web-based survey technologies represent a flexible, low-cost tool for evaluation efforts as a means of influencing policy and practice. In this context, the technologies were utilized to conduct pre/post-surveys of attendee knowledge gain over a weekend-long event. These technologies could be applied to any number of purposes for prevention science, including but not limited to: communicating feedback to or from participants; providing technical assistance or coaching to staff; or keeping in contact with participants to reduce attrition in longitudinal studies.