Measuring Fidelity in the Transfer of Suicide Prevention Training

Introduction: We examined fidelity in the transfer of training of a manualized suicide prevention program (ASIST; Lang et al., 2008) conducted as part of a large randomized control trial of the intervention for crisis hotline counselors.  The program utilizes a train-the-trainer (TTT) approach for new instructors to learn to deliver a 2-day workshop (referred to as T4T). To our knowledge, this is the first rigorously conducted observational investigation of the quality of a TTT. Methods: 17 of 18 (due to loss of video recordings) national crisis hotline centers recruited for the dynamic waitlist design and randomized trial participated in the fidelity study.  Two staff from each center participated in the TTT program (n=34).  Each pair of trainers conducted and video-taped 2-day training with center staff on two separate occasions for a total of 66 trainings.  Seven segments from the training were rated for fidelity operationalized as adherence to the manual and competence in delivery measured by seven unique Adherence and one general Competence measure. A total of 324 observations were coded. Coders average ICCs ranged from .71-.98. The mixed model was conducted for the nested data: two training times within each trainer, and two trainers within each center.  Results: Trainer Competence and Adherence ratings were not predicted by a number of demographic and other variables (e.g., education level, confidence).  Adherence and Competence ratings were correlated at r = .49, indicating at 25% overlap of unique, but related concepts. Trainer, in the model as a random effect, was consistently significant in all models. We examined if trainers improved from time 1 to time 2, with trainer as a random effect and training time as a fixed effect. This analysis showed that time was not significant for either Adherence or Competence.  Overall Trainer Adherence was categorized as percent of program content delivered:  below 60% (30% of trainers), 60-75% (35%), or above 75% (35%). For Competence, trainers were grouped into categories:  predominantly deficient (41% of trainers), more capable than deficient (41%), and predominantly capable (18%). On average, trainers delivered about two-thirds of the manualized program; however, there was enormous variation.  Competence also showed wide variability.  Fidelity differed across the seven segments:  higher ratings were seen for lecture-based segments and the lowest ratings were for segments with active learning and practice elements.  Conclusion: There was considerable variability in fidelity which challenges the assumption that the target audience (in this case hotline counselors) received training as expected.  How faithful the second generation training must be in order to be a test of the intervention is an empirical question.