Measuring Fidelity of Universal Prevention Curriculum Delivery in Pakistan: Development of Fidelity Checklists

The Universal Prevention Curriculum (UPC) is designed to demonstrate the application of prevention science to the design and implementation of evidence-based prevention interventions and policies. The concern about the implementation of prevention interventions is well represented in the mental health, health education and substance use prevention literature. Much of the literature is focused on the factors that are related to high levels of implementation fidelity of the delivery of the interventions. One of these factors that has received more attention recently is on the training received by the intervention implementer and on how to measure the impact of implementer training methods and more recently, although to a much lesser extent, the impact of training of master trainers or training of trainers (ToT). The major issues are related to reliability and validity.

This paper will present a pilot study to develop and validate self-assessment checklists for the delivery of the Universal Prevention Curriculum Coordinator Series (UPCC) by those trained to either serve as trainers-of-trainers or to train prevention-related professionals in monitoring the quality of the training including fidelity to key content, to structure of the training materials, and to the use of appropriate instructional strategies. The checklists along with the results of the pre-/post-tests serve to monitor the quality of the trainings. Although there are limitations to the use of self-assessment checklists, given the costs associated with direct observations by trained observers or through videos, self-assessment checklists have been developed and used effectively.

The fidelity checklists will be piloted during UPC trainings in Pakistan. There will be two trainers for each of these trainings plus an independent observer. Each trainer will make a self-assessment and an assessment of the other trainer during the training. The independent observer will complete the assessment for both trainers. In this way we will have at least two assessments for each module. Comparisons will be made on assessment scores and diverse differences will be discussed to refine the assessment measure. It will be important to assure that the self-assessments are comparable to those of the co-trainer and independent observer. In addition, the ease by which the self-assessments were made will be reviewed. Finally, an assessment form will be developed for completion by those being trained. This assessment will focus more on perceptions of presentation clarity, responsiveness, engagement in activities etc.

The development methodology and final tool will be presented. Focus will be placed on experience of using the newly developed fidelity checklist and its piloting results in Pakistan.