Development of a GRADE Extension for Complex Interventions

Introduction: Interventions commonly used in social disciplines that aim to prevent health disparities and enhance population quality of life, such as community-based HIV prevention interventions, are most frequently complex. These interventions typically operate via a sequence of changes in psychosocial, behavioural, and/or structural processes, targeting multiple outcomes while interacting with local contexts and the mode of intervention delivery. In this light, several researchers argue that evidence synthesis of complex interventions need to go beyond traditional meta-analysis that yield overall effect estimates. Specifically, reviews of complex interventions should assemble evidence of different types to explore variation in effects across populations, contexts and intervention implementation. This approach may be better suited to tackle health disparities and facilitate policy-relevant evidence synthesis. The Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach is the most widely adopted system worldwide for rating the quality of evidence and making healthcare recommendations. This paper outlines the specific challenges of applying GRADE to synthesise and assess the quality of evidence of complex interventions, and describes an ongoing project to extend, i.e. adapt GRADE for these interventions.

Methods: We retrieved forty systematic reviews from the Cochrane Database of Systematic Reviews (only those published from 2013 to May 2014 in CDPLPG, CCDAN and CPH), and we analysed their application of GRADE and the quality of evidence ratings. We then contacted the review authors to explore the challenges they encountered when applying GRADE to complex interventions.

Results: Specific challenges were identified in applying GRADE to complex interventions, namely assessments of heterogeneity, indirectness, performance bias and use of non-randomised studies in complex interventions. Authors perceived these challenges to contribute to frequent downgrading of the “best evidence possible” for these interventions. Meanwhile, GRADE was found to lack an analytic approach to integrate additional “parallel” evidence on intervention implementation and context.

Conclusions: An extension to the GRADE approach is needed to address identified challenges and enhance its value for evidence syntheses of complex interventions. To extend GRADE for complex interventions, we will use formal consensus methods. This will include an e-Delphi process to elicit the criteria for the new methodology, followed by a consensus meeting to finalise the criteria and build consensus on the GRADE guidance for complex interventions. We invite interested researchers to participate in this project and help us achieve the best-informed consensus on this guidance.