How to Critically Appraise Systematic Reviews in Order to Inform Clinical Decision Making

Globally, published peer-reviewed health care literature has grown exponentially, making it unrealistic for nurses to review the literature in a timely manner. Systematic reviews now provide a tool for nurses to be able to quickly access literature that is part of the evidence-based tools needed to inform decisions in clinical practice. However, in order to be able to fully and appropriately use systematic reviews to inform clinical decisions, nurses need to have an understanding of what a strong systematic review is, what the steps are in conducting a systematic review, and how to use existing tools to critically appraise systematic reviews. This presentation will address each of these three points.

Systematic review is the overarching name given to a review that examines and synthesizes the findings from multiple studies that are retrieved in response to answering a specific research question. Systematic reviews are to be conducted using a rigorous process that is transparent, and therefore can be replicated with similar results. Dependent on the research question, systematic reviews can be comprised of quantitative or qualitative studies, or in some cases, review both quantitative and qualitative studies within the same systematic review. With both quantitative and qualitative research methods, results can in part be presented in a narrative and tabular format. With quantitative studies, results are also typically determined statistically, using different statistical methods including effect sizes and meta-analyses. Several different synthesis approaches are used when reviewing the findings from qualitative studies.

In the systematic review process, specific steps are used to conduct a systematic review. In order to understand how to appraise a systematic review, it is important to understand the steps that occur in the systematic review process. These are the basic steps in a systematic review:

1. Determine who will be on the systematic review team
2. Determine a clear research question
3. Develop a study protocol that will guide the systematic review process
4. Choose the databases that will be searched, which key words will be used, and decide upon the inclusion and exclusion criteria that will be used for study retrieval
5. Literature retrieval 
6. Complete a dual blinded review
   1. Title review
   2. Abstract review
   3. Full-text review
7. Extract data from full-texts that meet inclusion criteria
8. Synthesize data  
9. Report and provide implications on the findings

When critically appraising a systematic review to be used for clinical decision making, essentially the nurse is seeking to find out if a clearly focused question was addressed, if valid methods were used in the review, what are the results of the review, and if the results are useful for clinical decision making. There are multiple tools that already exist in the international literature, that are available for nurses to use when appraising systematic reviews. For example these tools include the “10 questions to help you make sense of a review” sheet from the Critical Appraisal Skills Programme. The Centre for Evidence-Based Medicine “Systematic Review Appraisal Sheet” sheet asks five questions to determine if the results of the review are valid, and one question that examines how the results are presented. AMSTAR is a tool that uses 11 questions to assess the methodological quality of systematic reviews. A seven step checklist to analyze systematic reviews is provided by Ressing, Blettner & Klug (2009). Joanna Briggs Institute provides an eight step table “Critical Appraisal of a Systematic Review” in their “Appraising Systematic Reviews” information sheet.

Although each of these tools presents a different approach to critical appraisal, with some presenting very specific questions and others questions that are broad, briefly a critical appraisal of a systematic review for clinical decision making is guided by asking these types of questions:

1. Was there a clear question that guided the systematic review?
2. Was an ‘a priori’ study protocol used?
3. What were the inclusion and exclusion criteria in the systematic review?
4. Was the literature search comprehensive and described in detail?
5. Were there at least two reviewers working independently to determine study inclusion and data extraction?
6. Was the scientific quality of individual studies assessed?
7. Was heterogeneity between studies assessed?
8. Were the statistical methods used appropriate and precise?
9. Was the synthesis of the studies appropriately conducted?
10. Were limitations of the review provided?
11. Were the results presented clearly?
12. Are the results of the study applicable to guide clinical decisions?

Systematic reviews provide a rigorous appraisal and synthesis of large bodies of original research. Systematic reviews provide an efficient method for clinicians to quickly access healthcare literature that can be used to inform clinical decision making. However it is important for nurses to be able to quickly appraise systematic reviews to determine if these reviews are trustworthy and reliable sources of evidence-based research. By understanding what comprises a strong systematic review, the steps that are used to conduct a systematic review, and questions that are typically asked in appraising systematic reviews, nurses can confidently determine when and how systematic review results can be used to inform day-to-day clinical decisions.