Critical Appraisal of Systematic Reviews: Where Do You Start?

Critical Appraisal of Systematic Reviews: Where Do You Start?

Objectives: After the presentation the student will be able to:

1. Recognize the value of systematic reviews in clinical practice 2. Understand the methodology of Systematic Reviews 3. Applied previous knowledge to identify the parts of a systematic review. 4. Organized the steps to conduct a systematic review. 5. Appraise Systematic Reviews based on it’s validity, and applicability to practice

Systematic Reviews

A review in which evidence on a topic or research question has been systematically identified, appraised and summarised according to predetermined criteria. Systematic reviews may incorporate metaanalysis, but don’t have to.

High-quality systematic reviews are considered the best possible sources of evidence for evaluating treatment effectiveness.

Meta-analysis A statistical technique. Summarises the results of several studies into a single estimate, giving more weight to larger studies.

Publication bias When only studies with positive results are published, not the neutral or negative studies. If only published studies are included in a systematic review, it may overestimate the effect of the treatment or intervention.

Critical Appraisal of Systematic Reviews: Asking the Right Questions

A critical appraisal is basically a detailed examination of published research for the purpose of making a decision about scientific merit and, therefore, for making a decision about the use of the evidence in practice.

What is a Critical Appraisal?

Critical appraisal is also known as a research or evidence critique. Critiquing the literature, appraising the literature, or critically evaluating the literature are phrases that refer to a deliberate examination of a published research study and making a judgment as to the validity of its methods, importance of the question and results, and application to practice. A critical appraisal is basically a detailed examination of published research for the purpose of making a decision about scientific merit.

Critical Appraisal Steps A critical appraisal of a systematic review starts with asking questions about each of the which are related to validity and reliability of the methods used to produce the systematic review; the meaning and importance of the synthesized recommendations; and, finally, the ability to translate the findings into clinical practice.

There are critical appraisal tools available to specifically appraise systematic reviews that you can find from a variety of organizations.

DOES THIS REVIEW ADDRESS A CLEAR QUESTION? Remember that this first set of subquestions is related to the methodological rigor of the study.

You are assessing whether the question was focused and important and whether the study methods reduced the possibility of bias and minimized error as much as possible

Did the review address a focused clinical question? What was the specific question (should include population, exposure/intervention, and outcomes) and was it sensible – that is, was it too broad to be useful? You should find this information in the introduction and background sections of the

Did the authors look for the appropriate sort of papers?

The ‘best sort of studies’ would · Address the review’s question · Have an appropriate study design.

Were the criteria used to select articles for inclusion appropriate? Which criteria were used to select the articles that were reviewed. Did the systematic review only used randomized trials or if non-randomized trials were also included. Reviews which use only high-quality randomized controlled trials (RCTs) are less likely to have biased results and the likelihood of random error is reduced. If nonrandomized trials are included, then you would expect that the findings from the RCTs and nonRCTs are separated out for analysis . You should find inclusion and exclusion criteria in the Methods section of the article.

Do you think the important, relevant studies were included? Look for

Which bibliographic databases were used Follow up from reference lists Personal contact with experts Search for unpublished as well as published studies Search for non-English language studies

Did the review’s authors do enough to assess the quality of the included studies?

The authors need to consider the rigor of the studies they have identified. Lack of rigor may affect the studies results.

ARE THE RESULTS OF THIS REVIEW VALID? Were the individual studies assessed for validity? The methods section should give you this information.

You want to know how the researchers determined the studies were of high methodologic quality –

What were the predetermined quality criteria? The use of independent reviewers that showed good agreement (the kappa statistic is usually reported) would be a good thing to report. There should be details given so you can make your own assessment of how well they did their job. If the results of the review have been combined, was it reasonable to do so?

Consider whether ·

The results were similar from study to study The results of all the included studies are clearly displayed The results of the different studies are similar The reasons for any variations are discussed

WHAT ARE THE RESULTS? The “results” in these critical appraisal of systematic reviews questions refer to the conclusions made as a result of the synthesis of the evidence sources selected for this systematic review.

What is the overall result of the review?

Consider · If you are clear about the reviews ‘bottom line’ results · What these are (numerically if appropriate) · How were the results expressed (NNT, odds ratio, etc)

Are the results of the included studies clearly presented and discussed?

The purpose of the review is to summarize the data from the individual studies into a bottom-line answer. Remember that not all systematic reviews are meta-analyses, so the results may not be provided as a numerical answer, but as a qualitative summary.

How precise are the results?

Are the results presented with confidence intervals?

CAN I APPLIED THE RESULTS OF THE SYSTEMATIC REVIEW TO PRACTICE? Can the results be applied to my patient care?

This question is fairly self-explanatory. The critical question is “How like your patient were the patients in the review?”

Were all clinically important outcomes considered?

Are there outcomes that are associated with the outcome under review that you should be thinking about or would want information about? What are the costs associated with the therapy? What are the adverse effects associated with the therapy?

Are the likely treatment benefits worth the potential harms and costs?

Your patients need to be informed of the risks and benefits of all potential therapies. Patient decisions will change patient-to-patient. You cannot make assumptions about the patient’s beliefs. The patient’s values and beliefs must be taken into account when making clinical decisions.

Resources

Evidence-Based Practice: Critical Appraisal Tools for Systematic Reviews/Meta- Analyses Foley Center Library

Systematic Reviews & Meta-analyses: Critical Appraisal Virginia Tech Library