JOURNAL HOME CME HOME THIS MONTH PAST ISSUES ETOC COLLECTIONS
AUTHORS REVIEWERS EDITORIAL BOARD FEEDBACK RSS HELP
A&A International Anesthesia Research Society
QUICK SEARCH:   [advanced]


     

This Article
Right arrow Full Text (PDF)
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Email this article to a colleague
Right arrow Similar articles in this journal
Right arrow Similar articles in Web of Science
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Download to citation manager
Right arrowRequest Permissions
Citing Articles
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Lee, A.
Right arrow Articles by Gin, T.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Lee, A.
Right arrow Articles by Gin, T.
Related Collections
Right arrow Economics and Health Care Research
Anesth Analg 2002;94:372-377
© 2002 International Anesthesia Research Society

ECONOMICS AND HEALTH SYSTEMS RESEARCH

Applying the Results of Quantitative Systematic Reviews to Clinical Practice

Anna Lee, PhD, MPH, and Tony Gin, MD, FANZCA, FRCA

Department of Anaesthesia and Intensive Care, The Chinese University of Hong Kong, Prince of Wales Hospital, Shatin, New Territories, Hong Kong

Address correspondence to Anna Lee, PhD, Department of Anesthesia and Intensive Care, The Chinese University of Hong Kong, Prince of Wales Hospital, Shatin, New Territories, Hong Kong. Address e-mail to annalee{at}cuhk.edu.hk


   Introduction
Top
 Introduction
Conclusions
 References
 
Systematic reviews provide the best estimates of the true effects (both beneficial and adverse) of medical interventions (1). In this era of evidence-based medicine, clinicians are increasingly using systematic reviews to keep up with new evidence and to guide their clinical decision-making. Yet the main challenges for clinicians are to translate the results of systematic reviews into clinical practice and to provide optimal patient care. This concept is known as "applicability." Applicability addresses whether a particular treatment that showed an overall benefit in a study or systematic review can be expected to convey the same benefit to an individual patient (2).

This paper outlines a framework for how quantitative systematic reviews (meta-analyses) should be reported and how they may be used to identify those individuals in whom the treatment is likely to do more good than harm. We illustrate the concepts by using data from systematic reviews of ondansetron for the treatment and prevention of postoperative nausea and vomiting (PONV). Throughout this paper, we use the terms "baseline" and "underlying risk" interchangeably. Underlying risk is defined as the risk of event for a patient under the control condition; it indicates the average risk of a patient if not treated (3).

Limitations of Systematic Reviews and Meta-Analysis
In a quantitative systematic review, results of primary studies are combined by using complex statistical methods to estimate a common pooled effect with increased precision (4). The overall results of a meta-analysis represent an "average" treatment effect.

Before applying the results of systematic reviews to individual patients, clinicians must consider some of the limitations and biases (5). One of the weaknesses of meta-analysis is that it magnifies the problems of individual trials. In the case of systematic reviews of ondansetron, for example, the pooling of results from different studies may lead to inconsistencies in separate incidences of nausea, vomiting, and combined PONV because investigators do not often distinguish between nausea and vomiting but combine the symptoms of PONV into a single outcome (6). Second, if the overall conduct of the individual trial is poor, a summary treatment effect will generally be an overestimate of the "true" effect by up to 41% (7). Other specific methodological issues related to systematic reviews of ondansetron include variability of the underlying risk and event rates and small trial sample size, problems related to antiemetic comparisons without a placebo group and covert duplication of primary trials (8,9). Therefore, the interpretation of the relative efficacy and harm of antiemetic treatment can be difficult and a cautious attitude to accepting the results is warranted.

Existing Methods for Applying Results to Individual Patients
Despite the above limitations, systematic reviews and meta-analyses can highlight how effective a treatment is and identify subgroups of patients who would most benefit from an intervention. Systematic reviews and meta-analyses can guide clinical decision-making, but they are integrated with other factors in caring for an individual patient (10).

Common methods of applying the overall results of meta-analysis to individual patients are outlined below but these have limitations. First, the number needed to treat (NNT) to achieve one unit of benefit is a common clinically useful measure. However, pooled NNT may be misleading because NNT are sensitive to factors that change the baseline risk, such as the outcome considered, trends in disease risk, and clinical setting (11). A solution to this is that NNT should be derived by applying the relative risk reductions from treatment estimated by the meta-analysis to relevant baseline risks for different types of patients (Table 1) (11).


View this table:
[in this window]
[in a new window]
  		Table 1. Calculating Numbers Needed to Treat (NNT) for High- and Low-Risk Groups
 
Another erroneous method is to apply the estimated average treatment effect to all patients that meet the eligibility criteria of the clinical trials (12). Not all of these individuals will experience a net benefit, and some that were not eligible for the trial may benefit from treatment (2). There is evidence to suggest that treatments found to be beneficial in a narrow range of patients can have broader application in actual practice (1). Differences between study participants and patients in real-world practice tend to be quantitative (differences in degree of risk of the outcome or responsiveness to therapy) rather than qualitative (no risk or adverse response to therapy) (1).

Economic evaluation seeks to predict net change in benefits and costs arising from alternative approaches to providing a particular form of care (13) and can be useful in deciding the optimal strategy for treatment in a patient. The applicability of the results of an economic evaluation depends on the characteristics of the patients treated, the resources used that are related to outcome, the intrinsic effectiveness of the treatments compared, the viewpoint of the analysis and time scale of the decision to be made (14). However, a review of published economic evaluations showed that two-thirds of articles gave misleading conclusions about the relative costs of alternative treatments without supporting statistical evidence (15). Therefore, the reliability and validity of cost-effectiveness analyses of antiemetics to manage PONV and its subsequent applicability to individual patients may be questionable. Currently, there is no formal methodology for the conduct and reporting of systematic reviews of economic evaluations (13).

Framework for Applying the Evidence from Systematic Reviews to Individual Patients
An approach for assessing the applicability of results from systematic reviews to individual patients has been proposed by O’Connell et al. (2). This requires the following five questions to be addressed:

1. What are the beneficial and harmful effects of the intervention?
2. Are there variations in the relative treatment effect?
3. How does the treatment effect vary with baseline risk level?
4. What are the predicted absolute risk reductions for individuals?
5. Do the expected benefits outweigh the harms?

The first three questions address the issue of transferability of the average treatment effect (2). The last two questions cover aspects of individualizing the treatment decision through estimating the expected absolute risk reduction based on an individual’s baseline risk and then taking into account the patient’s preferences in determining benefits and harms (2).

When considering these five issues, there may be insufficient data (2). However, this will highlight the additional information required to be reported in future systematic reviews. In the next section, we use data from the systematic reviews of ondansetron to illustrate the five issues.

Appraising Current Systematic Reviews of Ondansetron
We identified seven quantitative systematic reviews of ondansetron for the prevention and/or treatment of PONV (16–22) by searching the MEDLINE (1966 to August 2000) and EMBASE (1980 to August 2000) electronic databases. One paper was a meta-analysis of individual patient data from three randomized controlled trials (16). Each systematic review was assessed according to the five criteria outlined above (Table 2). Table 3 shows how each systematic review met these criteria.


View this table:
[in this window]
[in a new window]
  		Table 2. Criteria for Assessing Applicability of Systematic Reviews of Ondansetron for Management of Postoperative Nausea and Vomiting (PONV)
 

View this table:
[in this window]
[in a new window]
  		Table 3. Applicability Criteria Reported by Systematic Reviews of Ondansetron for Management of Postoperative Nausea and Vomiting
 
What are the Beneficial and Harmful Effects of the Intervention?
In considering the potential benefits and harms, all authors reported surrogate measures (nausea, vomiting, nausea and/or vomiting) but did not consider patient-relevant outcome measures (patient satisfaction, length of stay in the postanesthesia care unit or hospital, time to full recovery), as these were not reported in the primary trials. The incidences of nausea (16,18,22), vomiting (16,18–22), and nausea or vomiting (17) were measured. The incidence of headache was measured in three papers (18,19,22). In addition, Tramer et al. (18) measured the incidence of increased liver enzymes, hypotension, constipation, and abdominal cramps. Other side effects, such as sedation, anxiety, restlessness, and abdominal muscle movements, were considered in a meta-analysis that compared ondansetron with metoclopramide and droperidol (22). The severity of adverse effects was not reported.

Are there Variations in the Relative Treatment Effect?
To understand how an intervention works, consideration is given to determine whether treatment effect is constant or whether it varies according to patient, disease, intervention, and measure of effect used (2). Variations in relative treatment effect were best examined using individual patient data (16) rather than a series of meta-analyses of subgroups from primary randomized controlled trials. In the individual patient data meta-analysis, significant factors associated with the incidence of PONV were dose of ondansetron, neuromuscular blocker use, country, volatile anesthetic, type of surgery, and age, the last two factors showing a significant interaction (16). Subgroup meta-analyses of patients undergoing eye surgery, ear/nose/throat surgery, and anesthesia with or without propofol were conducted in one paper (22).

Many authors examined the dose-response efficacy of ondansetron (17–21), different routes of administration (18,19), and timing of ondansetron administration in relation to surgery (19). There were variations in the optimal dose of ondansetron for prophylaxis; 4 mg (19), 8 mg in patients with a history of PONV (21), 4 mg and 8 mg both equally effective in patients with a history of motion sickness (20), 8 mg IV (18), 16 mg orally (18), or repeated 8 mg oral doses (16). For the treatment of established PONV, there was no clinically relevant dose response between 1 mg and 8 mg (17).

Heterogeneity can be defined as the variability or differences between studies in the estimates of effect (23). All systematic reviews except one (16) tested for heterogeneity because if present, potential sources of heterogeneity should be identified, as this can affect the overall conclusion as well as the clinical implications of the review. An important source of heterogeneity is the underlying risk (24). As the underlying risk is not a measurable quantity; the best estimate is the observed risk of events in the control group (24).

How Does the Treatment Effect Vary with Baseline Risk Level?
Of the seven systematic reviews, only one (18) assessed how treatment varied with baseline risk. A sensitivity analysis of two predefined ranges of baseline risk—early PONV within 20% to 60% of control event rates and late PONV within 40% to 80% of control event rates was conducted in one systematic review of ondansetron used for prophylaxis of PONV (18). As there was little variation in the control group in the trials included in the systematic review of ondansetron for the treatment of PONV (17), the criteria of how treatment varies with baseline risk may not be relevant. Which patients will obtain a net benefit will depend on the harms associated with the treatment. This tradeoff and how it relates to the underlying risk is discussed in more detail in the next section.

What are the Predicted Absolute Risk Reductions for Individuals?
To judge whether treatment is worthwhile for an individual patient, an appropriate NNT is estimated. This was done in one systematic review (19) using the Cook and Sackett method of applying patient’s baseline risk to the overall NNT (25).

Do the Expected Benefits Outweigh the Harms?
None of the seven systematic reviews discussed patient preferences associated with risk-benefit ratio, as the original trials did not report these data. The tradeoffs between risks and benefits can involve patient-centered outcomes, such as patient satisfaction and quality of recovery after anesthesia (26), measures that have been developed and validated recently.

Which Patients Will Benefit?
We applied the risk-benefit approach (12) to individualizing treatment as an example of how an anesthesiologist may consider a risk-benefit ratio using data from a meta-analysis and a cohort study. We show how this approach can be applied to the prophylactic use of IV ondansetron 4 mg using data from a systematic review (18). For the purposes of this example, benefit was defined as reduction in postoperative vomiting in the first 48 h and harm was defined as the incidence of headache. In clinical practice, consideration is given to all clinically important adverse effects.

Schmid et al. (27) showed that across a wide range of therapies, low-risk patients gain less absolute benefit compared with high-risk patients and as patient’s expected risk increases, the absolute risk reduction increases proportionally. We assessed the net benefit of prophylactic ondansetron by estimating the absolute risk reduction from the primary randomized controlled trials included in a systematic review (18). The overall risk reduction in postoperative vomiting associated with ondansetron was 20% (95% confidence interval [CI], 17%–24%) using a random-effects model that took into account within and between study variability. Therefore, the pooled NNT was 5 (95% CI, 4–6).

The risk of vomiting in the placebo group, an estimate of the underlying risk (x axis) was plotted against reduction in absolute risk of vomiting (y axis, plot 1) and excess absolute risk of headache (y axis, plot 2). Patients should weight the benefit and harm against each other. This can be done by assigning monetary value for the two items. In this example, the monetary value given by patients in decreasing and eliminating emesis compared with experiencing headache was a 1:1 ratio (28). The mean risk of vomiting in the control group was 56% (95% CI, 54%–59%), and the excess risk of headache was 1% (95% CI, -1% to 3%). The point at which the line of benefit and line of harm crossed was the threshold. Net benefit occurred only when the line of benefit was above the threshold of 5% (Fig. 1). It is important to note that this threshold will vary according to the weighting of benefit to harm ratio.



View larger version (25K):
[in this window]
[in a new window]
  		Figure 1. Benefit compared with harm for the prophylaxis of vomiting (0–48 h) by ondansetron 4 mg IV dose. Data taken from Tramer’s et al.’s systematic review (18). The number of risk factors is described in Koivuranta et al.’s study (29).

 
We also plotted the risk of postoperative vomiting against the number of risk factors (female sex, previous PONV, duration of operation more than 1 h, history of motion sickness, and nonsmoking status) as estimated by Koivuranta et al. (29) in Figure 1. This score was chosen because it has moderate accuracy in predicting PONV (30). These points on the x axis were 7% (0 to 1 risk factors), 17% (2 risk factors), 25% (3 risk factors), 38% (4 risk factors), and 61% (5 risk factors). Figure 1 shows that patients with 0 or 1 risk factors of postoperative vomiting would not benefit greatly from ondansetron prophylaxis and that high-risk patients would benefit most from prophylaxis.

Caution is needed in assessing the relationship between treatment effect and underlying risk as it can be misleading as a result of the "regression to the mean" phenomenon as the baseline risk forms part of the definition of treatment difference (24). In another words, even if there is no true relationship between baseline risk and treatment effectiveness, one is likely to be observed because of this statistical artifact. This problem is reduced if studies are large, if variation in true underlying risks is large, and if there are a large number of studies (24,31). A Bayesian procedure for random-effects meta-analysis has been proposed (3,24) to overcome this problem. Another alternative, if individual patient data are available for meta-analysis, is to relate treatment effects to individual patient covariates (predictors of risk). This would be directly useful for the clinician considering treatment for an individual patient (24).


   Conclusions
Top
 Introduction
 Conclusions
References
 
Appropriate reporting of meta-analysis is required to allow evidence-based decisions to be made. In this study, all systematic reviews adequately addressed the issue of transferability but failed to predict the absolute risk reduction for individuals. Therefore, it is difficult for anesthesiologists to clearly assess whether the expected benefits of ondansetron outweigh the harms for individual patients. More effort is required in reporting the results of meta-analysis in a way that will help anesthesiologists to individualize treatment. The framework used in this paper may help to improve the reporting of results from quantitative systematic reviews. Currently, there is limited applicability of results of ondansetron meta-analyses to individuals. A guideline, "the QUOROM statement" (32), may improve the quality of reports of meta-analyses of randomized controlled trials. However, we note that there is no emphasis in the guideline on how to apply the results of meta-analysis to the clinical setting.

Systematic reviews should address the following two questions:

Does the treatment work?
Which individuals will benefit most from the treatment?

Depending on the availability of data from primary trials and observational prognostic studies, we believe that if the framework described is applied after considering some of the limitations, the divergence between research evidence and current clinical practice may narrow.


   Acknowledgments
 
Supported, in part, by institutional and departmental funding.


   References
Top
 Introduction
 Conclusions
 References
 

  1. McAlister FA. Applying the results of systematic reviews at the bedside. In: Egger M, Davey Smith G, Altman DG, eds. Systematic reviews in health care: meta-analysis in context. London: British Medical Journal Publishing Group, 2001: 373–85.
  2. O’Connell D, Glasziou P, Hill S, et al. Results of clinical trials and systematic reviews: to whom do they apply? In:Stevens A, Abrams K, Brazier J,et al. The advanced handbook of methods in evidence-based healthcare. London: Sage Publications, 2001: 56–72.
  3. Arends L, Hoes AW, Lubsen J, et al. Baseline risk as predictor of treatment effect: three clinical meta-re-analyses. Stat Med 2000; 19: 3497–518.[Web of Science][Medline]
  4. Lau J, Ioannidis JP, Schmid CH. Quantitative synthesis in systematic reviews. Ann Intern Med 1997; 127: 820–6.[Abstract/Free Full Text]
  5. Egger M, Dickersin K, Davey Smith G. Problems and limitations in conducting systematic reviews. In: Egger M, Davey Smith G, Altman DG, eds. Systematic reviews in health care: meta-analysis in context. London: British Medical Journal Publishing Group, 2001: 43–68.
  6. White PF, Watcha MF. Has the use of meta-analysis enhanced our understanding of therapies for postoperative nausea and vomiting? Anesth Analg 1999; 88: 1200–2.[Free Full Text]
  7. Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995; 273: 408–12.[Abstract/Free Full Text]
  8. Tramer MR. A rational approach to the control of postoperative nausea and vomiting: evidence from systematic reviews. Part I. Efficacy and harm of antiemetic interventions, and methodological issues. Acta Anaesthesiol Scand 2001; 45: 4–13.[Web of Science][Medline]
  9. Tramer MR, Reynolds DJ, Moore RA, McQuay HJ. Impact of covert duplicate publication on meta-analysis: a case study. BMJ 1997; 315: 635–40.[Abstract/Free Full Text]
  10. Cook DJ, Mulrow CD, Haynes B. Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern Med 1997; 126: 376–80.[Abstract/Free Full Text]
  11. Smeeth L, Haines A, Ebrahim S. Numbers needed to treat derived from meta-analyses-sometimes informative, usually misleading. BMJ 1999; 318: 1548–51.[Free Full Text]
  12. Glasziou PP, Irwig LM. An evidence based approach to individualising treatment. BMJ 1995; 311: 356–9.[Free Full Text]
  13. Mugford M. Using systematic reviews for economic evaluation. In: Egger M, Davey Smith G, Altman DG, eds. Systematic reviews in health care: meta-analysis in context. London: British Medical Journal Publishing Group, 2001: 419–28.
  14. Kernick DP. Economic evaluation in health: a thumbnail sketch. BMJ 1998; 316: 1663–5.[Free Full Text]
  15. Barber JA, Thompson SG. Analysis and interpretation of cost data in randomised controlled trials: review of published studies. BMJ 1998; 317: 1195–200.[Abstract/Free Full Text]
  16. Haigh CG, Kaplan LA, Durham JM, et al. Nausea and vomiting after gynaecological surgery: a meta-analysis of factors affecting their incidence. Br J Anaesth 1993; 71: 517–22.[Abstract/Free Full Text]
  17. Tramer MR, Moore RA, Reynolds DJ, McQuay HJ. A quantitative systematic review of ondansetron in treatment of established postoperative nausea and vomiting. BMJ 1997; 314: 1088–92.[Abstract/Free Full Text]
  18. Tramer MR, Reynolds DJ, Moore RA, McQuay HJ. Efficacy, dose-response, and safety of ondansetron in prevention of postoperative nausea and vomiting: a quantitative systematic review of randomized placebo-controlled trials. Anesthesiology 1997; 87: 1277–89.[Web of Science][Medline]
  19. Figueredo ED, Canosa LG. Ondansetron in the prophylaxis of postoperative vomiting: a meta-analysis. J Clin Anesth 1998; 10: 211–21.[Web of Science][Medline]
  20. Figueredo E, Canosa L. Prophylactic ondansetron for post-operative emesis: meta-analysis of its effectiveness in patients with and without a previous history of motion sickness. Eur J Anaesthesiol 1999; 16: 556–64.[Web of Science][Medline]
  21. Figueredo E, Canosa L. Prophylactic ondansetron for post-operative emesis: meta-analysis of its effectiveness in patients with previous history of postoperative nausea and vomiting. Acta Anaesthesiol Scand 1999; 43: 637–44.[Web of Science][Medline]
  22. Domino KB, Anderson EA, Polissar NL, Posner KL. Comparative efficacy and safety of ondansetron, droperidol, and metoclopramide for preventing postoperative nausea and vomiting: a meta-analysis. Anesth Analg 1999; 88: 1370–9.[Abstract/Free Full Text]
  23. Clarke M, Oxman AD. Cochrane Reviewers’ Handbook 4 0 [updated July 1999]. In: Review Manager (RevMan) [Computer program], Version 4.0. Oxford: The Cochrane Collaboration, 1999.
  24. Thompson SG, Smith TC, Sharp SJ. Investigating underlying risk as a source of heterogeneity in meta-analysis. Stat Med 1997; 16: 2741–58.[Web of Science][Medline]
  25. Cook RJ, Sackett DL. The number needed to treat: a clinically useful measure of treatment effect. BMJ 1995; 310: 452–4.[Free Full Text]
  26. Myles PS, Weitkamp B, Jones K, et al. Validity and reliability of a postoperative quality of recovery score: the QoR-40. Br J Anaesth 2000; 84: 11–5.[Abstract/Free Full Text]
  27. Schmid CH, Lau J, McIntosh MW, Cappelleri JC. An empirical study of the effect of the control rate as a predictor of treatment efficacy in meta-analysis of clinical trials. Stat Med 1998; 17: 1923–42.[Web of Science][Medline]
  28. Engoren M, Steffel C. How much will patients pay to avoid unpleasant side effects of anesthesia and surgery [abstract]? Anesthesiology 1999; 91: A-1211.
  29. Koivuranta M, Laara E, Snare L, Alahuhta S. A survey of postoperative nausea and vomiting. Anaesthesia 1997; 52: 443–9.[Web of Science][Medline]
  30. Eberhart LH, Hogel J, Seeling W, et al. Evaluation of three risk scores to predict postoperative nausea and vomiting. Acta Anaesthesiol Scand 2000; 44: 480–8.[Web of Science][Medline]
  31. Sutton AJ, Abrams KR, Jones DR, et al. Methods for meta-analysis in medical research. Chichester: John Wiley and Sons, 2000.
  32. Moher D, Cook DJ, Eastwood S, et al. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUORUM statement. Lancet 1999; 354: 1896–900.[Web of Science][Medline]
Accepted for publication September 14, 2001.





This Article
Right arrow Full Text (PDF)
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Email this article to a colleague
Right arrow Similar articles in this journal
Right arrow Similar articles in Web of Science
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Download to citation manager
Right arrowRequest Permissions
Citing Articles
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Lee, A.
Right arrow Articles by Gin, T.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Lee, A.
Right arrow Articles by Gin, T.
Related Collections
Right arrow Economics and Health Care Research

Lippincott, Williams & Wilkins Anesthesia & Analgesia® is published for the International Anesthesia Research Society® by Lippincott Williams & Wilkins and Stanford University Libraries' HighWire Press®. Copyright 2002 by the International Anesthesia Research Society. Online ISSN: 1526-7598   Print ISSN: 0003-2999 HighWire Press