| JOURNAL HOME | CME HOME | THIS MONTH | PAST ISSUES | ETOC | COLLECTIONS |
| AUTHORS | REVIEWERS | EDITORIAL BOARD | FEEDBACK | RSS | HELP |
| ||||||||||||||
| ||||||||||||||
|
|
|
|
||||||||||||
|
||||||||||||||
From the Northern Clinical School, University of Sydney, Department of Intensive Care, Royal North Shore Hospital, Sydney, Australia, the Department of Nutrition, Royal North Shore Hospital, Sydney, Australia and Foothills Medical Centre, Calgary, Alberta, Canada
 |
Abstract |
|---|
 Top Abstract IntroductionMethodsResultsDiscussionSummaryReferences |
|---|
|
Introduction |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionSummaryReferences |
|---|
A problem that faces clinicians when examining the literature is how to discern which trials are of sufficient methodological quality to merit changes to clinical practice. Previous systematic reviews of nutritional support studies (4,5) have identified high-quality trials using composite methodological quality scales (6). Research suggests that composite quality scales may actually mask important differences in true methodological quality (7).
In an evaluation of a 25-item composite quality scale based on the 1996 Consolidated Standards of Reporting Trials (CONSORT) statement (8), Huwiler-Muntener et al. found that "the true quality of a substantial portion of well conducted trials and of trials of low methodological quality will be misjudged." True methodological quality was defined using three key criteria: 1) concealment of random treatment allocation, 2) appropriate use of blinding, and 3) presentation of an intention-to-treat (ITT) analysis. These three key criteria are relevant to all types of clinical trials, and when they are not addressed, estimates of benefit (or harm) obtained from any randomized controlled trial (RCT) are more likely to be biased (9).
The purpose of this review is to use the three key criteria of true methodological quality to assess nutritional support studies conducted in the critically ill and to determine if true quality is improving over time. To provide a benchmark reference, the true methodological quality of nutritional support studies will be compared with results obtained from a previously published review of clinical trials of sepsis therapies (10).
|
Methods |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionSummaryReferences |
|---|
All RCTs comparing primary nutritional support interventions, conducted in adult critically ill patient populations, and reporting clinically meaningful outcomes (11) were eligible for consideration. Publications based on subgroups of patients from larger published trials and studies reporting only surrogate outcomes (12) were not eligible. Detailed appraisal was restricted to manuscripts published in the English language (13).
A study was determined to have been conducted in a critically ill patient population if the manuscript reported any of the following:
A study was judged not to involve a critically ill patient population if none of the above criteria were met and:
We used a broad definition of trials of nutritional support to include any comparison of the process of providing, or provision of, macronutrients, micronutrients, vitamins, and/or minerals. Parenteral nutrition was defined as an IV solution containing protein and a source of nonprotein energy with or without lipids.
Manuscripts of all included trials were assessed for the presence of three key criteria: 1) maintenance of allocation concealment in the randomization process, 2) use of blinding at any level (e.g., blinding of research personnel, health care workers, data collectors, outcome adjudicators, biostatistician), and 3) presentation of data such that an ITT analysis could be performed.
In addition to the three key criteria listed above, each manuscript was also assessed on the overall adequacy of the statement describing the randomization process, and, if loss to follow-up was present, whether missing outcomes were reported by treatment arm. Additionally, loss to follow-up was quantified and judged as to whether it was excessive. Excessive loss to follow-up was defined a priori as missing outcomes on more than 10% of all randomized patients (10).
The appraisal of nutritional support studies was conducted independently by all three authors. Any differences of opinion were resolved by discussion.
The original CONSORT statement was published in 1996 (8). To determine whether methodological quality changed over time, trials published in 1996 or earlier were compared with trials published after 1996 (post-CONSORT).
The nature of the literature search, study selection, appraisal process, and methodological quality of the included sepsis trials has been reported elsewhere (10). Sepsis trials reporting clinically meaningful outcomes (11) were eligible for comparison. All sepsis trials were originally graded using an extensive 57-item composite scale, which included an assessment of the key domains of true methodological quality listed above. Primary comparisons between nutritional support studies and sepsis trials were restricted to these key domains.
All P values reported for dichotomous outcomes were obtained using Fisher’s exact test. A P value 
0.05 was considered to indicate statistical significance. A P value 0.10 but more than 0.05 was considered to indicated a trend towards statistical significance.
|
Results |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionSummaryReferences |
|---|
Eight-thousand-three-hundred-one patients were randomized into the 111 eligible nutritional support studies. The median number of patients randomized per study was 51, with a range from 12 to 398. Only 3 of the 111 nutritional support studies adequately addressed all 3 domains of true quality. Nine of the 111 studies failed to address any of the 3 key criteria. With regard to the maintenance of allocation concealment and presentation of results in an ITT format, 12 of 111 studies addressed both and 40 of 111 failed to address either.
Forty sepsis trials reporting clinically meaningful outcomes were eligible for primary comparison. The eligible sepsis trials randomized 13,240 patients, with a median of 121 patients per trial and a range from 22 to 2,199 patients. Further details on the eligible sepsis trials are presented elsewhere (10).
Seventeen percent (19 of 111) of nutritional support studies used a randomization approach that is known to adequately maintain concealment of the allocation process. Twenty-seven percent (30 of 111) of studies adequately described the approach used to randomly allocate patients into treatment arms.
Compared to sepsis trials, there was a trend towards the more frequent use of randomization methods that are less likely to maintain allocation concealment (12 of 40 versus 19 of 111, P = 0.10). Nutritional support studies were also significantly less likely to provide an adequate overall description of the method of random allocation used (26 of 40 versus 30 of 111, P < 0.001).
Nutritional support investigators reported the use of blinding, at any level, in 31% of studies (35 of 111). Sepsis trials used blinding at any level significantly more often than nutritional support studies (32 of 40 versus 35 of 111, P < 0.001).
Nutritional support studies presented results in an ITT format (outcomes reported on all randomized patients) 57% of the time (64 of 111). This was significantly less often than sepsis trials (37 of 40 versus 64 of 111, P < 0.001).
Eighteen percent of all nutritional support studies had an excessive degree of loss to follow-up (20 of 111). Compared to sepsis trials, nutritional support studies were significantly more likely to have excessive loss to follow-up (0 of 40 versus 20 of 111, P < 0.001).
Seventy percent of nutritional support studies with <10% loss to follow-up reported patient loss by allocated treatment arm, whereas only 25% of nutritional support studies with excessive loss to follow-up (more than 10%) reported losses by treatment arm (19 of 27 versus 5 of 20, P < 0.001). In nutritional support studies that failed to report patient outcomes, the median loss to follow-up was 9.5%, with a range of <1% to 34%.
Sixty-four of the 111 nutritional support studies were published after the dissemination of the CONSORT statement in 1996 (post-CONSORT). There was no significant improvement over time in the maintenance of allocation concealment (5 of 47 versus 14 of 64 post-CONSORT, P = 0.14). However, significantly more post-CONSORT trials provided an adequate description of the method of random allocation actually used to assign patients to treatment groups (7 of 47 versus 23 of 64, P = 0.017).
Although there was a significant increase in the use of blinding in the post-CONSORT period (9 of 47 versus 26 of 64, P = 0.023), there was no significant improvement in the presentation of ITT results (29 of 47 versus 35 of 64, P = 0.56). In the 47 trials with documented loss to follow-up, there was no significant improvement in the reporting of losses by study arm (10 of 18 versus 14 of 29 post-CONSORT, P = 0.766).
|
Discussion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionSummaryReferences |
|---|
It is possible that sepsis trials do not represent a standard for the conduct of clinical trials to which nutritional support studies should be held. Based on the findings of a previously published review (10), the overall methodological quality of sepsis trials has been described as "poor" (14). Indeed, an informal comparison to other multidisciplinary trials suggests that there is much room for improvement in sepsis trials (15). For these reasons, it is possible that sepsis trials provide an unreasonably low standard with which nutritional support studies have been compared.
There are many reasons that could explain the current disparity in methodological quality between sepsis trials and nutritional support studies. It is possible that the more stringent licensing requirements that must be addressed by most sepsis therapies results in better designs. It is also possible that sepsis trials receive better funding, are more often peer reviewed, or are more likely to have a collaborative approach to their design. Any of these explanations is possible; however, none justifies the continued poor conduct of future nutritional support studies. An increase in the use of blinding by nutritional support investigators since the publication of the CONSORT statement suggests that improvements in other domains are possible. The use of sealed, opaque, sequentially numbered envelopes is an inexpensive and effective way to maintain allocation concealment in any clinical trial. Similarly, complete follow-up and reporting on all randomized patients, especially if outcomes are readily available, is all that is required to present an ITT analysis.
Despite the existence of excellent resources to guide the trialist in the conduct (16,17) and reporting (18) of RCTs, methodological deficiencies have been reported in many areas of medical research (19–24). Although there is evidence to suggest that methodological quality is improving over time (15,10), objective research specifically evaluating intensive care-based clinical trials suggests "that more consideration to the methodological quality" (25) is needed.
The majority of methodological appraisals use a composite scale to assess overall quality; however, there has been controversy in the literature over the appropriateness of this approach (14). The goal of a composite scale is to numerically combine information on different features of a trial into an overall score. Many different published scales exist and they "differ from one another in almost every respect: how and why the items were selected for inclusion, the number of items, reliability, approximate time to complete, and scoring range. Little attention has been given to the construct that the scales are assessing. With one exception, the scales are uniformly weak in how they were developed" (26).
Although there may be disagreement over the relative importance of many items included in composite scales, extensive reviews of the methodological literature consistently agree on the importance of three criteria: 1) the maintenance of allocation concealment (9), 2) the presentation of results from all randomized patients in the form of an ITT analysis (27), and 3) the appropriate use of blinding (28). Appropriate application of these three design features consistently results in a more unbiased estimate of treatment effect. Because methodological research has demonstrated that overall scores based on composite scales may not adequately reflect the presence of these three key criteria (7), it has been recommended that an assessment of true methodological quality should be based on an appraisal of these three key criteria separately (9).
The term allocation concealment is used to describe a process of randomizing patients into a clinical trial that protects researchers, clinicians, and patients from predicting upcoming group assignments. Concealing the knowledge of upcoming group assignments "prevents researchers from (unconsciously or otherwise) influencing which participants are assigned to a given intervention group" (29). As early as 1898, medical investigators realized the importance of removing "subjective judgement" from the process used to assign patients to treatment groups (30). Indeed, although many believe that the 1948 Medical Research Council trial of streptomycin in tuberculosis (31) is a landmark trial because it was the first to use random allocation, it was truly innovative because it was the first trial to use sealed envelops to maintain allocation concealment (32).
An extensive review of 148 trials published in the British Medical Journal, Journal of the American Medical Association and the Lancet found that 50% of the trials used a randomization technique that maintains allocation concealment; however the reporting of allocation concealment improved significantly, from 39% to 61%, in all 3 journals after the publication of the original CONSORT statement (15). We found that only 17% of published nutritional support studies used a process for allocating patients to treatment groups that maintained allocation concealment, which was approximately half as often as sepsis trials (30%). Numerous reviews have demonstrated that trials with inadequate or unclear allocation concealment can produce up to 40% larger estimates of treatment effects (33).
In general, it is very difficult to maintain allocation concealment if some form of alternating group allocation process (pseudo-randomization) is used (32). Examples of sentences describing the randomization process obtained from nutritional support studies include: "Patients were randomized into one of two nutritional subgroups (Total Parenteral Nutrition or Naso-Gastric feeding) based on their date of admission" (34) and "Patient selection was carried out in a random fashion, patients being assigned alternately to one or the other group according to the order in which they were admitted to the ICU" (35). Although both of these trials clearly assign patients to treatment based on an alternating group allocation process (pseudo-randomization), both also incorrectly claim to be "randomized." These examples serve to illustrate that, to determine whether allocation concealment was maintained, it is extremely important for trials to include a sentence adequately describing the randomization process (32).
The simplest randomization process that maintains allocation concealment is the use of "sequentially numbered, opaque, sealed envelopes" that are opened after a patient is recruited into a trial (33). Likewise, a centrally controlled randomization process, whereby a randomization center is contacted by phone, fax, e-mail, or via the web after a patient is recruited into the trial, is also an excellent way to maintain allocation concealment (33). The use of either approach requires only one sentence to describe adequately; however, it should be noted that the minimum description of either approach includes the statement that randomization occurred after the patient was screened as truly eligible and recruited into the trial.
The term "double-blind" is frequently used to refer to a process whereby both the patient and the health care team are unaware of which study treatment the patient is actually receiving. Because it may be important to understand whether the researchers, outcome adjudicators, data collectors, and even the analysts were also blinded, use of the term double-blind is discouraged and an explicit list of exactly who was blinded is preferable (36).
Our appraisal found that nutritional support studies used blinding significantly less often than sepsis trials (31% versus 80%, P < 0.001). Although there has been a significant increase in the use of blinding in the nutritional support literature since the publication of the CONSORT statement (19% versus 40%, P = 0.023), blinding was still used half as often as it was in sepsis trials.
We appreciate that it may be easier to achieve blinding in a trial that compares simple IV infusions, such as many sepsis trials. Although some are quick to claim that blinding is not possible when interventions are more complex, the critical care literature is replete with examples where trialists have been able to develop novel and innovative processes for establishing blinding of complex interventions (37,38). Indeed, the appropriate use of blinding may decrease over-optimistic estimates of treatment effect by up to 26% (39). Regardless of the intervention evaluated in a trial, if a subjective outcome such as ventilator-associated pneumonia or suspected infection is important, it is always possible to blind outcome adjudicators. Likewise, to ensure that the accuracy and completeness of follow-up and data collection are equivalent in both groups, it is usually possible to blind the primary data collectors.
An ITT analysis compares outcomes obtained from all patients enrolled and randomized into a clinical trial. Inclusion in an ITT analysis does not depend on whether patients actually satisfied the study entry or exclusion criteria, whether they actually received treatment, whether a protocol violation was recorded, or whether treatment was discontinued (27). To conduct an ITT analysis, full and complete follow-up is required on all patients randomized into the trial. As compared to an ITT analysis, an efficacy subset analysis may include only patients who received an adequate dose of the study treatment, who were truly eligible for study enrollment (met all eligibility criteria and none of the exclusion criteria), or who did not have any protocol violations.
Although the study protocol may identify the criteria used to define the efficacy subset analysis before the trial begins, because the subset of patients themselves cannot be identified until after randomization, it cannot be claimed that the properties of randomization apply or that the subset provides an unbiased assessment of treatment effects (40). An efficacy subset analysis is more susceptible to bias than many investigators believe.
For example, consider an efficacy subset analysis conducted on 80% of patients selected from a 200-patient study, with 20 patients excluded from each arm. Because the exclusion rate is similar in each arm, it might appear that there is very little chance of bias. Simulation studies conducted under reasonable assumptions have demonstrated that with as little as 10% loss to follow-up in each arm, the chance of obtaining a false positive result can easily double (40). The only way to ensure that bias has not entered an efficacy subset analysis and resulted in a false positive result is to report the actual outcomes for each patient not included in the analysis. In essence, this requires complete follow-up on all randomized patients until "the death of the patient...or the end of the study" (40).
Fifty-seven percent of nutritional support studies presented results in an ITT format. Because patients may withdraw informed consent after randomization, which leads to true loss to follow-up, it may not be reasonable to expect 100% follow-up from every single trial. However, 92% of sepsis trials presented a true ITT analysis.
The majority of nutritional support studies that failed to report outcomes on randomized patients presented an efficacy subset analysis in preference to an ITT analysis. In such a situation, it is always possible that outcomes were available on the missing patients and that investigators simply chose not to report outcomes on patients who did not qualify for the efficacy subset analysis. Failure to record or report an outcome on a patient who is not truly lost to follow-up should be strongly discouraged. The only way to determine whether an efficacy subset analysis is biased is to compare the results to those obtained in the ITT analysis (40). This requires full follow-up on all randomized patients.
|
Summary |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionSummaryReferences |
|---|
The use of three simple design features can improve the reliability of the results obtained from any clinical trial. Sealed, opaque, sequentially numbered envelopes are an inexpensive and effective way to maintain allocation concealment. There are many different levels at which blinding can be used to improve the reliability of any trial, and reporting of outcomes of all randomized patients, regardless of whether an efficacy subset analysis is conducted, should be considered mandatory. If appropriate nutritional support of the critically ill truly can improve mortality by 10% to 13% (3), the only way clinical trials will be able to detect an improvement of this magnitude is by implementing key design criteria that reduce bias.
In the absence of definitive (Level I) evidence (43), systematic reviews and evidence-based guidelines based on high-quality trials can support clinical decision making (44). However, the strength of the conclusions reached by overviews and guidelines is intimately related to the quality of the individual trials included (39). Because trials that assign patients to treatment based on an alternating group allocation process (pseudo-randomization) and those that contain excessive (>10%) loss to follow-up are prone to severe bias, we strongly recommend that these trials not be included in systematic reviews and evidence-based guidelines. In addition, wherever possible, methodologists should also conduct sensitivity analyses to determine if clinical recommendations differ when they are based on high versus low quality evidence as determined by the three key measures of true methodological quality.
|
Footnotes |
|---|
Supported, in part, by grants from the Australian and New Zealand Intensive Care Foundation and the NorthCare Foundation.
Accepted for publication July 20, 2004.
Address correspondence to Gordon S. Doig, PhD, Royal North Shore Hospital, Pacific Highway, St. Leonards, NSW, Australia 2065. Address e-mail to gdoig{at}med.usyd.edu.au.
|
References |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionSummaryReferences |
|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
