| JOURNAL HOME | CME HOME | THIS MONTH | PAST ISSUES | ETOC | COLLECTIONS |
| AUTHORS | REVIEWERS | EDITORIAL BOARD | FEEDBACK | RSS | HELP |
| ||||||||||||||
| ||||||||||||||
|
|
|
|
||||||||||||
|
||||||||||||||
From the Department of Anesthesiology, Intensive Care Medicine, and Pain Therapy, University Hospital Giessen, Giessen, Germany
Address correspondence and reprint requests to Dr. med. Bernd Hartmann, Abteilung Anaesthesiologie, Intensivmedizin, Schmerztherapie, Universitätsklinikum Giessen, Rudolf-Buchheim-Str. 7, D-35392 Giessen, Germany. Address email to Bernd.A.Hartmann{at}chiru.med.uni-giessen.de
 |
Abstract |
|---|
 Top Abstract IntroductionMethodsResultsDiscussionReferences |
|---|
IMPLICATIONS: The objective of this study was to evaluate prognostic models for quality assurance purposes to predict the occurrence of automatically detected intraoperative cardiovascular events in 58,458 patients undergoing noncardiac surgery. Two newly developed models showed good discrimination but, because of reduced calibration, their clinical use is limited. The ASA physical status classification and the Revised Cardiac Risk Index are unsuitable for the prediction of intraoperative cardiovascular events.
|
Introduction |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
The objective of this study was to evaluate prognostic models of perioperative outcome for quality assurance purposes. Automatically detected intraoperative CVEs taken from the database of a computerized record-keeping system were analyzed in patients undergoing noncardiac surgery. The performance of two established classification models used in routine for perioperative risk assessment, the Revised Cardiac Risk Index (RCRI) (10) and the ASA physical status classification (11), were assessed for their predictive value, although they were not developed primarily for these purposes. We then developed two new prediction models incorporating either ASA classification and operative data or the RCRI and operative data. Prognostic performance was tested using analysis of discrimination and calibration.
|
Methods |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
Systolic, diastolic, and mean arterial blood pressures (MAP), and heart rate (HR) were recorded at least every 5 min using noninvasive measurement or every 3 min with invasive measurement. All drugs used were entered manually at the moment of administration.
Structured query language queries were used for retrospective detection of CVEs out of the database. Relevant CVEs were defined as follows:
A randomly selected sample of 20% of the anesthetic records containing at least one CVE was re- viewed independently by two investigators to verify automatically detected CVEs and to eliminate possible artifacts.
The predictive power of a total of 24 variables was studied. Biometric variables measured were age (yr), sex, and body mass index (BMI, kg/m2). Disease-related variables measured were as follows:
Surgical variables measured were type of surgical procedure and urgency of surgery (elective, urgent: surgery within 6 h after admission; emergency: surgery within 2 h after admission).
Additionally, we calculated the RCRI according to Lee et al. (10). The RCRI was developed for prediction of cardiac risk based on six simple prognostic factors: high-risk type of surgery, ischemic heart disease, congestive heart failure, history of cerebrovascular disease, insulin therapy for diabetes, and preoperative serum creatinine >2.0 mg/dL. Patients with 0, 1, 2, or more factors were assigned to classes I, II, III, or IV, respectively.
Data for statistical analysis were exported from the database into the SPSS® statistics program (SPSS Software GmbH, Munich, Germany). The data pool was randomly divided into an evaluation (n = 29,437) and validation (n = 29,021) set. The association between CVEs and hospital mortality was assessed for significance using Fisher’s exact test.
Logistic regression with a forward stepwise algorithm (inclusion criteria: log likelihood test ratio based on maximum likelihood function) was performed to identify independent variables having a significant association with CVEs within a multivariate model. The variable with the strongest significant contribution (P < 0.05) improving the model was included. Variables already included in the logistic regression equation were tested for exclusion based on the probability of a log likelihood test ratio. The analysis ended when no further variables for inclusion or exclusion were available.
Two models for risk-prediction were built because of existing associations of ASA classification and variables concerning patients’ preexisting diseases. In modeling the first predictive system (model 1), all biometric, disease-related and surgical variables were considered. For the second model (model 2), only the ASA classification was used instead of the disease-related variables. Logistic regression was also used to derive new equations for risk prediction using CVEs as the dependent variable and the ASA classification/RCRI as the independent variable.
Prognostic model performance was assessed using analysis of discrimination and calibration. Discrimination refers to the model’s ability to distinguish between patients with or without CVEs. Calibration refers to the accuracy of the prediction when the number of predicted and observed CVEs is compared over the strata of an increasing risk of CVEs. The discriminative power of the new models, ASA classification and the RCRI, were assessed with ROC (receiver operating characteristic) curves (13). The ROC curve plots the percentage of true positive values (sensitivity) based on the individual score against the percentage of false positive values (1 - specificity). We used the Hosmer-Lemeshow goodness-of-fit C and H statistics (14) to evaluate the overall calibration of the two new predictive models, the ASA classification and the RCRI,which was considered satisfactory when the P value was >0.05.
|
Results |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
|
A comparable case mix of patients was found within the evaluation and validation sets. Table 2 shows the biometrical data (age, BMI, sex) of the patients with a minimum of one CVE and of those without any CVEs for the evaluation and validation set. Furthermore, the intensive care unit time and the length of hospital stay, ASA physical status, type of surgical procedure, its urgency, and the preoperative long-term cardiac treatment are shown in Table 2.
|
|
|
|
with z = 0.025 x (age) + 0.067 x (male gender) + 0.185 x (ACB/PTCA) + 0.546 x (valvular heart disease) + 1.430 x (arrhythmia) + 0.285 x (art. hyper- tension) + 0.508 x (carotid stenosis) + 0.330 x (hypovolemia) + 0.240 x (chronic renal failure) + 0.229 x (emergency) + 0.784 x (neurosurgery) + 0.696 x (thoracic surgery) + 1.303 x (major vascular surgery) + 1.048 x (hematopoietic/lymphatic surgery) + 1.148 x (gastrointestinal surgery) - 3.408.
In model 2, the probability of a relevant CVE can be calculated with the following equation: equation
|
|
with z = 0.024 x (age) + 0.260 x (ASA) + 0.716 x (neurosurgery) + 0.586 x (thoracic surgery) + 1.478 x (major vascular surgery) + 0.999 x (hematopoietic/lymphatic surgery) + 1.105 x (gastrointestinal surgery) - 3.708.
"Male gender" is coded as 1 and female gender coded as 0. Presence of the following factors is coded as 1, absence of these factors coded as 0: "ACB/PTCA" (condition after ACB or PTCA), "valvular heart disease" (aortic and mitral valve disorder), "arrhythmia," "art. hypertension" (arterial hypertension), "carotid stenosis," "hypovolemia," "chronic renal failure," "emergency," "neurosurgery," "thoracic surgery," "major vascular surgery," "hematopoietic/lymphatic surgery" (surgery of the hematopoietic and lymphatic system: splenectomy and extended lymphadenectomy), and "GI surgery" (gastrointestinal surgery: large intestinal operations such as gastrectomy, hemicolectomy, colectomy, and rectum resection, as well as hepatobiliary and pancreatic operations). The variable "ASA" (ASA classification) is equivalent to the patient’s ASA class (1–5).
The odds ratios and predictive models of ASA classification and the RCRI used in calculating the probabilities of occurrence of a relevant CVE are shown in Tables 3 and 4. Table 5 includes the AUC and results of the Hosmer-Lemeshow statistics. Neither the two new models nor ASA classification nor the RCRI showed acceptable calibration. Calibration curves for all predictive models are presented in Figure 1. However, the new models showed good discrimination with an AUC of 0.709 and 0.707 (Fig. 2) during validation. The discriminative power of model 1, which included detailed patient-related variables, was not superior to model 2, which contained only the ASA classification for describing the patients’ physical condition. The accuracy of the ASA classification and the RCRI to distinguish between patients with and without CVEs was lower (AUC 0.647 and 0.620, respectively).
|
|
|
|
|
Discussion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
With an incidence of 6% to 15%, CVEs are the most frequent events described in studies using manual recording (2,4,6,17,18). Reasons for the varying incidences include differing definitions, quality of anesthesia care, personal judgment of the anesthesiologist, and types of recording. The majority of the above-mentioned studies either recorded perioperative events manually or with computerized machine-readable record sheets. However, manual recording of vital signs data is often inaccurate compared with computerized record-keeping systems that record data automatically in real-time (19). The anesthesiologist’s tendency to record "corrected values" rather than those supplied by the vital monitor, thus tending towards smoothed trends, has been repeatedly described (5). Furthermore, significantly more frequent incidences of automatically detected perioperative events compared with conventional manual recording were found (1,20). This is an important finding, considering the significant association of hospital mortality and automatically detected intraoperative adverse events.
Some authors argue that artifacts may have a negative influence on quality of documentation. The study of Sanborn et al. (1) demonstrated that 25 of 494 adverse events were caused by artifacts, equivalent to 0.46% in a group of 5454 patients. Based on a 10% sample, the sensitivity rate of electronic scanning was 97.2%, and specificity rate was 98.4% (1). In contrast to Sanborn et al., we chose a definition for detection of adverse events that not only associated an event with a numerical reading but also included therapeutic intervention, thus eliminating the risk of falsely detected artifacts. However, it seems that incidences of adverse events are much more likely to be biased by manual recording than by artifacts.
External quality management studies essentially require exactly defined quality indicators that apply to a wide range of institutions. In this view, automatic detection of events and complications with an AIMS certainly offers advantages to resolve the problem of variability in incidences also depending on other factors, such as quality of anesthesia care or type of recording. However, individual therapy standards, monitoring techniques and sampling rate have to be considered. It should be acknowledged that prognostic models must be validated externally before they can be used in other settings and institutions. In the case of imperfect prognostic performance (21), customization of a model has proven to be a sufficient method to fit the model to local conditions (22).
Advanced age is a well-acknowledged risk factor for intraoperative CVEs, perioperative morbidity, and mortality (1,18,23–25). This variable is included in almost all risk indices (3,8,9,26). Thus it was also incorporated in the predictive models described in this study. Because of reduced organ function with increasing age, it may be assumed that advanced age bears an increased risk of intraoperative CVEs. However, in one study evaluating the RCRI, Lee et al. (10) could not prove a significant association between age and perioperative CVEs. An increased risk for male gender of perioperative CVEs has also been described in previous studies (3,18,23). In model 1, male gender was included as the independent variable, its predictive association, however, was low compared with other variables (odds ratio, 1.07). In other risk indices, gender is not included (8–10). Other authors found an increased risk for perioperative events in obese patients (3,24). This finding cannot be confirmed by our study. No association was found between BMI or body weight and CVEs. It has been previously shown that patients with preexisting cardiovascular diseases have an increased perioperative risk (7,8,26–28). This is also confirmed by the results of our study. Contrary to other risk indices (8–10), the presence of signs of heart failure or coronary heart disease did not increase the power of model 1. The key factor in our study was a history of therapeutic intervention in coronary arteries (ACB or PTCA). The association of ASA classification and perioperative risk is widely accepted (3,18,23,29) even though the original classification was designed only to describe physical status without an impact on prognosis. Our study, however, shows that inclusion of the ASA classification into a prognostic model substantially improves performance. It seems that the clinical judgment of the attending anesthesiologist is of paramount importance and exceeds in accuracy even more sophisticated models. However, within a predictive model, ASA classification alone does not allow adequate differentiation of patients with or without intraoperative CVEs. Urgency and the type of surgery have a critical influence on perioperative risk (8–10,18,23). Emergency operations increase the incidence of perioperative complications along with increased mortality. An increased risk prevails in intracranial, thoracic and abdominal procedures and major vascular surgery in other studies (8–10,18,23). We observed a higher risk for intraoperative CVEs in neurosurgery, thoracic surgery, vascular surgery, GI, and hemopoietic and lymphatic surgery. Lymphatic surgery was mainly comprised of splenectomy and major regional lymphadenectomy. These procedures had the strongest association among the variables in both predictive models. Although the newly modeled predictive systems showed good discrimination, they have a rather limited value for predicting intraoperative risk for CVEs because of poor calibration.
It has been repeatedly described that simple risk prediction indices showed similar or superior predictive characteristics than more complex indices. An example is the Cardiac Anesthesia Risk Evaluation Index developed by Dupuis et al. (30). Similar to the ASA classification, this index is simple, classifying patients into five groups according to their state of health and the planned surgical procedure. Its results are comparable to those of complex multifactorial indices. The RCRI according to Lee et al. (10) showed a significantly more frequent rate of discrimination for perioperative cardiac complications in the original study, compared with the cardiac risk index devised by Goldman et al. (8) and its modification by Detsky et al. (9). Despite its advantages, the RCRI had a poor predictive performance in this investigation. The index, originally derived for a prediction of severe cardiac complications in major elective noncardiac surgery, proved unsuitable for the prediction of intraoperative CVEs.
Reasons for poor prognostic performance of models including detailed data of medical conditions may be related to associations between variables. Logistic regression analysis is usually applied for modeling predictive systems allowing independent variables only. However, in cardiovascular diseases, this is rarely the case. Coronary artery disease often is associated with reduced myocardial function or arterial hypertension. When these variables are all included into a model, their contribution to discrimination remains small. Cohen et al. (23) investigated the association of anesthesia technique and mortality risk. They observed a significant correlation of mortality with preexisting diseases. This is why they included ASA classification instead of specific medical conditions in their model.
The predictive performance of our models was assessed using analysis of discrimination and calibration. This is state-of-the-art for predictive models, not only in anesthesiology, but also in critical care (31,32). Measures of discrimination and calibration are not provided in the original study of the cardiac risk index by Goldman et al. (8), nor in the modified cardiac risk index described by Detsky et al. (9). Thus, it is impossible to judge the discrimination and calibration of these indices. In their study, Lee et al. (10) described the discriminative power but not the calibration of the RCRI.
Although we were able to demonstrate in this exploratory investigation that data collected with an AIMS are suitable for developing multivariate models for identifying risk factors for CVEs, existing limitations of this study must be addressed. Both new predictive models calibrate poorly, especially in the area of higher probability, i.e., the expected incidence of a CVE is more frequent than the real incidence. A possible reason could be the common practice of allowing sicker patients to be treated by experienced anesthesiologists, leading to fewer CVEs than expected. Additionally, precautionary measures are used depending on the severity of accompanying illness and the scope of the procedure in both monitoring and timely therapeutic intervention. The models could be improved through a complex statistical analysis by weighing of the incidences of comorbities. Using the power of a modern AIMS, such complicated models could be created. However, in clinical routine without an AIMS, the knowledge of risk factors should be useful in increasing vigilance in those patients most at risk for CVEs, allowing for more timely therapeutic intervention. A further limitation deals with evaluating a possible influence of the depth of anesthesia on CVEs because, in our hospital, neuromonitoring is only used for selected cases. Furthermore, preoperative fluid status may have an impact on the incidence of intraoperative CVEs. Fluid status, however, cannot be assessed uniformly. We therefore relied on the clinical judgment of the performing anesthesiologists, who entered significant deviations from normal status into the AIMS. Finally, preoperative long-term cardiac medications obviously may have an effect on the incidence of CVEs. There is so much diversity in possible medications and their combinations that even in such a large number of patients the statistical power to detect influences would be substantially decreased.
In summary, the newly derived risk-prediction models 1 and 2 show good discrimination; however, their clinical use remains limited because of reduced calibration. Consideration of detailed information regarding individual diseases and symptoms in model 1 does not increase the predictive performance compared with model 2, which contains only the ASA classification to describe physical status. ASA classification without any other factors and the RCRI are unsuitable for the prediction of intraoperative CVEs. Contrary to manual recording of perioperative adverse events, computerized detection allows an exact definition of perioperative complications based on objective criteria.
|
Acknowledgments |
|---|
Statistical analysis was performed in cooperation with Moredata GmbH in Giessen, Germany.
|
Footnotes |
|---|
|
References |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
