| JOURNAL HOME | CME HOME | THIS MONTH | PAST ISSUES | ETOC | COLLECTIONS |
| AUTHORS | REVIEWERS | EDITORIAL BOARD | FEEDBACK | RSS | HELP |
| ||||||||||||||
| ||||||||||||||
|
|
|
|
||||||||||||
|
||||||||||||||
Department of Anesthesiology, Dartmouth Hitchcock Medical Center, Lebanon, New Hampshire
Address correspondence and reprint requests to Joseph P. Cravero, Department of Anesthesiology, Dartmouth Hitchcock Medical Center, One Medical Center Dr., Lebanon, NH 03756. Address e-mail to Joseph.Cravero{at}Hitchcock.org.
 |
Abstract |
|---|
 Top Abstract IntroductionMethodsResultsDiscussionReferences |
|---|
|
Introduction |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
The Dartmouth Operative Conditions Scale (DOCS) was designed specifically to codify the appropriateness of the conditions during medical procedures for children facilitated by various sedation/analgesia techniques (i.e., simple distraction, sedation/anesthesia, or pure analgesia). The DOCS compares conditions (or patient state) present during a procedure in terms of pain, sedation, movement, and side effects from sedation (oversedation). In this way the scale is designed to be simple and flexible enough to allow for comparisons among different providers, pharmacologic interventions, and nonpharmacologic techniques.
We present the method that was used to develop DOCS and the validation of its use as a tool in evaluating interventions used to decrease pain and anxiety. The scale is not meant for daily use but rather is intended for use in conjunction with video-taping to allow an objective comparison among techniques across a broad range of practice. Careful application and evaluation of video data with DOCS will allow the formulation of ideal procedural sedation/distraction interventions for procedures in a given institution or medical system.
|
Methods |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
We used techniques frequently applied in the field of human factors engineering (6–8). Complex systems such as pediatric sedation may be characterized by their inputs and outputs and by the transfer function that relates the two. Rather than a direct input-output model, modern control theory assumes that there is an intermediate variable—the state. The fundamental hypothesis of this approach is that detecting and defining state will allow better control, allowing one to avoid problem states and gravitate toward desirable states (8). When applying this approach in the context of pediatric sedation, patient state is the critical system variable. As operators, sedation providers take actions that are aimed at producing the goal state from baseline and correcting patients who enter problem states (screaming in pain or hypoxia from apnea or airway obstruction) to goal states (sleeping or other nonstressed conditions). To ensure a comprehensive model of patient states for pediatric sedation, four tasks were completed:
The final working model of pediatric procedural sedation is presented in Figure 1. In this model, time is represented on the x axis, and patient state is displayed on the y axis. At time = 0, the patient (black circle) is in his/her preprocedural state not experiencing pain from the procedure or side effects from sedation. The patient is therefore (by definition) in the goal state. As time elapses, the patient is taken through the time frame of the procedure. During the procedure, there is potential for the patient to experience pain and anxiety that result in adverse behaviors, thus putting the patient at risk. Likewise, there is potential for the patient to experience unintended side effects of the sedation that is delivered (apnea or hypotension), which can also yield harm. All of these adverse effects are represented by the R factors in the model. Control loops, or C factors, represent the work performed by the health care providers attempting to counteract the possible adverse effects or accomplish a task. These interventions include giving sedation to an anxious child or providing jaw thrust to a child whose airway is obstructed. The objective for the sedation provider is to keep the patient in the goal zone (away from dangerous side effects) and deliver him/her back to the baseline state and ready for discharge.
|
A rating scheme was devised from the spectrum of patient states identified during development of the pediatric sedation model described above. Numerical values were assigned so that positive numbers reflected increasing activity states and negative numbers reflected decreasing activity states. The numbers were assigned to signify the balance desired between increasing activity associated with reaction to the procedure being performed, and effects of interventions (usually medications) were aimed at ameliorating pain and stress (thus decreasing activity). The resulting scale was named the Dartmouth Operative Conditions Scale (Table 1). The score for any patient at a particular moment in time was derived by summing the scores in all four categories to yield one numerical value describing the patient state. It is important to recognize that there is no connotation of adverse effect associated with negative numbers in this scale. Rather, the scale was designed to reflect the desire to balance patient activity and sedation intervention to yield neutral (acceptable) states for procedures.
|
Careful review of the video data versus summed DOCS scores revealed that the total DOCS scores could be divided into three general categories of patient states during the procedures: (a) under-controlled states with DOCS score of more than 2, correlating with patient behaviors such as crying, thrashing, and severe agitation; (b) safe states with DOCS scores of 2 through –2, correlating with patient behaviors that lacked any evidence of pain, stress, or side effects from sedation medications; and (c) oversedated states where the total DOCS score was less than –2, correlating with states in which the sedative and respiratory compromise associated with sedation medications were not adequately treated.
From the conception of this scale, we recognized the possibility for mixed states where a child might show signs of distress from the procedure (crying) but would also show signs of side effects from sedation (eyes closed or noisy breathing). We hypothesized that these states would be relatively rare and undertook this validation process (in part) to test the frequency and impact of these states.
After the design of the DOCS and IRB approval, another 95 procedural sedations were video-taped from the time sedation was administered to the time patients were considered to be back to their baseline consciousness. The camera was placed in the most unobtrusive location possible that also allowed visualization of the patient and monitors. No attempt was made to select out certain types of procedures, specific providers, medications, or types of patients. For this phase of the study, providers of sedation/anesthesia included radiology nurses, pediatric cardiologists, pediatricians, pediatric residents, pediatric oncologists, and anesthesiologists. These video tapes represented more than 356 h of sedation.
Out of this video tape data, 70 video clips of 30-s duration were created to give short segments that represented sedation practice as it would be viewed by providers using the DOCS with video tape data. The clips could not be absolutely randomly selected because we needed to assure that the general state or status of the patient did not change during the 30-s duration. In addition, we wanted to assure the inclusion of a range of patient states (calm awake states, agitated states, general anesthesia, deep sedation, moderate sedation, minimal sedation, and oversedation states). Because most of our video data included patients who were calm and sedated, the majority of randomly selected clips would have been of that state only. We therefore selected 20% of the video clip examples of this state and then continued to randomly select clips until all possible DOCS scores were included.
The video clips were selected by one investigator who had no knowledge of the development of the DOCS and no background in medical sedation (JJ).
The DOCS was tested with 10 health care providers to assess inter-rater reliability and again a year later to confirm intra-rater reliability. These health care providers were chosen to reflect the variety of providers who might use the scale including one nurse practitioner, two certified nurse anesthetists, two recovery room registered nurses, one medical student, one pediatric resident, one anesthesiology resident, and two attending physicians in anesthesiology. All raters were given a 5-min introduction to the use of the DOCS from the same investigator (GTB). They were then given two sample video clips with the rating done for them. At this point they were presented with the 25 video clips and asked to rate them using a computerized DOCS rating screen. The four categories for DOCS rating were arranged so that a point and mouse click entered the data in each category, and the cumulative score was automatically calculated. (Fig. 2). The clips were shuffled and presented to the rater a second time at the same sitting to evaluate the consistency with which the initial rating was performed. One year later, each of the raters was retested using the same 25 video clips and computer application and all clips were again shuffled to assure that the order of presentation would be different from the original rating experience. The data from the DOCS scoring of the video clips were analyzed in two ways: (a) raw score and (b) inclusion into under-sedated states, acceptable sedation states, and oversedation states, as outlined above.
|
DOCS scoring (both intra-rater and inter-rater) were compared using the kappa statistic (11). The kappa statistic is a measure of agreement. A kappa of zero would imply the amount of agreement results from chance alone, whereas a kappa of 1.0 suggests perfect agreement. For this study, a kappa more than 0.61, which implies substantial agreement, was selected as the minimum for acceptable agreement (12).
Criterion validity refers to the extent to which a measure relates to the other measures that would theoretically support the concept (or construct) being measured. It also should be measured whenever there is no universally accepted criterion. Because there are no scales designed to measure the same state characteristics during procedures as the DOCS, we searched other disciplines for a tool with similar end-points to the DOCS. We believed that purely sedation-oriented scales address only the level of consciousness and therefore are inappropriate for a DOCS comparison because DOCS is designed to measure acceptable conditions during a procedure regardless of sedation level. However, the COMFORT® scale (intended to measure the state of patients on ventilators in the intensive care unit [ICU]) is not based strictly on sedation level, but rather measures overall comfort of critically ill patients. It was therefore chosen as an appropriate comparison tool (4,5).
Because it is specifically designed for the ICU patient, some adjustments had to be made to the COMFORT® score to make it similar enough to the DOCS to enable its use for comparison purposes. The specific changes to the COMFORT® score included: (a) one component of the score is an evaluation of the patient’s comfort or cooperation while on a ventilator. This component was deleted for the purposes of this investigation because procedural sedation patients are not (generally) on ventilators. (b) Because of the relatively dynamic nature of procedural sedation practice, the 2-min observation time required by the COMFORT® score was not practical. Instead, for the purposes of this comparison, we had to rate the patient over a 30-s period.
The result of this alteration was a modified COMFORT® score that grades each of the following categories on a scale from 1–5: alertness, calmness or agitation, physical movement, muscle tone, facial tension, arterial blood pressure, and heart rate. The criteria used for rating in these categories were identical for that applied in the standard COMFORT® score.
Twenty separate comparisons were made between the DOCS and the COMFORT® scores based on 20 unique, randomly selected 30-s video clips from our video data set (using the same criteria outlined for the inter-rater reliability test). One investigator (JPC) was presented the video clips in random order and asked to perform a DOCS rating. The clips were then shuffled and represented for rating with the modified COMFORT® score. Comparison of the DOCS and COMFORT® scores were made using linear regression and Spearman rank-order coefficient. The raw DOCS score was used in this analysis and was compared with the sum total of the COMFORT® score.
Construct validity refers to the ability of a scale to measure what it is intended to measure. To perform this validation test, we scored 30 unique 30-s video clips that were selected (using the same criteria as mentioned for inter-rater reliability) at different times after the initial sedation/analgesia intervention was initiated. The clips were scored (using DOCS) by one rater who was blinded to the time after initiation of the sedation intervention. After completion of scoring, the results were cross-referenced with the time after the sedation intervention to determine how scores changed with time. Changes in scores revealing improved operative conditions with time would indicate that the DOCS measured the effect of the sedation interventions as desired. With extremes of time after sedation intervention, scores should return to baseline as the effect of the intervention wears off. The Wilcoxon signed rank test was used to evaluate the changes in scores measured at time of the induction compared with 20 min after the induction and at 20 min after the induction compared to 20 min after emergence.
|
Results |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
In the 25 paired assessments of video clips that represented sedation practice, the DOCS score was completed with excellent inter-rater reliability among the 10 health care providers (kappa = 0.84; se = 0.021; P < 0.001).
Ten health care providers completed the DOCS score for each of the 25 video clips on 2 separate occasions 1 yr apart. There were 250 scoring pairs, including 10 where the rater scored the same scenario differently. The DOCS score was completed with excellent intra-rater reliability over a 1-yr interval (kappa = 0.91; se = 0.051; P < 0.001).
The DOCS scores were observed to correlate well with the COMFORT® scores for the 20 video clips (Spearman correlation coefficient = 0.88; P < 0.001) (Fig. 3).
|
The distribution of DOCS scores 20 min after the anesthetic induction was significantly lower than the scores before the induction (P < 0.001). In addition, DOCS scores 20 min after emergence were significantly higher than DOCS scores 20 min after initiation of sedation/distraction intervention (Fig. 4).
|
|
Discussion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
The data on safety related to pediatric procedural sedation are even more troublesome. Most sedation studies look at tens of patients, with very few including several hundred to one thousand patients (11–14). Data concerning oxygen desaturation and or the need for airway intervention in a cohort are sometimes (but not always) included. Most often, the data on these complications or near-critical events have been collected in a retrospective manner and depend on the charting of an individual at the time of the sedation without knowledge that the data would be used for study purposes. With few exceptions, the conclusion of any one of these studies is that the sedation technique studied is safe and effective because no serious morbidity or mortality occurred in the time frame of the study. If we accept that the rate of serious morbidity or mortality for pediatric sedation should be no more than that for general anesthesia (approximately 1/10,000), we are left with the fact that all of these studies are underpowered to conclude that a given technique meets this standard. More detailed analysis of the state of the patients who undergo a given sedation technique is required. Observation of subtle events that may well represent harbingers of unsafe practice would advance our ability to judge the true safety of one type of intervention versus another.
The DOCS was conceived to help with the above difficulties in evaluating sedation practice. The scoring system performed well in all aspects of the validation process. There was acceptable agreement in DOCS scores determined by different health care professionals with a spectrum of professional backgrounds and levels of experience. The DOCS scores determined by the same health care professional at two separate sessions a year apart also displayed acceptable agreement. Furthermore, the DOCS scoring system correlated closely with another similar scoring system, the COMFORT® score, confirming the criterion validity of the DOCS score. Construct validity was demonstrated by description of the change in DOCS score over time, with a return to presedation score after completion of the procedure. This validation process confirms that the DOCS scoring system can allow quantification of the conditions present during the procedure.
Whereas we do not suggest that this scale will be used on a daily basis, the use of video data collection and DOCS scoring will allow a more detailed analysis of sedation techniques. In this way, two techniques for bone marrow biopsy (i.e., ketamine versus midazolam) could be compared not only for "was the procedure completed?", but also for what the child’s behavior, degree of movement, and pain was like during the procedure. This type of detail is critical in comparing how well sedation techniques work and will yield much more useful information than the studies that currently only give the most crude data of procedure completion.
The DOCS will also be helpful in assessing alternative sedation techniques. Previously published sedation scales do not allow comparison of the conditions produced through the use of sedation with those accomplished through distraction or other noninvasive techniques. Because documentation is focused on the sedation level, the nonsedation techniques (by definition) do not work. DOCS scoring allows comparison of the patient state during pediatric procedures regardless of the depth of sedation. For instance, a lumbar puncture performed with excellent distraction methods could be compared directly with one performed under deep sedation, with respect to comfort of the child and the operating environment during the procedure. It would also allow an investigator to identify potentially dangerous conditions present during the procedure. We hope that this type of direct quantifiable technique for comparison will lead to more consideration of these nonsedating interventions and promote those strategies that really work.
Pediatric sedation is delivered by a wide range of practitioners in a variety of practice settings. Comparing the effectiveness of work performed across such a broad spectrum has always been difficult because of the huge number of variables involved. In developing the DOCS, we used a human factors methodology and process control theory to distill the critical features of the work output and to model how any practitioner must approach the state feedback control challenge of procedural sedation. We strongly believe that the application of these concepts to sedation practice has allowed us to develop a scheme for thinking about and evaluating medical work that is unique and particularly helpful for areas of practice that are as difficult to evaluate as pediatric sedation practice.
Methodological difficulties with our study include the fact that our scale is attempting to measure a new outcome—that of patient state as opposed to sedation level. Whereas there are many pain and sedation scales, none of these is intended for the purposes of judging the same operating conditions that we were interested in investigating. In our evaluation of criterion validity, we were compelled to use a modified version of the frequently used Comfort® score for comparison. Whereas we feel this is not ideal, because this modification negates the proven validity of the Comfort® score, this modified scale actually comes quite close to measuring the same general patient state as DOCS. Another problem involves the possibility of mixed states, where a child could be both under-controlled and sedated at the same time. Our analysis reveals that this type of state is relatively rare and (in our experience) did not change the general state categorization of the patient.
Finally, we once again recognize that the methods described for using DOCS would not be practical on a day-to-day basis. The scale is actually intended to be used as a tool for research in conjunction with video-tape data collection. The scale would be ideal for use as an objective measure in comparing work among different sedation providers at a given institution or among providers at separate institutions. It could also be used to compare work by the same provider using two different methods for sedation. Appropriately applied, the scale could be used to compare efficiency of sedation (time to achieve sedation and the time required to return to baseline) and safety of techniques, even when a relatively small cohort is involved.
We have described a new scale for evaluating pediatric patients undergoing sedation for diagnostic and therapeutic procedures. This scale, DOCS, was shown to be a valid measure for qualifying the state of a patient during a sedation or distraction intervention.
|
Footnotes |
|---|
|
References |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
