This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Email this article to a colleague Similar articles in this journal Similar articles in Web of Science Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Request Permissions Citing Articles Citing Articles via HighWire Citing Articles via Web of Science (3) Citing Articles via Google Scholar Google Scholar Articles by Vigoda, M. M. Articles by Lubarsky, D. A. Search for Related Content PubMed PubMed Citation Articles by Vigoda, M. M. Articles by Lubarsky, D. A. Related Collections Economics and Health Care Research Complications Technology

---

ECONOMICS, EDUCATION, AND POLICY

Changing Medical Group Behaviors: Increasing the Rate of Documentation of Quality Assurance Events Using an Anesthesia Information System

From the Center for Informatics and Perioperative Management, Department of Anesthesiology, Perioperative Medicine, and Pain Management, University of Miami/Jackson Medical Center, Florida.

Address correspondence and reprint requests to Michael M. Vigoda, MD, MBA, University of Miami/Jackson Memorial Hospital, Department of Anesthesiology, Perioperative Medicine, and Pain Management, 1611 NW 12th Ave. (C-300), Miami, FL 33136. Address e-mail to mvigoda{at}med.miami.edu.

Abstract

In the United States, quality assurance (QA) documentation is required by numerous agencies for each anesthetic performed. The goal of this study was to determine the effect of several interventions on the voluntary completion rate of QA documentation. We hypothesized that optimizing workflow integration would increase both QA completion rates and complication capture rates and promote long-term successful changes in reporting behavior. Whereas electronic scanning of anesthetic records may automate some aspects of QA, there will continue to be a need for anesthesiologists to enter QA documentation that cannot be automated. Starting from a baseline completion rate of 48%, we instituted a series of interventions. We successively increased the completion rate to 55% (education), 68% (workflow integration), and 78% (individual feedback). Each intervention increased the completion rate from the previous intervention (P < 0.001). The increased completion rate suggests better overall data capture, because the percentage of "no complication" entries decreased. After the study period, the completion rate increased to 94%, principally because of the improved workflow integration. As implementation of anesthesia information systems increases, attention to user interface design may be a key component in the functionality and utility of these systems.

Quality assurance (QA) documentation is required by numerous health care agencies (e.g., Joint Commission on Accreditation of Healthcare Organizations (1), Residency Review Committees (2) for Anesthesiology, and subspecialty fellowships (3,4)). Other industries have used quality control techniques (e.g., six sigma and statistical process control charts) to identify and correct deficiencies in their business processes. Health care (and specifically anesthesia) could benefit from a systemized approach to quality control. Ideally, the anesthesia QA process starts with the collection of accurate and reliable data from the anesthesiologist involved in the case. Most anesthetics in the United States are recorded on paper forms. Typically, a separate single-page form is used for QA data entry. There are few data on the completeness or accuracy of this process when using paper records, because data analysis using a paper-based system is time consuming and labor intensive. Voluntary QA reporting by the provider has been stigmatized for identifying only a small fraction of actual problems (5,6). Similarly, there are few published studies of how to maximize voluntary reporting using an electronic medical record (EMR) or the effect of default entries on QA data reporting. It is also clear that behavioral changes after educational initiatives in medicine (with or without an EMR) are difficult to sustain (7). We hypothesized that optimizing the workflow integration of an EMR (using intelligent design features) would markedly enhance both QA form completion rates and complication capture rates and promote long-term successful changes in reporting behavior.

EMRs can facilitate the collection and analysis of QA data by incorporating an electronic form into the anesthetic record. With a paper record, QA recording is always voluntary and self-reported. When an EMR is used, completion may be voluntary or mandatory, self-reported or automated. Voluntary reporting implies that the provider may or may not enter data into the system before closing out the case. Mandatory completion implies the QA form must be completed before closing a case. Self-reporting means the provider assesses the outcome, whereas automated reporting relies on database analysis by the computer.

METHODS

In 1997, an automated anesthesia record-keeping system (Picis, Wakefield, MA) was installed in all operating rooms (OR) at one institution where the department's faculty members provided anesthetic care. In addition to recording the vital sign data, the system allows the anesthesiologist to document events, medications, and QA entries. The electronic QA form, based on the preexisting hospital-approved paper form, was accessible to the anesthesiologists during the time they were providing care to patients. This form contains several subheadings (e.g., cardiac, respiratory, etc.), each of which contains a list of defined QA indicators that the user may select. In addition to these predefined QA indicators, the user is able to enter a QA complication not otherwise specified under any of these subheadings.

The completion rates of QA documentation were analyzed using Standard Query Language queries on the database containing all QA documentation from November 2003 to April 2004 (156 days). The completion rates during four distinct phases were categorized separately. Two months of baseline data were collected. Interventions occurred at approximately monthly intervals and were described in e-mail communications to the users, which discussed other issues with the application.

1. Baseline: Two months before any intervention
   
2. Education: The following month we notified all attending physicians, residents, and certified registered nurse anesthetists, informing them of departmental policy regarding QA documentation. Announcements were made at well-attended faculty meetings and departmental grand rounds.
   
3. Improved Workflow Integration: The following month we integrated the QA form into the users' workflow. After leaving the OR and transporting the patient to the postanesthesia care unit (PACU)/intensive care unit (ICU), users (anesthesiology residents or certified registered nurse anesthetists) launched the application to complete the documentation and print the record. After launching the application in the PACU/ICU, the first window which appears is the QA form (Fig. 1). Before this intervention, the initial window that appeared contained the graphical display of the vital signs, which resembled the traditional paper-based record. (Whereas QA complications could be entered throughout the case, the "No complications" entry was no longer accessible until opening the record in the PACU/ICU, just before printing the anesthesia record.)
   
   
4. Performance Feedback (one time): The following month, an individualized report summarizing the previous month's cases was sent to all anesthesia residents and certified registered nurse anesthetists stating their QA documentation completion rates.
   
5. Long term: For the next 3 mo, the improved workflow integration was continued, and e-mails were periodically sent reporting on the entire group's QA completion rate.
   

View larger version (39K): [in this window] [in a new window]   Figure 1. Quality assurance (QA) template is displayed as a flow sheet so that anesthesiologist can indicate the QA event in relation to other events in the anesthesia record. The template is accessible at any time during anesthesia care while in the operating room (OR). The above template, which includes the possibility of entering "No complications" is only available in the postanesthesia care unit (PACU) or intensive care unit (ICU).

RESULTS

The QA completion rate increased with each serial intervention (P < 0.001), as seen in Table 1. The daily completion rate ranged between 35% and 100% after the baseline period (Fig. 2). This variation resulted from a combination of factors. An occasional noncompleted form from a weekend case (when 10–15 cases per day are performed) accounted for a larger percentage of noncompleted cases than if this occurred during a typical weekday, which has a caseload of 50–70 cases per day.

View larger version (41K): [in this window] [in a new window]   Figure 2. Aggregate view of the quality assurance (QA) completion rates of each sequential intervention. The bars represent the number of cases performed each day, and the line graph represents the percentage of cases with completed QA forms. Note that there is variation in the number of cases during weekdays versus weekends, which accounts, in part, for some of the variation seen in the percentage completed. The mean completion rate is indicated within each phase.

After the study period was over, the QA completion rate continued to increase (Fig. 3). There was a positive association between the proportion of cases with the QA data form completed and the proportion of QA complications reported (Spearman rank correlation coefficient has a value = 0.81; P < 0.00001) (Fig. 4).

View larger version (11K): [in this window] [in a new window]   Figure 3. Quality assurance (QA) documentation completion rates from March 2003 to March 2005. The black rectangle represents the baseline period. The horizontal rectangles indicate the period of time that the three interventions (education, workflow integration, and individual feedback) were performed. Once initiated, the workflow integration was continued.

View larger version (17K): [in this window] [in a new window]   Figure 4. Rate of quality assurance (QA) complications reported as a function of the QA documentation completion rate.

DISCUSSION

Quality control techniques are used in both manufacturing and service industries to improve internal processes. By contrast, health care has primarily used QA to demonstrate to external reviewers that their processes meet specified standards. In general, major events (e.g., death, cardiac arrest, or myocardial infarction) that occur in the OR are reported and investigated. However, less obvious events may not be as well reported, as evidenced by several studies (5,6).

Traditional paper-based QA reporting systems have been used in identifying adverse outcomes, but these efforts are extremely labor intensive and subject to reporter bias. Katz and Lagasse (8), in reviewing almost 38,000 anesthetics, concluded that the method of reporting can influence the yield in detection of adverse outcomes because anesthesiologists reported 71% of these events, whereas chart reviewers reported 38%, and other hospital personnel (using critical incident forms) reported only 9.1%. Moreover, "there were significant differences among anesthesiologists, chart reviewers, OR nurses, and PACU nurses regarding factors that might lead them to report an adverse outcome" (8).

Others have shown that voluntary reporting does not ensure that all potentially reportable events are actually recorded in the QA database. Sanborn et al. (5) demonstrated that voluntary reporting severely under-estimated the incidence of intraoperative incidents when compared with electronic scanning and detection, because <5% of electronically detected events were self-reported. This group asked the anesthesiologist six questions relating to vital signs (i.e., heart rate, arterial blood pressure, temperature, and oxygen saturation). As the default answer to all questions was "No complication," their completion rate was 100%. The baseline completion rate seen in our study and positive correlation with increasing reporting and complications rates would suggest that default answers might falsely increase the completion rate and even more markedly under-estimate the true complication rate.

Our study was designed to analyze QA completion rates in a truly voluntary reporting structure. We hypothesized that making the user more aware of their documentation responsibilities and optimizing the user interface of an EMR using improved workflow integration would markedly increase both QA form completion rates and complication capture rates and promote long-term successful changes in reporting behavior.

We demonstrated that education, improved workflow integration, and individual feedback were all helpful in increasing the completion rate. However, we were concerned that continuing education and individual feedback were not desirable long-term strategies because they would be time consuming (education) or project a big-brother mentality (individual feedback), which might mitigate the intent of the QA process. Furthermore, whereas automated feedback works (9), it is likely that large numbers of continuous, ever increasing feedback e-mails would be counterproductive.

Body et al. (10) demonstrated that individual feedback and education regarding volatile anesthetic use was effective for reducing fresh gas flow rates, but the effectiveness was reduced without continued feedback. Repeated educational efforts may become ignored over time. Individual feedback may encourage anesthesiologists to simply complete the form without providing useful information. Furthermore, the highly touted individualized feedback begins to wane as an effective tool if every performance, economic, and documentation effort requires this degree of monitoring. Individualized feedback is successful at changing behavior and maintaining that behavior change as long as those efforts are not supplanted by another and another feedback to the same practitioners. One of the authors (DL) (7) with experience in the use of anesthesia information system-directed individualized feedback in the medical arena has termed this phenomenon "feedback pushback." Once behavior is changed, systems management must make it easier to do the right thing, versus slipping back into the less desirable older habits. Our long-term strategy in changing the process flow of QA form completion was designed with that in mind to make it easier for the anesthesiologist to complete the QA form.

The increase in QA reporting helped improve the quality of our departmental review process. Our anesthesia information system allowed us to do this in an inexpensive and time-efficient manner. Before these interventions, there were insufficient QA data to analyze, which limited our ability to formulate recommendations. Additionally, the increase in QA documentation yielded managerial benefits within the department, such as a timelier and more complete picture of QA events. A daily report of the cases involving complications, and those cases lacking QA documentation, is automatically e-mailed to the department chairman, QA director, and clinical director of the OR. The ability to notify these department members by automatically pushing the pertinent information to them increases their awareness and enables them to quickly act upon the results and implement changes.

A number of issues are beyond the scope of this study. Although the accuracy of the information reported in the QA documentation is essential to its usefulness in analyzing and potentially improving quality of care, verification of accuracy was not within the scope of our current effort. We recognize that there may be QA events that occur but are not reported, as demonstrated in other studies. Our next initiative is to design automated QA event reporting (based on electronic scanning of the record) that would prompt the user to either enter a QA event or clarify why an episode does not qualify as a QA event.

Our initial concern was raising the level of awareness regarding QA documentation responsibilities. Additionally, we did not attempt to define what constitutes a QA event or what level of deviation should trigger a QA report. Currently, there are two significant barriers limiting the usefulness of the QA process: the time required to collect and process the data and the reliability of the data. EMR analysis may partially address both of these obstacles. However, there will continue to be a need for voluntary reporting (e.g., difficult intubation, IV infiltration, obstructed endotracheal tube, and so on), and this was the focus of our study.

Closely related to QA reporting is outcome analysis. QA identifies bad outcomes, or a deviation from an accepted process. Ultimately, the practitioner and society really want a natural corollary to document optimal performance. Nascent efforts at aligning pay for performance in other medical specialties have been endorsed by both the Joint Commission on Accreditation of Healthcare Organizations (11) and the American Medical Association. Purchasers of health care (e.g., United States government (12), large employers (13), and health plans (14)) have launched pay-for-performance initiatives in an attempt to obtain quality health care for their dollars. Because anesthesiology is already regarded as a leader in patient safety efforts, some have suggested that pay for performance might have limited applicability in our specialty (15). However, in time, payers may reward anesthesiologists for appropriate usage of ß-adrenergic blockers, timely intervention in cases of hypotension, prophylactic postoperative nausea and vomiting treatment in appropriate patients, or documentation of bispectral index values during cases at higher risk of awareness. Because a robust QA process is required for participation in any of these eventual pay-for-performance schemes, the issue of how QA events can be documented in the anesthesiologist's workflow is not an insignificant concern. Anesthesia information systems are ideal for monitoring these trends in outcomes and QA completion.

There are a number of ways in which QA data can be obtained. When an EMR is used, automated QA reporting can be accomplished by electronic analysis of the record. A modification of this approach might be to have the record scanned for vital sign outliers and have the application ask the provider to comment on detected variances.

Anonymous reporting is another possibility, although some have questioned whether voluntary reporting of critical events is effective for QA (16). Despite its success in other industries (e.g., aviation), there is no mechanism for anonymous reporting outside of the closed claims database (17–19). Moreover, whereas anonymous reporting may hold promise for determining the cause of significant events, there remains a need for timely documentation of events that may not seem to be that significant but still merit attention in the overall care of the patient. This might include anesthesia-related events and delays in delivery of care (e.g., process issues such as waiting for surgeon, PACU/ICU hold, or waiting for equipment).

Whereas we support anonymous reporting, current practice dictates identifiable reporting, and we must pursue strategies that encourage a greater voluntary QA completion rate. We proved that multiple interventions yield higher completion rates. Not only did the completion rates increase, but an entirely unexpected benefit was the increase in the percentage of complications reported, suggesting that attention to detail yielded much more honest and complete recording of problems (Fig. 4).

One possible concern about the study's design is the nature of the serial interventions. Faced with a variety of possible interventions, we wanted to determine the relative increase as a result of going from a less targeted to a more targeted approach. Because others will face these same issues, we wanted to demonstrate the utility of various approaches. It is unclear whether we could have obtained equally impressive results by simply eliminating some of the earlier steps.

In as much as the ideal study would control for the inherent patient, personnel, and operational changes occurring within our health care system, we were not able to control for these variables. The baseline phase and the interventions were lengthy enough to reduce the effects of patient and personnel fluctuations on the study results. Additionally, the study took place in the middle of the academic training year, avoiding the significant change in personnel and their acclimation to our system.

One possible alternative study design might have been to divide the interventions among different physical locations (e.g., trauma, pediatrics, etc.) and observe the changes over time within these different locations. A randomized, controlled trial was considered but rejected because of concerns of crosstalk in our anesthesia care team environment.

Although it is possible that continued education and performance feedback would have eventually resulted in the same frequency of reporting, the integration of the QA form into the users' workflow was a less intrusive way of reminding them of their responsibility and increasing compliance. There was continued improvement despite the sequential elimination of the education and performance feedback phases. We attribute the continued increase in the QA documentation rate after the study period primarily to the improved workflow integration (Fig. 3).

The statistical analysis comparing each intervention to the previous state serves to identify the achievable incremental gain. Recognizing that there is reluctance to complete QA forms, we used a multifaceted approach that combined education, ease-of-use, and individual monitoring. We were concerned that starting with an individual monitoring approach would be perceived negatively, given the highly sensitive nature of QA reporting, and would ultimately reduce the benefit that our department could gain from an increased response rate.

Individual feedback and education are both effective interventions in the short run to change behavior. However, their effectiveness diminishes over time unless they are repeated. We demonstrated that optimizing the user interface of an EMR markedly enhanced both QA form completion rates and complication capture rates. Presenting the QA form to the user at the appropriate time for completion reminds the user to complete the form and helps lower the barrier to completion by decreasing the time and steps required for the user to complete the required documentation. Workflow integration is superior to education (and performance feedback), which remind the user to complete the QA form but do not assist in helping the user to do so. The tipping point for adoption of anesthesia software will only occur when applications are not only intuitive but simplify the anesthesiologist's required workload.

Footnotes

Presented, in part, at the 2004 American Society of Anesthesiologists' annual meeting, October 21–25, 2004, Las Vegas, Nevada; the 2005 International Anesthesia Research Society annual meeting, March 11–15, 2005, Honolulu, Hawaii.

Accepted for publication March 21, 2006.

REFERENCES

1. Manual for hospitals. Joint Commission on Accreditation of Healthcare Organizations.
2. Program requirements for graduate medical education in anesthesiology. www.acgme.org/downloads/RRC_progReq/040pr703.pdf. Accessed March 12, 2006.
3. Program requirements for residency education in pediatric anesthesiology. http://www.acgme.org/req/042pr297.asp.
4. Program requirements for graduate medical education www.acgme.org/acWebsite/downloads/RRC_PIF/045pif101.doc.
5. Sanborn KV, Castro J, Kuroda M, Thys DM. Detection of intraoperative incidents by electronic scanning of computerized anesthesia records: comparison with voluntary reporting. Anesthesiology 1996;85:977–87.[Web of Science][Medline]
6. Benson M, Junger A, Fuchs C, et al. Using an anesthesia information management system to prove a deficit in voluntary reporting of adverse events in a quality assurance program. J Clin Monit Comput 2000;16:211–7.[Web of Science][Medline]
7. Lubarsky DA, Glass PSA, Ginsberg B. The successful implementation of pharmaceutical practice guidelines. Anesthesiology 1997;86:1145–60.[Web of Science][Medline]
8. Katz RI, Lagasse RS. Factors influencing the reporting of adverse perioperative outcomes to a quality management program. Anesth Analg 2000;90:344–50.[Abstract/Free Full Text]
9. Vigoda M, Lubarsky D. The medico-legal importance of enhancing timeliness of documentation when using an anesthesia information system and response to automated feedback in an academic practice. Anesth Analg 2006;103:131–6.[Abstract/Free Full Text]
10. Body SC, Fanikos J, DePeiro D, et al. Individualized feedback of volatile agent use reduces fresh gas flow rate, but fails to favorably affect agent choice. Anesthesiology 1999;90:1171–5.[Web of Science][Medline]
11. Van Amringe M <http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&itool=PubMed_Abstract&term=%22Van+Amringe+M%22%5BAuthor%5D>. The JCAHO guidelines for pay for performance. Manag Care. <javascript:AL_get(this, ‘jour', ‘Manag Care.');> 2005 Dec;14(12Suppl):11–2.
12. MEDICARE "PAY FOR PERFORMANCE (P4P)" INITIATIVES <http://www.cms.hhs.gov/apps/media/press/release.asp?Counter=1343> http://www.cms.hhs.gov/apps/media/press/release.asp?Counter=1343 Accessed May 16, 2006.
13. Fletcher A. Health quality idea gains favor: Colorado group teaming up on effort to pay best docs more. Denver: Denver Business J, December 20, 2004.
14. Win-win program for docs patients: better care at lower cost—Blue Cross' goal. Detroit: Detroit Free Press, April 25, 2005.
15. Bierstein K. Pay for performance: the hot health policy topic of 2005. ASA Newsletter 2005;69:1.
16. Peterson GN, Domino KB, Caplan RA, et al. Management of the difficult airway: a closed claims analysis. Anesthesiology 2005;103:33–9.[Web of Science][Medline]
17. Lee LA, Posner KL, Domino KB, et al. Injuries associated with regional anesthesia in the 1980s and 1990s: a closed claims analysis. Anesthesiology 2004;101:143–52.[Web of Science][Medline]
18. Domino KB, Posner KL, Caplan RA, Cheney FW. Airway injury during anesthesia: a closed claims analysis. Anesthesiology 1999;91:1703.[Web of Science][Medline]
19. Cooper JB. Is voluntary reporting of critical events effective for quality assurance? Anesthesiology 1996;85:961–4.[Web of Science][Medline]

This article has been cited by other articles:

				S. Muravchick, J. E. Caldwell, R. H. Epstein, M. Galati, W. J. Levy, M. O'Reilly, J. S. Plagenhoef, M. Rehman, D. L. Reich, and M. M. Vigoda Anesthesia Information Management System Implementation: A Practical Guide Anesth. Analg., November 1, 2008; 107(5): 1598 - 1608. [Abstract] [Full Text] [PDF]

				W. S. Sandberg, E. H. Sandberg, A. R. Seim, S. Anupama, J. M. Ehrenfeld, S. F. Spring, and J. L. Walsh Real-Time Checking of Electronic Anesthesia Records for Documentation Errors and Automatically Text Messaging Clinicians Improves Quality of Documentation Anesth. Analg., January 1, 2008; 106(1): 192 - 201. [Abstract] [Full Text] [PDF]

				F. Dexter, A. Willemsen-Dunlap, and J. D. Lee Operating Room Managerial Decision-Making on the Day of Surgery With and Without Computer Recommendations and Status Displays Anesth. Analg., August 1, 2007; 105(2): 419 - 429. [Abstract] [Full Text] [PDF]

				W. D. Driscoll, M. A. Columbia, and R. A. Peterfreund An Observational Study of Anesthesia Record Completeness Using an Anesthesia Information Management System Anesth. Analg., June 1, 2007; 104(6): 1454 - 1461. [Abstract] [Full Text] [PDF]

This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Email this article to a colleague Similar articles in this journal Similar articles in Web of Science Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Request Permissions Citing Articles Citing Articles via HighWire Citing Articles via Web of Science (3) Citing Articles via Google Scholar Google Scholar Articles by Vigoda, M. M. Articles by Lubarsky, D. A. Search for Related Content PubMed PubMed Citation Articles by Vigoda, M. M. Articles by Lubarsky, D. A. Related Collections Economics and Health Care Research Complications Technology