This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Email this article to a colleague Similar articles in this journal Similar articles in Web of Science Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Request Permissions Citing Articles Citing Articles via HighWire Citing Articles via Web of Science (3) Citing Articles via Google Scholar Google Scholar Articles by Taekman, J. M. Articles by Stafford-Smith, M. Search for Related Content PubMed PubMed Citation Articles by Taekman, J. M. Articles by Stafford-Smith, M. Related Collections Economics and Health Care Research

---

ECONOMICS, EDUCATION, AND HEALTH SYSTEMS RESEARCH

Preliminary Report on the Use of High-Fidelity Simulation in the Training of Study Coordinators Conducting a Clinical Research Protocol

Jeffrey M. Taekman, MD^*, Gene Hobbs, CHT^*, Linda Barber, MSN, Barbara G. Phillips-Bute, PhD^*, Melanie C. Wright, PhD^*, Mark F. Newman, MD, and Mark Stafford-Smith, FRCPC

*Department of Anesthesiology and the Human Simulation and Patient Safety Center of Duke University; Global Perioperative Research Organization; and Duke Clinical Research Institute, Durham, North Carolina

Address correspondence and reprint requests to Jeffrey M. Taekman, MD, Department of Anesthesia, Box 3094, Duke University Medical Center, Durham, NC 27710. Address e-mail to jeffrey.taekman{at}duke.edu

	Abstract

Top Abstract Introduction Methods Results Discussion References

 
Training of health care research personnel is a critical component of quality assurance in clinical trials. Interactivity (such as simulation) is desirable compared with traditional methods of teaching. We hypothesized that the addition of an interactive simulation exercise to standard training methods would increase the confidence of study coordinators. A simulation exercise was developed to replicate a complex clinical trial. Eighteen study coordinators completed pre- and postexercise confidence questionnaires. Questions were targeted at key trial components using a 0–10 scale (not confident to confident) and were categorized using Bloom’s Taxonomy. The primary analysis compared overall mean pre- and postexercise responses. Secondary analyses assessed affective, psychomotor, and cognitive confidence. Significance was at P < 0.05. A significant increase in overall confidence (8.64 versus 5.77; P < 0.0001) was reproduced in the subcategory analyses (affective, 8.24 versus 4.89; P < 0.0001; cognitive, 8.75 versus 6.42; P = 0.0003; psychomotor, 8.63 versus 5.26; P < 0.0001). A high level of internal consistency and reliability in question responses within domains was observed, validating the questionnaire tool. In this preliminary report, we confirmed that addition of a simulation exercise to the training of study coordinators resulted in increased confidence. Simulation exercises should be considered when training study coordinators for clinical research trials.

IMPLICATIONS: A complex clinical research protocol was replicated in the Duke University Human Simulation and Patient Safety Center. Clinical research coordinators trained in simulation showed significantly greater confidence in their ability to perform the research protocol. We conclude that simulation training should be considered as a training method of research coordinators.

	Introduction

Top Abstract Introduction Methods Results Discussion References

 
Clinical trials play a major role in the advancement of medical care. More than $6 billion is spent each year on clinical research (1). Competent research personnel are integral to the safe and effective performance of a clinical trial. The acquisition of required knowledge, skills, and attitudes to achieve competency is called the "learning curve." The duration of this process is highly variable, being influenced by many factors including task type, complexity, and the method of education (2–5). Bloom and Krathwohl (6) categorized learning into three domains (affective, cognitive, and psychomotor), each with a hierarchy from the simplest to the most complex. Similarly, Miller (7) described competence as a progression through a series of interdependent steps. Competence is the ability to perform to a standard; confidence is the belief in one’s ability to perform. Confidence and competence have been shown to positively correlate in the performance of complex tasks (8–10). Both are required to adequately perform complex behaviors.

Little attention has been paid to the study of research personnel education, learning, competence, and performance. Current research coordinator training includes methods such as self-study, investigator meetings, and lectures. Modern theories of adult learning highlight the value of interactivity, immediate feedback, integration of experience, and placing information in context (11). Interactivity, as achieved in simulation, has been shown to be a superior method of teaching desired behaviors to medical professionals (12,13).

Therefore, using confidence as a surrogate for competence, we tested the hypothesis that the addition of an interactive simulation exercise to the education of research coordinators training to achieve mastery of a complex perioperative trial protocol would increase their confidence compared with traditional methods alone.

	Methods

Top Abstract Introduction Methods Results Discussion References

 
The study subjects included all research coordinators participating in a large multicenter pharmaceutical trial. Coordinator training consisted of self-study of the research protocol and attendance at an investigator’s meeting. The full-day investigator’s meeting featured lectures and question/answer sessions. The lectures addressed a high-level overview of the research topic as well as more specific characteristics of the protocol. Few interactive learning exercises were incorporated. Only coordinators who attended the investigator’s meeting before the simulation experience were included in our analysis. After IRB approval, groups of 7 to 11 coordinators attended 1 of 5 daylong simulation experiences at the Duke University Human Simulation and Patient Safety Center (http://simcenter.duhs.duke.edu/). Demographic data were collected.

A nine-question self-assessment (Appendix A) was designed to probe the 3 areas of learning as defined by Bloom and Krathwohl (6): psychomotor, cognitive, and affective. The questionnaire consisted of three psychomotor questions, two cognitive questions, and three affective questions targeted at key components of the trial. The questionnaire used a scale from 0 (not confident) to 10 (confident). In addition, there was a single question item that addressed participants’ overall impression of simulation as a learning method. Each study coordinator completed the questionnaire before and after their simulation experience.

Table 2 demonstrates representative behaviors of this complex clinical protocol. Psychomotor tasks were those involving equipment setup, study drug administrations, and data capture. Cognitive tasks involved performance of calculations, timing and administration of drugs, and correct responses to alterations in patient condition. Affective tasks were comfort with the study environment, equipment, and the ability to communicate effectively with operating room personnel.

Each study coordinator received a packet of mock study drug syringes and data collection sheets for use in the simulation exercise. Each coordinator was expected to treat the simulated patient as their first enrolled study patient. The protocol required frequent interaction and collaboration with the anesthesiologist. Representative activities include: (a) requests for procedural times, (b) requests for blood samples, (c) request for study drug administration, (d) request for dosages and times of standard drugs administered, and (e) notification of alterations in clinical status. The clinical trial principal investigator (MSS) was embedded in the exercise giving personalized, constructive, real-time feedback on the coordinator’s performance. Each subject, in turn, coordinated the simulated clinical trial in the simulation center. Study coordinators were expected to communicate with the anesthesiologist at appropriate times in the simulated trial, direct the timing of the study drugs, and collect data. The study coordinator was expected to recognize and vary the protocol in response to clinical cues.

The experience was observed by all other coordinators via a live video feed in a distant room. A facilitator was present in the observation room to answer questions during the simulation exercise. The experience was nonevaluative.

The audio and video of the simulation exercise were captured to mini digital videotape and to a computerized digital video array. After the 15- to 30-min exercise, the principle investigator, study subject, observing study coordinators, and coordinator facilitator gathered to debrief on the simulation. The debrief covered key positive and negative behaviors observed by the other study coordinator and principal investigator. The process was then repeated. Each study coordinator worked through the protocol in the simulation center. The simulation duration of 15–30 minutes was developed as a practical compromise between concerns of reduced educational value from a time-compressed exercise and the diminishing benefit monitors seemed to derive during our pilot simulation exercises from observing more than four to five simulations over 4 h.

The primary analysis was the mean change in questionnaire responses from presimulation to postsimulation (combining affective, cognitive, and psychomotor domains). Secondary analyses examined the four subcategories separately. Pre- and postscores were compared using paired t-tests. The study-wide significance level was set at 0.05. Bonferroni adjustment for 4 multiple comparisons required each individual test to have a P value of 0.01 or less to be considered significant.

Although questionnaire items were chosen to adhere to the three domains of Bloom’s Taxonomy, we performed a confirmatory factor analysis to verify that our items conformed to this structure (14).

	Results

Top Abstract Introduction Methods Results Discussion References

 
Of the 48 subjects, only 18 attended the investigator meeting before the simulation exercise. These 18 were included in the primary and secondary analyses. Demographic variables are presented in Table 1. A majority of study subjects had previous clinical experience, including exposure to operating room procedures. However, few had been previously exposed to simulation as a training tool.

View larger version (34K): [in this window] [in a new window]   Figure 1. Mean confidence scores before and after simulation exercise. Eighteen subjects were included. Scores ranged from 0 (not confident) to 10 (confident). Bars represent overall mean ± SE. Combined data are presented as well as the breakdown into the three domains of Bloom’s Taxonomy. The simulation exercise resulted in a significant gain in confidence in the affective, cognitive, and psychomotor domains.

	Discussion

Top Abstract Introduction Methods Results Discussion References

Today’s clinical trials require complex behaviors and an extensive comprehension of the study protocol. Mastery of a complex protocol requires a significant learning curve. Training of research coordinators fails to take into account one of the basic tenants of modern learning theory, interactivity. Current coordinator training consists of focused self-study, lectures, and small-group discussions. We believe a great deal of learning currently takes place at the patient’s bedside with enrolled patients. Simulation is a training method used extensively in the military, aviation, and nuclear industries to shorten the learning curve and improve performance. Many education and technology experts consider simulation the most powerful modality available for teaching complex integrative tasks.

Many studies have examined the role of simulation in health care professional training (15–20). Simulation allows the learning of complex behaviors without the risk to patients. An emerging body of work demonstrates improved real-world performance for those trained in simulation (21). Simulators are also used to examine the cause of medical mishaps and limitations in equipment and human performance (20,22,23). Simulators should similarly be used to discover, analyze, and resolve issues with clinical research protocols.

There were several limitations to our study. Study logistics limited our ability to designate a control group or randomize participants in this study. Because we lacked a control group, we could not determine if the effect seen was because of the simulation exercise or the longer nonspecific exposure to the trial protocol. Our future studies will attempt to include a control group and subject randomization.

There were several notable limitations to our survey instrument. We used confidence as a surrogate measure of competence. Ultimately, we are interested in assessing coordinator competence in performing the complex tasks of a clinical trial. Although there is literature demonstrating a correlation between confidence and competence, our future studies will use objective measures of coordinator performance and data integrity to assess competence directly.

Miller (7) described the progression of competence through a series of interdependent steps represented as a triangle. A learner may be able to achieve the "knows" or "knows how" levels of competence through self-study. But only through performance of the protocol could they demonstrate the higher levels of competence ("shows how" and "does").

Cognitive gains using other less labor-intensive forms of simulation (24) have been described. However, we suspect the greatest psychomotor and affective gains will be made by training in high-fidelity simulated or true clinical environments. Training in simulated environments offers the additional benefit of not placing patients (or study data) at risk. Regardless of the cause, the ability to affect performance argues for a hands-on simulation experience.

There are several potential problems with the simulated environment. Despite the sophistication of today’s simulation centers, learners realize they are working in an artificial environment. Adding to the artificiality is the need to compress time (compressing the key components of a 3 hour procedure into a 15–20 minute simulation) to accommodate all participants. Further studies need to address whether time compression during training has any effect on real-world performance.

In addition, participants tend to be hyper-aware during a simulation exercise. Many expect a problem to occur with every training session. This heightened state of awareness likely leads to greater vigilance and faster response times of the subjects when reacting to alterations in the protocol. Thus, what is seen in the simulation center may or may not reflect real-world behavior. We attempted to control this phenomenon by randomly choosing between normal and abnormal pathways during the simulation.

The application of simulation in medicine has been limited to students, physicians, nurses, and para-health professionals involved in direct patient care. Our study demonstrates that simulation centers may be used to train other personnel in vitally important medically related activities. We conclude that simulation should be considered in the training of study coordinators involved in complex clinical trials. Use of simulation in training study coordinators will expand the potential user-base of simulation centers and will contribute to improved patient safety through safer trials and better data integrity.

In this study, there were several interesting observations that deserve further investigation. We noticed a wide disparity of coordinator performance in the simulated protocol. The cause of this disparate performance was unclear. Potential sources of this observation include differing personality styles, differing or aberrant learning styles, teacher-learner disparities, or differing levels of preparation. Further investigation is warranted to uncover the cause of these disparities. A few coordinator performances raised an important ethical question: Should simulation be used to screen individuals involved in complex (especially potentially life-threatening) trials? This question requires further investigation.

Several individuals had combined confidence scores that worsened after the simulation exercise. The reality check of working through the replicated protocol brought these aberrant scores in line with other subjects. The worsening of confidence after simulation highlights a limitation of using confidence as a surrogate for competence. Although confidence worsened, performance may have remained the same (or even improved). The root cause of overconfidence, the effect of overconfidence on protocol performance, and the effect of simulation in correcting overconfidence are all areas for further investigation.

Another observation worth noting is that several changes were made to the protocol by the principle investigator after observing the study coordinator’s performance in simulation. Observation of the study coordinators brought to light several timing, data collection issues, and physical task impediments that were not obvious during the paper-and-pencil preparation of the protocol. The potential use of high-fidelity simulated environments for complex clinical trial protocol development warrants further investigation.

Clinical trials have a wide range of complexity and inherent danger to the patient. The cost of developing and delivering a simulated protocol training experience is significant. Further studies need to be conducted to determine the type of clinical trials most amenable to simulation.

In summary, the current work is an assessment of the role of simulation in the education of practitioners of clinical research. We believe our preliminary findings identify potential ways to improve the quality of clinical research including patient safety issues relative to other teaching methods; however, it will be important to confirm our thesis. This is particularly significant because many other alternate teaching modalities (e.g., one-on-one teaching, group teaching, videotape review of procedures, and web-based instruction) are significantly less costly. In the next phase of our studies, we hope to show, using objective measures of performance, that simulation training improves the quality of data obtained in a clinical trial compared with a control group that receives an equivalent amount of traditional training. Our ultimate goal is to improve both the safety and efficiency of clinical research trials.

View larger version (50K): [in this window] [in a new window]

View larger version (44K): [in this window] [in a new window]

	Acknowledgments

 
I would like to acknowledge the generous contributions of Edward Halperin, MD, FACP, Barbara Turner, RN, DNSc, FAAN, Mary Champagne, RN, PhD, FAAN, Jonathan Mark, MD, and Russel Kaufman, MD, to the success of the Human Simulation and Patient Safety Center. Thanks is also due to Fiona Clements, MD, for programming the simulation model and for Iain Sanderson, BM, BCh, MAMSc, FRCA, for his contribution to the development of our perioperative record keeping system. Thanks is due to Rob Califf, MD, and support staff of the Duke Clinical Research Institute. Additionally, I would like to thank David Warner, MD, for his critical evaluation of this manuscript and Andrew Wortham for his tireless work on behalf of the Simulation Center.

	Footnotes

 
Presented, in part, at the American Society of Anesthesiologist’s Annual Meeting, San Francisco, CA, October 14, 2003.

	References

Top Abstract Introduction Methods Results Discussion References

 

1. Bodenheimer T. Uneasy alliance: clinical investigators and the pharmaceutical industry. N Engl J Med 2000; 342: 1539–44.[Free Full Text]
2. Marshall JB. Technical proficiency of trainees performing colonoscopy: a learning curve. Gastrointest Endosc 1995; 42: 287–91.[Web of Science][Medline]
3. Parry BR, Williams SM. Competency and the colonoscopist: a learning curve. Aust N Z J Surg 1991; 61: 419–22.[Web of Science][Medline]
4. Bull C, Yates R, Sarkar D, et al. Scientific, ethical, and logistical considerations in introducing a new operation: a retrospective cohort study from paediatric cardiac surgery. Br Med J 2000; 320: 1168–73.[Abstract/Free Full Text]
5. Hasan A, Pozzi M, Hamilton JR. New surgical procedures: can we minimise the learning curve? Br Med J 2000; 320: 171–3.[Free Full Text]
6. Bloom BS, Krathwohl DR. Taxonomy of educational objectives: the classification of educational goals. New York: D. McKay Co. Inc., 1956.
7. Miller GE. The assessment of clinical skills/competence/performance. Acad Med 1990; 65: S63–7.[Web of Science][Medline]
8. Smith LF. The effects of confidence and perception of test-taking skills on performance. North Am J Psych 2002; 4: 37–50.
9. Nickerson RS, McGoldrick CC. Confidence ratings and level of performance on a judgmental task. Percept Mot Skills 1965; 20: 311–6.[Web of Science][Medline]
10. Smith RE, Christensen DS. Psychological skills as predictors of performance and survival in professional baseball. J Sport Exerc Psychol 1996; 17: 399–415.
11. Ritchie D, Hoffman R. Incorporating instructional design principles with the world wide web. In: Khan B, ed. Web based instruction. Englewood Cliffs, NJ: Educational Technology Publications, 1997: 135–8.
12. Davis D, O’Brien MA, Freemantle N, et al. Impact of formal continuing medical education: do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes? JAMA 1999; 282: 867–74.[Abstract/Free Full Text]
13. Davis DA, Thomson MA, Oxman AD, Haynes RB. Evidence for the effectiveness of CME: a review of 50 randomized controlled trials. JAMA 1992; 268: 1111–7.[Abstract/Free Full Text]
14. Thurstone L. The theory of multiple factors. Ann Arbor, MI: Edwards Brothers, Inc., 1934.
15. Morgan PJ, Cleave-Hogg D. A worldwide survey of the use of simulation in anesthesia. Can J Anaesth 2002; 49: 659–62.[Web of Science][Medline]
16. Gaba DM. Simulator training in anesthesia growing rapidly. Anesthesia Patient Safety Foundation Newsletter, 1995.
17. Gaba DM. Improving anesthesiologists’ performance by simulating reality. Anesthesiology 1992; 76: 491–4.[Web of Science][Medline]
18. Marshall RL, Smith JS, Gorman PJ, et al. Use of a human patient simulator in the development of resident trauma management skills. J Trauma: Injury, Infection, and Critical Care 2001; 51: 17–21.
19. Gaba DM, Howard SK, Flanagan B, et al. Assessment of clinical performance during simulated crises using both technical and behavioral ratings. Anesthesiology 1998; 89: 8–18.[Web of Science][Medline]
20. Howard SK, Gaba DM, Fish KJ, et al. Anesthesia crisis resource management training: teaching anesthesiologists to handle critical incidents. Aviat Space Environ Med 1992; 63: 763–70.[Medline]
21. Seymour NE, Gallagher AG, Roman SA, et al. Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg 2002; 236: 458–63;Discussion 63–4.
22. Gaba DM, DeAnda A. A comprehensive anesthesia simulation environment: re-creating the operating room for research and training. Anesthesiology 1988; 69: 387–94.[Web of Science][Medline]
23. Weinger MB, Ancoli-Israel S. Sleep deprivation and clinical performance. J Am Med Assoc 2002; 287: 955–7.[Free Full Text]
24. Nyssen AS, Larbuisson R, Janssens M, et al. A comparison of the training value of two types of anesthesia simulators: computer screen-based and mannequin-based simulators. Anesth Analg 2002; 94: 1560–5.[Abstract/Free Full Text]

This article has been cited by other articles:

				C. L. Lake Simulation in Cardiothoracic and Vascular Anesthesia Education: Tool or Toy? Seminars in Cardiothoracic and Vascular Anesthesia, December 1, 2005; 9(4): 265 - 273. [Abstract] [PDF]

This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Email this article to a colleague Similar articles in this journal Similar articles in Web of Science Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Request Permissions Citing Articles Citing Articles via HighWire Citing Articles via Web of Science (3) Citing Articles via Google Scholar Google Scholar Articles by Taekman, J. M. Articles by Stafford-Smith, M. Search for Related Content PubMed PubMed Citation Articles by Taekman, J. M. Articles by Stafford-Smith, M. Related Collections Economics and Health Care Research