| JOURNAL HOME | CME HOME | THIS MONTH | PAST ISSUES | ETOC | COLLECTIONS |
| AUTHORS | REVIEWERS | EDITORIAL BOARD | FEEDBACK | RSS | HELP |
| ||||||||||||||
| ||||||||||||||
|
|
|
|
||||||||||||
|
||||||||||||||
Department of Anesthesiology, Hospital Governador Celso Ramos, Florianópolis-SC, Brazil
Address correspondence and reprint requests to Rua Luiz Delfino 111/902, 88015360 Florianópolis-SC, Brazil. Address e-mail to grof{at}th.com.br
 |
Abstract |
|---|
 Top Abstract IntroductionMethodsResultsDiscussionAppendixReferences |
|---|
IMPLICATIONS: A resident assessment score at outpatient preanesthesia consultation was constructed, validated, and applied. Quality assurance statistics were used for its interpretation. The instrument proved useful in scoring resident performance and in identifying different patterns of performance.
|
Introduction |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionAppendixReferences |
|---|
PAC is defined as the process of clinical assessment that precedes the delivery of anesthesia care for surgery and for nonsurgical procedures. Information gathered during the PAC process may be used to educate the patient, organize resources for perioperative care, and formulate plans for intraoperative care, postoperative recovery, and perioperative pain management. Review of patient’s medical records, interview, questionnaires, physical examination, ordering of appropriate preoperative tests, and consulting other specialists are documented strategies used by anesthesiologists to gather relevant information (3). Attesting competence at PAC requires the objective documentation of accomplishment of the above-mentioned behaviors and tasks during residents’ training period.
Structured patient interviews have been successfully used to obtain data on patients’ medical conditions (4). However, this history-taking approach contrasts with the history-building approach emphasized at our institution. By using nonstructured (narrative) interviews, the history-building approach stimulates both patient and physician to construct a complete and contextualized medical history, including both biomedical and patient-defined points of view (5). Moreover, a narrative approach facilitates the disclosure of sensitive issues, such as fears, level of anxiety, and the need for specific knowledge about the anesthetic procedure and its complications, which vary among patients (6,7). Nonstructured interviews require residents to have a reasonable knowledge base before all relevant issues can be determined, which makes the demonstration of learning patterns theoretically possible (5,8).
Assessing resident competence in collecting relevant data and providing patient education in a history-building PAC requires the definition of criterion items. The same applies to other related skills, including patient-physician relationship, anesthesia-focused physical examination, preoperative test ordering, and assurance of quality medical records. A dependable assessment instrument should contain criterion items on all relevant domains (content and face validities), enough internal consistency (reliability), ease of administration, high probability of inter-rater agreement, and ability to differentiate various levels of competence (construct validity) (1,9–11). It has been demonstrated that structured forms, in which items are prompted to evaluators, increase the accuracy of faculty assessment of resident competence at medical consultations (12).
Statistical methods for making inferences about the collected data should preferentially involve simple calculations, should allow for visual interpretation of individual subject performance, should have memory, i.e., each data point should be statistically related to their antecessors, and should be suitable for the analysis of sequential samples. The exponentially weighted moving average (EWMA) chart, a quality assurance statistical tool, fulfills the above-mentioned criteria (13). EWMA charts are constructed from sequential data samples, and control limits are drawn on them. Within upper (UCL) and lower (LCL) control limits is the in-control zone. Points lying outside the in-control zone reflect statistically significant deviation from the sample mean. Points lying below the LCL represent insufficient performance, points lying between control limits reflect stable performance, and points lying above the UCL reflect improved performance. Upward shifts of EWMA curves crossing the adjacent control limit are interpreted as significant improvement. Downwards trends, crossing the adjacent control limit, indicate significant deterioration of performance (13–15).
The purpose of this study was to develop an instrument for scoring performance of residents during PAC suitable for collecting data that could be used for sequential and quantitative assessment of competence.
|
Methods |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionAppendixReferences |
|---|
The construction of items phase aimed at constructing items that could guarantee content and face validities to the instrument. Based on the definition, minimum components, and goals of PAC (3,6), we defined educational objectives to obtain competence at outpatient PAC, which were divided into 6 areas related to (a) establishing physician-patient relationship, (b) obtaining anesthesia-focused data from medical history, (c) performing an anesthesia-focused physical examination, (d) selectively ordering preoperative tests, (e) educating the patient about the perianesthetic experience, and (f) writing quality information on patient’s medical records.
Items that described each area were generated by a focus group of five board-certified anesthesiologists at our institution by answering the following questions: (a) what should be done to demonstrate good interpersonal and communication skills during an outpatient PAC? (b) What information should be routinely sought in constructing the medical history of a patient presenting for an ambulatory or same-day admission surgery? (c) What constitutes an airway examination, a lung examination, and a heart examination? (d) Which topics should be included in patient education? (e) What information should be written on preanesthesia records? (f) What information or physical signs should be sought only in special situations?
Sixty-six items were generated. In building a physician-patient relationship, the resident would be expected to greet the patient, introduce herself to the patient, call the patient by his name, and inform him about the objectives of the PAC. For the medical history, the following data would be obtained: disease motivating the surgery, existence of concurrent diseases, details of any preexisting disease, degree of physical limitation imposed by concurrent diseases, medications in use, presence of allergies, history of patient’s previous surgeries and anesthetics, anesthetic complications of patient’s relatives, smoking and drinking habits, illicit drug abuse, possibility of pregnancy in female patients, risk factors for ischemic heart disease, previous hepatitis, and intensity of preoperative anxiety. The physical examination would include: vital signs (arterial blood pressure, heart rate, respiratory frequency, and temperature), airway evaluation (Mallampati score, measurement of thyro-mental distance, sternum-mental distance, and mouth opening), cardiovascular examination (precordial inspection, percussion, palpation and auscultation, peripheral pulse palpation, routine carotid auscultation or, in case of patient’s medical history, suggested cerebrovascular disease, and routine Allen test or programmed perioperative arterial cannulation), lung examination (thoracic inspection, percussion, palpation, and auscultation), routine abdominal examination (inspection, palpation, percussion, and auscultation), and abdominal examination only if medical history suggested abdominal mass or the possibility of visceromegaly. Complementary examinations would include ordering of complementary tests guided by medical history and physical examination, review of all complementary tests ordered by other physicians, and review of patient’s medical records. Patient education would include discussion of perianesthetic plans, anesthesia-related risks disclosure, orientation about preoperative fasting, and about continuance, withdrawal, or replacement of current medications. Information about reasons for eventual request of specialist consultations, eventual need for surgery postponement, and preanesthesia medication should be provided to the patient. A preanesthesia orientation form should be read to and signed by the patient. Anesthetic records would contain weight and height, all relevant data obtained in medical history and physical examination, patient’s current and past medications, pertinent complementary tests, and ASA physical status.
Items were submitted to 11 board-certified anesthesiologists with relevant scientific engagement in the study of PAC, from different parts of the country, who accepted to make up a panel of experts engaged in a multiple-round, consensus-searching process (Delphi technique) (15). Contacts were made via electronic mail. Panelists were requested to rate each item, considering the requirement to include it into routine PAC of ambulatory ASA I–III patients (excluding obstetric and pediatric cases), as a measure of proficient performance of residents. Ratings were based on a 5-point Likert scale (1 = absolutely unnecessary, 2 = unnecessary in most consultations, 3 = neutral, 4 = required in most consultations, 5 = absolutely required). The median and 25th–75th percentiles of panel scores on each item were calculated. Consensus on each item was defined as 25th–75th percentiles equal to or <1. Items not fulfilling this criterion were resubmitted to the panel with the respective medians and interquartile limits for reassessment. The process was repeated until consensus was obtained. Panelists could suggest the inclusion, exclusion, or modification of items. Changes were incorporated if agreed by at least 80% of panelists.
Items with median scores of 4 (applicable to most PAC) or 5 (applicable to all PAC) were incorporated into the pilot checklist as routine or special situation items. Routine items were defined as those applicable to any patient. They were rated positively with the median value attributed to them by the expert panel. Special situation items were those that should be investigated if the patient reported some preexisting disease (e.g., degree of physical limitation imposed by the disease), fulfilled some preestablished criterion (e.g., neck auscultation in elderly patients), or if the item related to some educational information expected to be provided to the patient (e.g., motives for specialist consultation). Special situation items were incorporated in the checklist as negative statements and were rated negatively with the respective median (10).
In the second phase of the study, six residents (two 3-member groups) independently observed and rated, using the pilot checklist, 53 PAC performed by four board-certified anesthesiologists. The resulting 159 scores were used to assess the performance of the checklist. Internal consistency (reliability) was assessed by Cronbach’s 
coefficients within each domain and between each domain and the total score. Construct validity was assessed by stepwise multiple-linear regression. Inter-item correlations evaluated discriminant validity. Items were excluded if contributing to decrease the coefficient of the respective domain (10,11,16).
Given the large percentages of agreement on each item, an alternative approach to Cohen’s 
(17), designed to assess the probability that if one rater checked a given item, a second rater would do the same (second rater’s agreement probability), was applied to each item according to the method described elsewhere (18,19). An electronic form of the final checklist was constructed using the Delphi 5 language.
The third phase of the study consisted of the application of the instrument to assess the performance of seven novice residents at their initial set of consecutive PAC. Three-hundred-seventeen ambulatory PAC of patients aged between 15 and 85 yr old, of both sexes, ASA physical status I–III, whose surgeries were scheduled on outpatient or same-day admission basis, within 2 wk from the PAC, were observed and rated from February 21 through April 25, 2003. Obstetric patients were not included. Residents were coded A through G. Numbers of consecutive PAC performed by each resident during the study period were 50, 38, 39, 56, 26, 69, and 39, for residents A, B, C, D, E, F, and G, respectively. Each resident’s set of PAC was indexed according to the respective date and start time. The same supervising instructor rated all consultations. For ethical reasons, he collected missing relevant information and provided relevant patient education at the end of the consultation. Theoretical background on PAC was provided to residents before the start of the study period as one lecture, two problem-based learning sessions, and one set of four to six observations of PAC performed by instructors.
Data were imported into a Microsoft® Excel 2000 electronic spreadsheet (Microsoft, Bellvue, WA). EWMA charts for domain scores and for the total score were constructed, as described in the Appendix (14,15). For improving clarity of charts, scores were averaged across residents so that a single curve represented the performance of the sample. The weighting constant 
was set to 0.2 and starting points (Z(0)) were the mean of the initial 10 mean values. Average sample sizes to detect shifts equal to 4 (the minimum individual item score of the checklist) with (type I error) = ß (type II error) = 0.05 were estimated as 2–12 observations.
|
Results |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionAppendixReferences |
|---|
Multiple-linear regression on data of the validation sample resulted in multiple r = 0.99, multiple R2 = 0.98, and adjusted R2 = 0.98 (P = 0). The complementary examination domain, which contained two items (review of patient’s medical records and selective ordering of preoperative tests), was rejected during stepwise selection. Table 1 shows other relevant statistical results.
|
|
|
|
|
|
|
Discussion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionAppendixReferences |
|---|
The Accreditation Council for Graduate Medical Education has defined six areas of competence to be developed during medical education: patient care, medical knowledge, professionalism, systems-based practice, practice-based learning and improvement, and interpersonal and communication skills. The Accreditation Council for Graduate Medical Education recommends training programs to develop learning outcomes and objectives that reflect the general competences, which should be addressed by specific assessment methods to demonstrate valid and reliable evidence of residents’ achievements at competence-based educational objectives (1).
Given the multiple dimensions of competence, multiple assessment instruments and formats have been recommended, depending on the dimension being evaluated.
Checklists consist of essential or desired specific behaviors, activities, or steps that characterize a more complex competence. Consensus of several experts is required, with agreement on essential behaviors or actions as criteria for evaluating performance (1). Checklists’ usefulness has been documented for evaluating processes of performance, such as history taking and physical examination (12), surgical skills (21), and performance of continuous epidural anesthesia (22,23). Scoring checklists increases their reliability (1,24,25). Item scores in our checklist represent the opinion of experts about the need of being present and should not be taken as indicators of their relative importance in the process of PAC.
Evaluation of processes should start early in training to preclude the consolidation of inaccurate or incomplete care processes that, once automatized through repeated application, could jeopardize patient safety. Moreover, assessment of clinical skills should be a continuous process. During the learning phase, it is useful for providing formative assessment. After this, assessment is important to determine retention of knowledge and skills and to detect eventually deteriorating performance. Statistical control charts supply the best means of recording routine data and form a sound basis for performance review and improvement programs (15). In this study, EWMA charts contributed to the interpretability of scores, provided statistical significance to changing levels of performance, and reliably identified trends after small numbers of observations, contributing to the functional work of our checklist in terms of the educational consequences of its use as a formative and summative assessment tool.
The scoring checklist possesses high construct validity, as suggested by the high multiple R2 and by its ability to distinguish between different levels of performance, high discriminant validity, as inferred by the low inter-item correlation coefficients, and high probability of inter-raters agreement, as assessed by the high probabilities of second-rater agreement. Content and face validities were sought by assuring that items represented expert opinion about tasks and behaviors that describe competent performance at PAC. The overall internal consistency (reliability) of our instrument might be considered moderate, as indicated by Cronbach’s coefficient of 0.61. However, moderate coefficients are adequate when multiple measures are performed (1).
Our instrument lacks items measuring performance at selectively ordering preoperative tests and reviewing patients’ medical records. Because patients’ medical records are usually not available at the PAC facility of our institution and surgeons used to order preoperative tests in advance of the PAC, these items could not be rated in our samples. Also, our instrument was designed to measure performance at processes involved in PAC. The potential impact of its application on patients’ outcomes has not been addressed in this study.
Not unexpectedly, given their inexperience and the short interval used to collect data, residents did not attain maximum scores in most domains, although statistically significant improvement could be detected. In detecting areas of weak performance, the instrument revealed residents who did not inform their patients about the objectives of the PAC or even called them by their names, who were unable to obtain relevant information about preoperative anxiety, drinking habits, and drug abuse, and who were deficient in providing adequate patient education.
The above-mentioned features make the Preanesthesia Consultation Scoring Checklist a potentially useful instrument for both formative and summative assessment of residents during their training in processes involved in outpatient PAC.
|
Appendix |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionAppendixReferences |
|---|
2. Estimate sigma (
) of each sample, the process SD, using W/d(n) or MW/d(n). Constant d depends on the number of observations (n): (d(1)= d(2)= 1.128, d(3)= 1.693, d(4)= 2.059, d(5)= 2.326, d(6)= 2.534, d(7)= 2.704, d(8)= 2.847, d(9)= 2.970, d(10)= 3.078).
3. Set a value to lambda () that weights past and current information (usually between 0.1 and 0.3).
4. Determine the points on the EWMA chart denoted by Zt, where t is the number of the current sample: Zt = (*xt)+ ((1–)*Z(t–1)). The initial value Z(0) is user-defined and may be either a target value or the average of a subgroup of observations.
5. Calculate upper (UCL) and lower (LCL) control limits:
|
|
|
|
Calculation of Average Sample Sizes
1. Establish acceptable (ma) and unacceptable (mr) scores. Set values for (type I error) and ß (type II error).
2. Average sample sizes for sequences with means around ma, mr, and (ma + mr)/2 are calculated, respectively, as:
|
|
|
|
|
|
|
References |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionAppendixReferences |
|---|
This article has been cited by other articles:
|
|
 |
|
  G. R. de Oliveira Filho, A. J. Dal Mago, J. H. S. Garcia, and R. Goldschmidt An Instrument Designed for Faculty Supervision Evaluation by Anesthesia Residents and Its Psychometric Properties Anesth. Analg., October 1, 2008; 107(4): 1316 - 1322. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
