This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Email this article to a colleague Similar articles in this journal Alert me to new issues of the journal Download to citation manager Request Permissions Citing Articles Citing Articles via Google Scholar Google Scholar Articles by de Oliveira Filho, G. R. Articles by Vieira, J. E. Search for Related Content PubMed Articles by de Oliveira Filho, G. R. Articles by Vieira, J. E. Related Collections Education

---

ECONOMICS, EDUCATION, AND POLICY

The Relationship of Learning Environment, Quality of Life, and Study Strategies Measures to Anesthesiology Resident Academic Performance

Getúlio R. de Oliveira Filho, PhD^*, and Joaquim Edson Vieira, PhD

From the *Department of Anesthesiology, Nucleus for Research in Medical Education, Hospital Governador Celso Ramos, Florianópolis, SC, Brazil; and Center for Development of Medical Education, University of São Paulo Medical School, São Paulo, SP, Brazil.

Abstract

BACKGROUND: We designed this study to determine the academic performance of anesthesia residents as related to their differential characteristics on some affective-motivational variables, represented by perceptions about their educational environment, subjective quality of life, and learning and study strategies.

METHODS: The study sample consisted of 63 anesthesia residents who completed the World Health Organization Quality of Life Inventory, the Dundee Ready Educational Environment Measure, the Learning and Study Strategies Inventory, and a progress test on basic sciences on two to four measurement occasions during a 2-year period. A growth curve model was fit to the academic performance. Mantel-Haenszel tests identified independent predictors of academic performance on progress tests.

RESULTS: Mean rating at the first measuring occasion was 41%. There was a statistically significant improvement over time (slope = 7% per 6-m period; P < 0.01). Analysis of the random effects showed significant individual differences in the intercept. The residents’ scores improved at an equivalent rate over the course of the residency. The independent predictors of academic performance were anxiety, motivation, and ability in selecting main ideas.

CONCLUSIONS: Knowledge growth on basic sciences during anesthesia residency is significantly associated to the level of anxiety related to study and achievement, to the motivation for learning and for personal improvement, and to the ability in selecting main ideas from subject matters to which residents are exposed during learning episodes.

According to the Outcome Project of the Accreditation Council for Graduate Medical Education, resident performance is the measure of each program’s educational efficacy. Outcomes refer to the achievement of competency inpatient care, medical knowledge, practice-based learning, interpersonal skills, professionalism, and systems-based practice (1).

A number of factors may affect resident performance, including learning styles and personality characteristics, social and financial influences, preferences for practices, personal health, and responses to job environment. However, a review of the literature revealed that, resident performance can only be related to personality traits, resident burnout, stress/distress at work, and job satisfaction (2).

Among factors related to individual resident’s life, physical, psychological, social, and environmental issues can potentially affect performance (2). These are domains of the subjective quality of life as defined by the World Health Organization as "the perception of the individual of his position in life in the context of culture and value system in which he lives and in relation to his objectives, expectations, patterns, and worries." There are no available data about the relation between resident performance and the subjective quality of life. However, several of its facets, such as the quality of sleep and rest, the opportunity to get new information, personal relationships, atmosphere at home, and the general quality of life, have been negatively affected by violations of resident duty hours and supervision standards (3).

There is no evidence that resident performance is influenced by the learning environment (2). However, one study showed direct correlation between undergraduates’ perceptions about the learning milieu and their academic achievement. Accordingly, compared with under-achievers, academic achievers scored significantly higher on perceptions regarding teachers, academic atmosphere, and social self-perceptions (4).

Academic performance, as an outcome of residency (2,5), also depends on the quality of residents’ studying and learning strategies. It has been postulated that the way individuals choose or prefer to learn might be compared with a multilayer structure, in which the outer layer represents study and learning strategies, the middle comprises learning styles, and the inner layer represents the personality characteristics. Learning styles remain relatively constant throughout adult life (6). Personality traits remain constant during residency (7), and are hardly amenable to change because of educational interventions. Learning and study strategies can be learned, trained, and improved by specific educational interventions (8,9).

The hypothesis behind the present study was that distinct patterns of perceptions about the subjective quality of life and the learning environment, associated with differing learning and study strategies, could be associated with distinct patterns of knowledge improvement during residency. Based on this hypothesis, the study was designed to determine the academic performance of anesthesia residents as related to their differential characteristics on some affective-motivational variables, represented by perceptions about their educational environment, subjective quality of life, and learning and study strategies.

METHODS

Three training centers in anesthesiology participated in this study, with the respective Ethics Committees’ approval. The three centers are traditional teaching institutions, accredited by the Brazilian Society of Anesthesiologists and by the National Committee for Medical Residency of the Ministry of Education. All institutions are positioned above the 75th percentile in the rank of training centers, promoted by the Brazilian Society of Anesthesiology.

Thirty-three residents admitted in February 2003 and 30 residents admitted in February 2004 composed the study sample. Residents signed written informed consent. The study period was July 2003 through January 2005. Measurement occasions were July 2003, January 2004, July 2004, and January 2005. The instruments included:

1. The short version of the World Health Organization Quality of Life Inventory (WHOQOL-BREF), an instrument designed to assess general subjective quality of life with 26 questions addressing 24 facets, each representing one aspect of the general quality of life. Facets are grouped into four domains. Domain I (physical) discerns facets addressing pain and discomfort; energy and fatigue; sleep and rest; mobility; daily activities; dependence on medications; and working capacity. Domain II (psychological) comprises facets related to positive feelings; thinking, learning, memory, and concentration; self-esteem; body image and appearance; negative feelings; and spirituality, religion, and personal beliefs. Domain III (social relationships) measures personal relationships; social support; and sexual activity. Domain IV (environment) possesses facets measuring physical safety and security; home environment; financial resources; health and social care availability and quality; opportunities for acquiring new information and skills; participation in and new opportunities for recreation/leisure; physical environment; and means of transportation. WHOQOL does not yield a total score, so that each domain represents a measure of one specific aspect of quality of life, higher scores representing increasingly better perceptions (10).
   
2. Dundee Ready Educational Environment Measure (DREEM), an instrument designed to assess students’ perceptions about the learning environment. It consists of 50 items rated on 5-point Likert scales. DREEM comprises five subscales: perceptions about teaching, about teachers, about the atmosphere of the learning environment, academic self-perceptions, and social self-perceptions. A global 200-point score results from the addition of subscale scores. Scores are transformed in percentages of the respective scale, higher scores representing increasingly better perceptions. DREEM was investigated in a population of residents, and has shown robust psychometric properties (11). In this study, a Portuguese version (12) was adapted to our population of residents by replacing "students" with "residents," and "school" with "institution" in the original instrument.
   
3. Learning and Study Strategies Inventory (LASSI), an assessment tool of students’ awareness about the use of learning and study strategies related to three components of strategic learning: skill, will, and self-regulation. The skill component is assessed by the scales on information processing, selecting main ideas, and test strategies. The will component is assessed by the scales on attitude, motivation, and anxiety. The self-regulation component of strategic learning is assessed by the scales on concentration, time management, self-testing, and study aids. Each scale comprises eight items rated on 5-point Likert scores, so that maximum attainable summative scores are 40 points on each scale. Anxiety scale is inversely rated, so that higher scores correspond to lower levels of anxiety.
   
4. A 180-item Rasch modeled (13) progress test (PT) (14) on basic sciences applied to the anesthetic practice, from which the 80 best-fitting items were chosen as criterion-items to measure progress of cognitive ability. Reliability of item calibrations was 0.94 and of person measures was 0.84. Item calibrations (level of difficulty) varied between –2 and +2 logits, the unit used in Rasch analyses for item and person measures. The center of the item difficulty scale, zero logit, was associated with a 50% probability of correct answer. Extremes of the item difficulty scale were associated with 80% and 20% probability of correct answer, respectively. Item distribution according to the respective difficulty levels obeyed the following criterion: 1/3 of items between –1 logit, 33% between –1 and +1 logit, and 1/3 of item above +1 logit. This item selection approach excluded very easy and very difficult items. Most items (58%) addressed applied pharmacology, followed by applied physiology (23%), anatomy (11%), and physics applied to anesthesiology (9%). This item distribution is in accordance with the basic sciences curriculum established by the Brazilian Society of Anesthesiology (15). Scores on progress tests are presented as percentages of correct responses to the criterion-items instead of logit scores to facilitate interpretation.
   

Both residents and teaching faculty were blinded to test results. Academic performance data from the subset of subjects entering the study as first-year residents were analyzed using growth curve methods to determine the pattern of changes in academic performance over the study period. Individual growth curves were constructed using a Level I (unconditional) hierarchical linear model. Parameters of individual growth curves (intercept and slopes) were estimated using ordinary least squares. Homogeneity of intercepts and slopes of individual curves were tested by comparing the sum of the squared deviations of the maximum likelihood estimates from the pooled estimate to the corresponding cutoff of the ² distribution (16).

Residents were classified as having poor academic performance if they were below the median of each progress test scores for two or more of their evaluations. Otherwise, residents were classified as having good academic performance. The environmental and study strategies tests produced 20 different domain scores for each resident (10 on LASSI, 5 from WHOQOL, and 5 of DREEM). Each of the scores was divided into quartiles, and the outcomes were examined on ROC curves for suggestions of appropriate categories for the statistical analyses. On this basis, scores were grouped into two categories (at or above, or below cutoff points), and ² tests of association were used to identify which of these measures showed evidence of association with academic performance. Between-cell comparisons were performed by z tests for proportions. A value of P < 0.05 was the criterion used for this screening process.

Our units of analysis were individual measures on each test occasion. We decided that a minimum difference of 20% would be worthwhile to detect in comparing proportions between cells of 2 x 2 tables (intra- and interclass comparisons), which resulted in a minimum sample size of 35 measures per class for one-sided comparisons, at = 0.05 and β = 0.2 (Statistica 6.0, Statsoft, Tulsa).

RESULTS

Growth curve analyses were limited to the subset of subjects who entered the study as first-year residents and completed all four evaluations (n = 27). The mean rating at the first measuring occasion (intercept) was 41%. There was a statistically significant improvement over time (slope = 7% per 6-mo period; P < 0.01) (Fig. 1). Analysis of the random effects showed significant individual differences in the intercept [² (26 df) = 340.7, P < 0.01]. Random effects for slopes were not statistically significant [² (26 df) = 26.32, P = 0.45]. Reliability of the model was 0.96. Table 1 compares PT scores and model parameters between residents classified as having good or poor academic performances. Significant between-group differences were found between PT scores and intercepts of the growth curves. No differences were found on the slopes.

View larger version (23K): [in this window] [in a new window]   Figure 1. Scores on progress tests (PT) taken by the subset of subjects who entered the study as first-year residents and completed all four evaluations at 6-mo intervals during the first and second year in residency (n = 27). Plot A shows curves of residents classified as having good academic performance. Plot B shows curves of residents classified as having poor academic performance i.e., they were below the test median score for two or more of their evaluations. Resident code labels were omitted for clarity.

Of the 63 participating residents, 3 (5%) left the program for personal reasons. The study sample comprised 39 male and 24 female residents. The median age was 28 yr. Of the 60 participating residents who completed the study, 22 (37%) were classified as having poor academic performance based on their scores on PTs. Not all residents completed tests on the four measurement occasions, so that 161 (89%) of the 180 expected complete data sets were obtained. Sets of 21 variables (10 on LASSI, 5 from WHOQOL, 5 of DREEM, and 1 from the PT) obtained from each resident completing tests on each measurement occasion were defined as units of analysis.

Of the 20 affective-motivational variables, 3 (from LASSI) met our criterion (P < 0.05) for a univariate predictor variable: anxiety, motivation, and ability in selecting main ideas. Table 2 reveals that significant percentages of good performances on PTs were associated with low scores on anxiety, motivation, and selection of main ideas scales, indicating that, above the cutoff points, these variables are poor predictors of good academic performance. Conversely, the percentages of poor performances that were accompanied by high scores on predictor variables were significantly smaller than the percentage of poor performances accompanied by low scores on predictive variables, which suggests that scores below cutoff points on anxiety, motivation, and selection of main ideas scales are good predictors of poor academic performance.

Univariate predictors were significantly associated with each other (r_{(ANX x MOT)} = 0.35; r_{(ANX x SMI)} = 0.54; r_{(MOT x SMI)} = 0.36; P < 0.01). Based on this and the relatively small sample size, we believed that these data were not appropriate for multivariate parametric analyses. Instead, data are presented as contingency tables that allowed tests of the independence of association among each of the three univariate predictor variables. The Mantel-Haenszel test was used to test for statistical significance. For these analyses, the remaining variable and the academic performance were included as layers in the 2 x 2 contingency tables to examine the independence of association between pairs of variables (Tables 3–5).

DISCUSSION

This study was unable to detect any significant association between academic performance and residents’ perceptions about their subjective quality of life or about the educational environment of their residencies. These findings might suggest that knowledge gain occurred during residency independently on how residents perceived their quality of life or their educational environment. However, caution should be exercised in interpreting these results. Only violations to resident duty hours standards and low quality of supervision have been shown to negatively affect the perceptions of residents about their learning environment and their quality of life (3). Considering that the participating institutions strictly adhere to residency national standards, as suggested by their position above the 75th percentile of the national ranking (heavily based on adherence to such standards), these findings may have resulted from a selection bias of the convenience sample used in the present study. Further studies also including low-positioned institutions, which fail to follow residency rules, might clarify this issue.

Only factors associated with strategic learning were found to independently predict academic performance. Such factors were anxiety, motivation, and ability to properly select main ideas.

The anxiety scale of the LASSI assesses the degree to which learners worry about their academic performance. High levels of such anxiety can divert attention away from completing academic tasks, such as studying or taking a test (9). Academic performance-related anxiety can be an inhibiting factor in learning by impairing performance in cognitive functions such as attention, memory, concept formation, and problem solving (17). It is closely related to arousal, attention, and motivation. Tests and examinations, which involve a decision or judgment, are common precursors of anxiety in educational settings. Academic performance-related anxiety can be reduced by: 1) instructions that minimize stress and prepare individuals; 2) increased use of positive feedback during a task; and 3) reduced opportunities for failure in a task (18).

The motivation scale assesses learners’ diligence, self-discipline, and willingness to exert the effort necessary to successfully complete academic requirements. Learners who score low on this scale need to accept more responsibility for their academic outcomes and learn how to set and use goals to help accomplish specific tasks (9). Motivation is a pivotal concept in most theories of learning. It is closely related to arousal, attention, anxiety, and feedback/reinforcement. Motivation to achieve is a function of the individual’s desire for and expectancy of success, and the incentives provided (19).

Selecting main ideas signify a learner’s ability to select important information to concentrate on during classes or while reading and studying in order to achieve academic goals. Active listening and metacognitive thinking allows students to identify which topics are most important to learn (9).

The measuring instruments of this study consisted of reliable inventories (9–11). Academic achievement was measured on items showing high measuring reliability and displaying balanced proportions of item difficulties (20). The analysis of individual growth curves showed that there was a significant difference between residents at baseline measurements followed by a modest growth in knowledge over the 2-yr study period, with homogeneity of slopes of individual curves, so that the relative ranking of residents did not change over the study period, regarding their knowledge of basic sciences. In a previous study (21), attitudes toward the relevance of basic sciences to clinical anesthesiology were compared between residents and attending anesthesiologists. Residents were less positive than anesthesiologists toward the construct. They were also more favorable to a more superficial approach represented by learning general concepts instead of a deeper approach, which was preferred by attendings. We hypothesize that lack of perceived relevance of learning basic sciences during residency might, at least partially, explain the modest growth of knowledge found in the present study.

Unconditional hierarchical linear models are used to analyze individual growth trajectories in social sciences. Conditional models can be used to study the effects of variables to predict individual differences in individual growth curves, but are sensitive to significant correlations between variables (16). Because of the limited sample size, the boundary between high and low performances was set at the 50th percentile. This limitation left the question unanswered regarding predictors of extremes and intermediate levels of performance. The dependency of predictors on our criterion for classifying performance must be considered when generalizing our results to other levels of academic achievement.

We conclude that knowledge growth on basic sciences occurs at a relatively modest rate during anesthesia residency and is significantly associated with the level of anxiety related to academic performance, the motivation for learning, and the ability in selecting main ideas. Testing residents for these factors and for their individual basic knowledge at the beginning of the training period could help identify residents with high probabilities of academic failure and direct remediation measures.

Footnotes

Accepted for publication February 14, 2007.

Address for correspondence and reprints to Getúlio R. de Oliveira Filho, PhD, Department of Anesthesiology, Nucleus for Research in Medical Education, Hospital Governador Celso Ramos, Florianópolis, SC, Brazil. Address e-mail to grof{at}grof.med.br.

REFERENCES

1. Rose SH, Burkle CM. Accreditation Council for Graduate Medical Education competencies and the American Board of Anesthesiology Clinical Competence Committee: a comparison. Anesth Analg 2006;102:212–16.[Abstract/Free Full Text]
2. Mitchell M, Srinivasan M, West DC, et al. Factors affecting resident performance: development of a theoretical model and a focused literature review. Acad Med 2005;80:376–89.[Web of Science][Medline]
3. de Oliveira Filho GR, Sturm EJ, Sartorato AE. Compliance with common program requirements in Brazil: its effects on resident’s perceptions about quality of life and the educational environment. Acad Med 2005;80:98–102.[Web of Science][Medline]
4. Mayya S, Roff S. Students’ perceptions of educational environment: a comparison of academic achievers and under-achievers at Kasturba Medical College, India. Educ Health (Abingdon) 2004;17:280–91.[Medline]
5. Accreditation Council for Graduate Medical Education (ACGME). Outcome Project: Accreditation Council for Graduate Medical Education (ACGME), 2001.
6. Boyle EA, Duffy T, Dunleavy K. Learning styles and academic outcome: the validity and utility of Vermunt’s Inventory of Learning Styles in a British higher education setting. Br J Educ Psychol 2003;73:267–90.[Web of Science][Medline]
7. Pasnau R, Bayley SJ. Personality changes in the first year of psychiatric residency training. Am J Psychiatry 1971;128:79–84.[Abstract/Free Full Text]
8. Pringle RK, Lee J. The use of learning and study strategies inventory (LASSI) as a predictor for success or failure on part I of the National Board of Chiropractic Examiners Test. J Manipulative Physiol Ther 1998;21:164–6.[Web of Science][Medline]
9. Weinstein CE, Palmer DR. LASSI User’s manual. Clearwater, FL: H&H Publishing Company, 2002.
10. Skevington SM, Lotfy M, O’Connell KA. The World Health Organization’s WHOQOL-BREF quality of life assessment: psychometric properties and results of the international field trial. A report from the WHOQOL group. Qual Life Res 2004;13:299–310.[Web of Science][Medline]
11. de Oliveira Filho GR, Vieira JE, Schonhorst L. Psychometric properties of the Dundee ready educational environment measure (DREEM) applied to medical residents. Med Teach 2005;27:343–7.[Web of Science][Medline]
12. Vieira JE, do Patrocinio Tenorio Nunes M, de Arruda Martins M. Directing student response to early patient contact by questionnaire. Med Educ 2003;37:119–25.[Web of Science][Medline]
13. Downing SM. Item response theory: applications of modern test theory in medical education. Med Educ 2003;37:739–45.[Web of Science][Medline]
14. McHarg J, Bradley P, Chamberlain S, et al. Assessment of progress tests. Med Educ 2005;39:221–7.[Web of Science][Medline]
15. Brazilian Society of Anesthesiology. Curricular content for anesthesiology residents; 2003.
16. Bryk AS, Raudenbush SW. Application of hierarchical linear models to assessing change. Psychol Bull 1987;101:147–58.[Web of Science]
17. Kestenbaum JM, Weiner B. Achievement performance related to achievement motivation and test anxiety. J Consult Clin Psychol 1970;34:343–4.[Web of Science][Medline]
18. Sieber J, O’Neil HF, Tobias S. Anxiety, learning and instruction. Hillsdale, NJ: Erlbaum, 1977.
19. Weiner B. History of motivational research in education. J Educ Psychol 1990;82:616–22.[Web of Science]
20. Marshall J. Assessment during postgraduate training. Acad Med 1993;68:S23–S26.[Web of Science][Medline]
21. Filho GR, Schonhorst L. Attitudes of residents and anesthesiologists toward basic sciences. Anesth Analg 2006;103:137–43.[Abstract/Free Full Text]

This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Email this article to a colleague Similar articles in this journal Alert me to new issues of the journal Download to citation manager Request Permissions Citing Articles Citing Articles via Google Scholar Google Scholar Articles by de Oliveira Filho, G. R. Articles by Vieira, J. E. Search for Related Content PubMed Articles by de Oliveira Filho, G. R. Articles by Vieira, J. E. Related Collections Education