This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Email this article to a colleague Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Request Permissions Citing Articles Citing Articles via Google Scholar Google Scholar Articles by Ramsdell, R. Articles by Houle, T. T. Search for Related Content PubMed PubMed Citation Articles by Ramsdell, R. Articles by Houle, T. T. Related Collections Economics and Health Care Research Education Pediatrics

---

PEDIATRIC ANESTHESIOLOGY

Subspecialty Impact Factors: The Contribution of Pediatric Anesthesia and Pain Articles

Robert Ramsdell, MD^*, Jerrold Lerman, MD, FRCPC, FANZCA^*, Donald Pickhardt, MD^*, Doron Feldman, MD^*, James Foster, MBBS, FRCPC^*, and Timothy T. Houle, PhD

From the *Department of Anesthesiology, Women and Children’s Hospital of Buffalo, SUNY at Buffalo, New York; Department of Anesthesiology, Strong Memorial Hospital, University of Rochester, Rochester, New York; Department of Pediatrics, Women and Children’s Hospital of Buffalo, SUNY at Buffalo, New York; and Department of Anesthesiology, Wake Forest University School of Medicine, Winston-Salem, North Carolina.

Address correspondence and reprint requests to Dr. Jerrold Lerman, Department of Anesthesiology, Women and Children’s Hospital of Buffalo, 219 Bryant St, Buffalo, NY 14222. Address e-mail to jerrold.lerman{at}gmail.com.

Abstract

BACKGROUND: The Science Citation Index "Journal Impact Factor" (JIF) is widely used to assess journal quality and prestige. The JIFs for the specialty anesthesia are reported annually, however, the impact factors (IFs) for subspecialties in those journals have not been reported. Therefore, we compared the IFs of pediatric anesthesia (Ped IFs) and pain (Pain IFs) articles from four anesthesia journals for two epochs.

METHODS: An article-by-article manual search for "source" pediatric anesthesia and pain articles published in 1998, 1999, 2003, and 2004 in Anesthesiology, Anesthesia & Analgesia, British Journal of Anaesthesia, and Canadian Journal of Anesthesia was performed. The citations for each of these articles in each of the years were surveyed in the ISI Web of Science database. Ped IFs and Pain IFs for the 2000 and 2005 epochs were calculated and compared with the JIF from which they were derived and to those of the journal, Pediatric Anesthesia.

RESULTS: Ped IFs for the four journals in 2005 exceeded those in 2000, whereas the Pain IFs were unchanged. For both the Ped IFs and the Pain IFs, there was a significant effect of the journal. The Pain IFs were 70% greater than the Ped IFs.

CONCLUSIONS: Ped IFs were consistently less than the JIFs in which they were published and the Pain IFs, except for the British Journal of Anaesthesia 2005 in the latter case. The numbers of citations of pediatric anesthesia articles were greater in journals with greater IFs. The implications of subspecialty IFs warrant further consideration.

The Journal Impact Factor (JIF), first described by Garfield,1 remains an important, albeit controversial, bibliometric tool for citation analysis. Published annually by the Institute of Scientific Information as part of the Science Citation Index (SCI),^¶ the JIF is widely accepted as the best measure of a scientific publication’s prestige and relevance.2

The JIF is a measure of the frequency with which an average article in a journal has been cited in a given year. It is calculated as the ratio of the number of citations in a given year from "source" articles published in the preceding 2 years to the number of the source articles published in the same 2 years. For example, if 100 source articles published in a journal in 2000 and 2001 were cited 400 times in 2002, the impact factor (IF) for that journal in 2002 would be 4.0.

For a citation to be included in the numerator of the IF ratio, it must appear in, as well as reference, an article published in one of the journals included in the SCI database. The current Institute of Scientific Information-SCI database includes approximately 5900 of the more than 120,000 scientific journals currently published.^¶ In addition, only specific types of articles qualify for inclusion in the denominator of the IF. Accepted source articles include original research, review articles, and case reports. Articles that are excluded from the calculation include editorials, letters, and news stories.3

The JIF influences both journal editorial staff decisions and author submissions.4–6 The concept of the IF is frequently applied to entities other than individual journals. IFs may be integral in the decision-making process when considering staffing and academic promotion for individual authors, research groups and institutions.7 IFs can also influence grant resource allocation.8

The purpose of this study was to examine the relative contribution of the pediatric anesthesia subspecialty articles to the major anesthesia journals. As noted above, IF calculations can and have been used for various purposes to individual authors, institutions, and even countries and geographic regions,9 though Garfield had cautioned against such applications.3 However, little attention has been paid to the contributions of subspecialty articles to JIFs and to the significance of a subspecialty IF to the major journals in a respective medical discipline.

A widely held perception among pediatric anesthesiologists and, in fact, among pediatric subspecialists in most fields, is that pediatric research is under-represented in major journals.10 Thus, this analysis was designed to determine whether there is evidence to support this belief.

METHODS

The JIF for the epochs 2000 and 2005 were chosen for analysis. A manual search of Anesthesiology, British Journal of Anaesthesia (BJA), Anesthesia & Analgesia, and Canadian Journal of Anesthesia for 1998, 1999, 2003, and 2004 for all source articles on pediatric anesthesia and pain was performed. A manual search of the journal Pediatric Anesthesia for the same years for all articles on pediatric anesthesia was performed as a reference journal. Source articles included primary scientific publications, case reports, review articles, and clinical trials. Editorials, letters, news articles, and commentary were excluded. An article was deemed to be a pediatric anesthesia article for the purposes of this investigation if the subject matter directly pertained to the care or management of children undergoing anesthesia and the study population was <18 years old. An article was deemed to be a "pain" article if it related to perioperative pain. Articles that pertained to intraoperative pain only (including intraoperative regional blocks) were not included. Articles that involved both pediatric anesthesia and pain were included in both categories. Each journal was then surveyed through the Web of Science to verify that there were no additional articles on pediatric anesthesia and pain that were missed during the manual search of the journals. Once that was completed, each article in turn was searched in the Web of Science for citations in the corresponding year, either 2000 or 2005, using the first author’s name and the journal year and name. The article’s identity was then verified using the article’s title and the corresponding journal and page number. If the search using the first author’s name failed to produce the article in question, the search was repeated using subsequent authors’ names to confirm the existence of the article and the number of citations associated with it.

The subspecialty IF for each of the journals for each epoch was then calculated as follows:

The citation counts for each of the pediatric anesthesia and pain articles were summarized using descriptive statistics for each journal by each epoch. Because the distributions of these citation counts greatly deviated from a normal distribution (Fig. 1), the mean and median, as well as the standard deviation and range are presented. Although formal IFs are reported as single values that can be compared without statistical inference (i.e., an IF of 2.5 is considered to be greater than 2.2 regardless of the variability of citations within the journal), we examined the citation counts as random variables. To examine the pediatric anesthesia and pain citation counts as a function of journal and time, a generalized linear model was fit using a negative binomial distribution and log link function. This type of model has been widely used to model count data when there are more than the expected number of "0" counts than would be expected given a Poisson distribution. For this model, both journal and time were specified as factors, allowing evaluation of the main effects as well as the journal x time interaction. Pairwise post hoc testing for mean group differences was conducted using a Bonferroni adjustment. All parameter estimates are presented with unadjusted Wald 95% confidence intervals. For graphical purposes, the subspecialty IFs and not the means of the model-based citation counts are presented because they more closely represent the traditional IF. Lastly, to examine the relationship between subspecialty IF and JIF, IFs were displayed in a scatter plot for illustration purposes. All analyses were performed using SPSS 15.0 (SPSS, Chicago, IL). Statistical significance was interpreted as P < 0.05.

View larger version (17K): [in this window] [in a new window]   Figure 1. Numbers of articles with up to 12 citations from the five anesthesia journals in each of the two epochs, 2000 and 2005.

RESULTS

Pediatric Anesthesia Articles
The citations counts for each of the 902 identified pediatric anesthesia articles are presented for each journal and each epoch (n = 399 in 2000 epoch; n = 503 in 2005 epoch) in Table 1. The number of published articles ranged widely among the journals with the BJA publishing 25 articles and the journal Pediatric Anesthesia publishing 185 articles during the 2000 epoch. The pediatric (Ped) IFs also ranged widely from 0.4 to 2.1 in 2000 and from 0.8 to 2.2 in 2005. The overall Ped IF increased 17% between 2000 and 2005 (Table 1).

Examination of the pediatric anesthesia subspecialty articles by journal and epoch revealed that the journals differed in their Ped IF but that the Ped IFs increased uniformly over time (Fig. 2a). A main effect for the journal was observed (P < 0.0001; Fig. 2a) such that articles from Anesthesiology (Ped IF = 2.2, 95% CI: 1.65, 2.68), Anesthesia & Analgesia (Ped IF = 1.75, 95% CI: 1.44, 2.06), and the BJA (Ped IF = 1.89, 95% CI: 1.41, 2.36) all had similar mean citation counts; however, articles from the Canadian Journal of Anesthesia (Ped IF = 0.54, 95% CI: 0.31, 0.77) had fewer citations than all of the other journals, and articles from the journal Pediatric Anesthesia (Ped IF = 1.18, 95% CI: 1.03, 1.33) had fewer citations than Anesthesiology and Anesthesia & Analgesia. Further, a main effect for time was observed (P = 0.009; Fig. 2a) such that the mean Ped IF in the 2005 epoch (Ped IF = 1.59, 95% CI: 1.32, 1.86) was greater than that in the 2000 epoch (Ped IF = 1.16, 95% CI: 0.97, 1.35). Importantly, the variance due to the journal x epoch interaction was small (P = 0.36) indicating that the growth in the Ped IF should be interpreted as being uniform among the journals.

View larger version (26K): [in this window] [in a new window]   Figure 2. a, Pediatric anesthesia Impact Factor (Ped IF) (mean ± sd) for each of the five journals for the epochs, 2000 and 2005. The displayed values are derived from actual citation counts (Table 1) and not from the model. Statistical analysis of individual pediatric anesthesia articles revealed an effect of journal and time for the Ped IF, but not for the interaction term (discussed in Results section). The mean (sd) overall Ped IF based on the citation counts for the five journals in each epoch is shown by the star symbol. The mean (sd) overall Ped IF increased 17% between 2000 and 2005. b, pain impact factor (Pain IF) (mean ± sd) for each of the four journals for the epochs, 2000 and 2005. The displayed values are derived from the actual citation counts (Table 1) and not from the model. Statistical analysis of individual pain articles revealed an effect of journal for the Pain IF but neither time nor an interaction term (discussed in Results section). The mean (sd) overall Pain IF based on the citation counts for the four journals in each epoch is shown by the star symbol. The mean (sd) overall Pain IF increased 12% between 2000 and 2005.

Pain Articles
The citation counts for each of the 520 identified pain articles are presented for each journal and each epoch (n = 238 in 2000 epoch; n = 282 in 2005 epoch) in Table 1. The number of published articles ranged widely among the journals with the BJA publishing only 22 articles and Anesthesia & Analgesia publishing 143 articles during the 2000 epoch. The Pain IFs also ranged widely from 1.4 to 3.4 in the 2000 epoch and from 1.7 to 3.5 in the 2005 epoch. The overall Pain IF increased 12% between 2000 and 2005 (Table 1).

Examination of the pain subspecialty articles by journal and epoch revealed that journals differed in their Pain IFs, although the IFs remained relatively consistent over time. A main effect for journals (collapsing across epochs) was observed (P < 0.0001; Fig. 2b) such that articles from Anesthesiology (Pain IF = 3.4, 95% CI: 2.7, 4.1) had a statistically significant greater IF than all of the other journals excluding Anesthesia & Analgesia (Pain IF = 3.05, 95% CI: 2.7, 3.4). Anesthesia & Analgesia Pain IF did not differ significantly from that in the BJA (Pain IF = 1.9, 95% CI: 1.2, 2.7). The Canadian Journal of Anesthesia (Pain IF = 1.52, 95% CI: 1.0, 2.0) had a smaller Pain IF than all of the other journals. In contrast to the Ped IF, neither a main effect for time (P = 0.41) nor the interaction of journal x time (P = 0.76) was observed indicating that the growth of Pain IF could be interpreted as largely static and uniform among the journals, although the mean citation count per pain article did increase from 2.2 to 2.5 between the 2000 and 2005 epochs (Fig. 2b).

Relationship Between JIF and Subspecialty IF
Table 2 presents the IFs and subspecialty IFs for each journal that were calculated by dividing the total citation counts during the considered epoch by the number of articles (Table 1 for citation counts and number of articles for each epoch). The relationship between subspecialty IFs was distinctive with the Ped IF closely related to Pain IF (Fig. 3) and, except for BJA in the 2005 epoch, Pain IFs exceeded the Ped IFs from the same journal/epoch (i.e., the IFs lie to the left of the line of unity in Fig. 3). Further, journals with greater overall IFs (larger circles in Fig. 3) had greater subspecialty IFs.

View larger version (15K): [in this window] [in a new window]   Figure 3. The pediatric anesthesia subspecialty impact factor (IF) (Ped IF) plotted against the pain subspecialty IF (Pain IF) for each of the four journals (the journal Pediatric Anesthesia was omitted because a Pain IF was not determined for that journal) for the two epochs, 2000 and 2005. The size of each bubble directly reflects the magnitude of the journal IF (JIF). Pain IFs exceeded those of the Ped IFs in corresponding journals and epochs except for British Journal of Anesthesia in 2005.

DISCUSSION

The purpose of this study was to determine the contribution of subspecialty articles on pediatric anesthesia published in four leading anesthesia journals to the IFs of those journals. We found that the Ped IFs were consistently less than the JIF, i.e., they were cited less frequently than the average article published in the four anesthesia journals in 2000 and 2005 (Table 1). In isolation, the impact of the small Ped IF on the JIF is likely limited as the number of pediatric anesthesia articles that are published in these journals is a small percentage of the total number of articles published. In contrast, the Pain IFs more closely paralleled or exceeded the JIFs during the two epochs studied and exceeded the Ped IF, except in one case (Table 1, Fig. 3).

The calculation of the IF lends itself to a number of methodological biases that may undermine the magnitude of small subspecialties, such as pediatric anesthesia. First, although basic science articles tend to generate more citations per article than clinical research articles,11 clinical research and practice guidelines in pediatric anesthesia far outnumber the basic science articles published in that subspecialty.6 Most basic science in anesthesia is conducted at the level of general (non-pediatric) anesthesia and, though the knowledge acquired potentially benefits infants and children, these articles would not be considered pediatric anesthesia articles. In contrast, pain medicine is a subspecialty that is rich in basic science research, and the greater citation counts this generates likely contributes to the greater subspecialty Pain IF determined.

Additionally, the arbitrary 2-year time interval for citing articles may negatively impact clinical specialties, such as pediatric anesthesia and the Ped IF. Garfield maintains that the 2-year interval is not a factor when journals within the same field are compared, although it could be a substantive factor when articles from different disciplines in science are compared.12 In the case of the subspecialty pediatric anesthesia, the citation interval may be a substantive bias against the Ped IF. Clinical studies in children often require more time to complete than those in adults because of the reduced consent rate and difficulty finding qualified subjects to enroll. Multicenter collaboration may be required to enroll the number of subjects and these are far more difficult to complete quickly.13 For these reasons, publication of the study may occur beyond the 2-year interval after substantive articles on the subject have been published. Hence, citations of those articles would fall outside the 2-year interval that is currently used and short-shrift clinical disciplines. Whether Ped IFs would improve if they were calculated over longer time intervals is a subject worthy of further examination. According to Garfield, the citation half-life and immediacy index were developed to address this issue, however, these somewhat obscure parameters do not carry the same weight or influence on authors and publishers as the IF.3

A second potential bias that may reduce the contribution of Ped IFs is that larger scientific audiences favor greater IFs. Although the increased number of citations generated is balanced by the larger number of source articles, bias arises from the fact that a subset of articles are more frequently cited than others. Evidence demonstrates that, overall, 80% of all citations are generated by only 20% of source articles.3 Therefore, publishing a large number of source articles will increase the likelihood that heavily cited articles are included in the denominator of the calculation and are not diluted by the large number of articles.14 In contrast, pediatric anesthesiology is a relatively small subspecialty and therefore the probability that a heavily cited article will be published is less.

It is widely held that one reason for the relatively small Ped IF compared with pain medicine or general anesthesia is that less research is conducted in pediatric anesthesia.15 Since the Accreditation Council for Graduate Medical Education recognized pediatric anesthesiology as an accredited fellowship in 1997, interest in this subspecialty has expanded, although our experience indicates that the majority of the fellows focus their efforts on clinical practice rather than academic pursuits. In fact, very few pediatric anesthesia fellowship programs require a research component and few offer research training and trial involvement. If the contribution of substantive research in pediatric anesthesia to the specialty of anesthesiology is to increase, then both the manpower to conduct studies as well as training opportunities for resident fellows to acquire experience and mentoring in research must increase.

The relative paucity of resources for pediatric research is often under-estimated. In the United States, despite an overall doubling of National Institute of Health funding between 1998 and 2003, the passing of the Children’s Health Act of 2000 and the establishment of the Pediatric Research Initiative in that same year by Congress, the proportion of funding devoted to research in fields of pediatric medicine actually decreased from 12.3 to 11.3%.16 In Europe, the EU Clinical Trials Directive promulgated in 2001, originally intended to improve the safety of clinical research and protection of trial subjects, initially converted research, especially in the field of pediatric medicine into a more expensive, time consuming and cumbersome process.17 Effort has been expended to reverse the errors in the original trials directive in Europe, but the impact of the changes has yet to reach the level of the individual hospital, IRB, and anesthetic department.

CONCLUSIONS

Little is known about the significance of subspecialty IFs. Given the potential importance of JIFs to both journal editors and authors, one cannot discount the possible conscious or subconscious consideration during the review process that articles which are likely to be highly cited are preferred. The subspecialty IF may bring bearing on such considerations. Furthermore, the subspecialty IF may prove to be a valuable barometer of the overall health and activity of research in a subspecialty discipline.

Footnotes

^¶Web of Science (http://isil.isiknowledge.com).

Accepted for publication July 30, 2008.

No external funding was used to complete this study.

Presented, in part, at the annual meeting of the International Anesthesia Research Society, Orlando, Florida, March 4, 2006.

REFERENCES

1. Garfield E. Citation indexes to science: a new documentation through association of ideas. Science 1955;122:108–11[Free Full Text]
2. Brody S. Impact Factor is the best operational measure for medical journals. Lancet 1995;346:1300–1[Web of Science][Medline]
3. Garfield E. History and meaning of the journal impact factor. JAMA 2006;295:90–3[Free Full Text]
4. Smith R. Commentary: the power of the unrelenting impact factor—is it a force for good or harm? Int J Epidemiol 2006;35:1129–30[Free Full Text]
5. van Teijlingten E. Getting your paper to the right journal: a case study of an academic paper. J Adv Nurs 2002;37:506–11[Web of Science][Medline]
6. Smith G. Impact factors in anaesthesia journals. Br J Anaesth 1996;76:753–4[Free Full Text]
7. Seglen PO. Why the impact factor of journals should not be used for evaluating research: education and debate. Br Med J 1997;314:498–502[Free Full Text]
8. Smith G. Research in anaesthesia—the key to the future. Br J Anaesth 1995;75:383–6[Free Full Text]
9. Boldt J, Maleck W, Koetter KP. Which countries publish in important anesthesia and critical care journals? Anesth Analg 1999;88:1175–80[Abstract/Free Full Text]
10. Brambrink AM, Ehrler D, Dick WF. Publications on paediatric anaesthesia: a quantitative analysis of publication activity and international recognition. Br J Anaesth 2000;85:556–62[Abstract/Free Full Text]
11. Amin M, Mabe M. Impact factors: use and abuse. Perspectives in Publishing 2000;1:1–6
12. Garfield E. Journal Impact Factor, a brief review. CMAJ 1999;161:979–80[Free Full Text]
13. Hansson S. Impact Factor as a misleading tool in evaluation of medical journals. Lancet 1995;346:906[Web of Science][Medline]
14. Jemec GB. Impact factors of dermatological journals for 1991-2000. BMC Dermatol 2001;1:7[Medline]
15. Rockoff MA, Hall SC. Subspecialty training in pediatric anesthesiology: what does it mean? Anesth Analg 1997;85:1185–90[Web of Science][Medline]
16. Gitterman DP, Greenwood RS, Kocis KC, Mayes R, McKethan AN. Did a rising tide lift all boats? The NIH budget and Pediatric Research Portfolio. Health Aff 2004;23:113–24[Abstract/Free Full Text]
17. Mitchell C. Clinical trials in paediatric haemotology oncology: are future successes threatened by the EU directive on the conduct of clinical trials? Arch Dis Child 2007;92:1024–7[Free Full Text]

This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Email this article to a colleague Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Request Permissions Citing Articles Citing Articles via Google Scholar Google Scholar Articles by Ramsdell, R. Articles by Houle, T. T. Search for Related Content PubMed PubMed Citation Articles by Ramsdell, R. Articles by Houle, T. T. Related Collections Economics and Health Care Research Education Pediatrics