This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Email this article to a colleague Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Request Permissions Citing Articles Citing Articles via Google Scholar Google Scholar Articles by Gehring, H. Articles by Dibbelt, L. Search for Related Content PubMed PubMed Citation Articles by Gehring, H. Articles by Dibbelt, L.

---

ARTICLE

Hemoximetry as the "Gold Standard"? Error Assessment Based on Differences Among Identical Blood Gas Analyzer Devices of Five Manufacturers

Hartmut Gehring, MD^*, Lutz Duembgen, PhD, Mareike Peterlein^*, Söhnke Hagelberg, MD^*, and Leif Dibbelt, PhD

From the *Department of Anesthesiology, University Clinic of Schleswig-Holstein, Campus Luebeck, Germany; Institute of Mathematical Statistics and Actuarial Science, University of Bern, Switzerland; Institute of Clinical Chemistry, University Clinic of Schleswig-Holstein, Campus Luebeck, Germany.

Address correspondence and reprints requests to Hartmut Gehring, MD, Professor, Department of Anesthesiology, University Clinic of Schleswig-Holstein, Campus Luebeck, Ratzeburger Allee 160, D-23538 Luebeck, Germany. Address e-mail to gehring{at}uni-luebeck.de.

Abstract

BACKGROUND: The calibration and testing procedures of a pulse oximeter with arterial blood samples from healthy subjects are based on reference values from the hemoximeter. There are no tests to identify the accuracy of the reference devices. Because of this limitation and since the true values of oxygen saturation (sO₂ in %) in blood samples were not known, we used the differences between two identical devices, A and B, for error assessment.

METHODS: Two identical devices, A and B, from five leading manufacturers were investigated. Seventy-two arterial blood samples from 12 healthy volunteers at three different levels of saturation between 100% and 70% sO₂ were randomly evaluated by the test systems.

RESULTS: The observed differences () between Devices A and B, as a measure for the error of the hemoximeters, increased significantly with all manufacturers from level 97 (min, –0.9%; max, 2.6%) to 85 (min, –2.4%; max, 4.3), this effect was even stronger between levels 97 and 75 (min, –4.6%; max, 4.3%). A variance proportion analysis revealed the concentration of the reduced hemoglobin as the main error source for sO₂ measurements. Independent from the sO₂ levels there were also significant differences for the carboxy hemoglobin concentration in the range of 0%–4% and for the methemoglobin concentration in the range of 0%–1%.

CONCLUSIONS: The variance of sO₂ measurements between identical devices increased significantly when saturation decreased from the normal level of 97% to the hypoxemic levels of 85% and 75%.

There is no test system for identifying the measurement error of the reference procedure for pulse oximetry, i.e., hemoximetry, also known as CO-oximetry. Hemoximetry discontinuously measures hemoglobin oxygen saturation and dyshemoglobines from blood samples. All medical devices deliver the measured values with a certain error.

CO-oximetry or hemoximetry is an essential component of blood gas analyzer systems. Hemoximetry presents hemoglobin oxygen saturation, dyshemoglobines, and total hemoglobin concentration data. The functional oxygen saturation measured by this procedure is the basis for calibrating pulse oximeters. Pulse oximeters cannot be calibrated using physical procedures, but only by directly comparing the reported measurements and the parallel arterial oxygen saturation measured by hemoximetry in a group of healthy subjects (1). Thus, the errors of hemoximetry are carried over into measurements made by the pulse oximeter.

There is no standard procedure for checking the measurement error of the hemoximeter. Usually the devices are calibrated by the manufacturer before delivery. For calibrating hemoximetry in everyday clinical routine, aqueous solutions supplied by the manufacturers are used. A gravimetric procedure that cannot be applied in normal clinical routine has been described (2).

Comparisons of hemoximeters from leading manufacturers, which are intended for use as reference procedures, have not been published. The hemoximeters of the companies participating in the present study were introduced in papers dealing with the reference analysis of pulse oximeters (3–9).

For pulse oximeter calibration and confirmation tests done on volunteers, the United States Food and Drug Administration (FDA) requires only one CO-oximeter device for measuring sO₂ used according to the hemoximeter manufacturer’s recommended procedures and supplies (1).

Bland and Altman pointed out that not just test devices but also reference devices produce errors (10). The Bland–Altman procedure for error analysis and presentation is useful for the presentation of pulse oximeter accuracy data (11).

The objective of this study was to determine the measurement error associated with the determination of arterial oxygen-saturation by hemoximetry.

Because no reference procedure is available for measuring oxygen saturation with CO-oximeters, the pairwise differences between identical devices from five leading manufacturers, as well as the differences between manufacturers, were assessed in order to estimate the error between the hemoximeters.

METHODS

Device Preparation
Table 1 lists the participating companies as well as the serial numbers and model descriptions of the devices. Table 2 describes the technical measurement procedure. All devices were assembled pairwise in a climatically controlled room and serviced by employees from the manufacturer before they were put into operation. The air pressure gauges integrated within the devices were checked and synchronized with the reference system (http://193.175.120.23/an/pt/solar/wetter/werte.htm). During monitoring phases, the prevailing air pressure values were recorded hourly. The devices were adjusted so that they reported identical units and denotations of the variables.

The amount of blood required to provide sufficient material for the entire range of blood gas analyzers was tested under several conditions. At least three syringes specially designed for blood gas sampling (PICO^TM 50 with dry electrolyte-balanced heparin (80 IU, Radiometer Copenhagen)), and filled with 2 mL of blood, appeared sufficient to provide all systems with an adequate amount of blood.

Study Protocol
A set up was chosen for the study which

1. reflected the reality for the operation or the intensive care unit with blood gas analyses from patients, and
   
2. also corresponded to the standards of the desaturation studies for the testing and calibration of pulse oximeters.
   

Twelve healthy male and female volunteers were investigated over 4 days regarding standardized calibration procedures for pulse oximeters corresponding to the requirements of the FDA over the range of 70%–100% sO₂. The study was approved by the Ethical Committee of the University of Luebeck, Germany, and all participants gave written informed consent. All volunteers breathed an oxygen/nitrogen mixture with high flow (15 L/min) given by a Trajan 808^TM (Draeger Medical, Luebeck) via a valve-less face mask. Three N-3000 with finger clip and one N-595 with a forehead sensor (all Nellcor, Pleasanton, CA) served as reference pulse oximeters for breathe down control. Mean values, as well as individual data points from the systems, were presented continuously on a display. The protocol followed the standard procedure of the FDA (1) where five levels (L) were established in the sO₂ range between 100% and 70%, and five blood samples were withdrawn under steady state plateau conditions for testing, with the hemoximeter at the breathe-down laboratory (2 OSM 3 and 2 ABL 725, Radiometer Copenhagen, Denmark). At the end of three of the levels (L₉₇ near to 97% sO₂, L₈₅ at 85% sO₂ and L₇₅ at 75% sO₂) and also under the presence of plateau conditions, three syringes were rapidly filled representing the sample for one level. The breathe-down procedure was repeated, so that at least six sampling points materialized for each volunteer. The three syringes marked only with a colored spot were mixed and then transferred within 30 s to the adjacent study laboratory. Three technical assistants, blinded to the syringes, were randomly assigned to a manufacturer and the Devices A or B for testing. Data from each blood gas analyzer were stored on disk and in printed form before they were transferred to an Excel^TM data file.

Statistical Analysis
It should be remembered that 1) each device received blood that was randomly selected from a sample, and that 2) there was no value for the true saturation.

The variables sO₂ (hemoglobin oxygen saturation), cO₂Hb (oxygen content of hemoglobin), cHHb (deoxyhemoglobin concentration), cCOHb (carboxyhemoglobin content), cMetHb (methemoglobin content), and ctHb (total hemoglobin content) were analyzed. The basis for evaluation were the raw data and the differences between the Devices A and B related to level, considering session, subject, manufacturer, and device.

The measurements are modeled as:

With i = 1, 2: session; j = 1, 2, 3: level; k = 1, ..., 12: subject; l = 1, ..., 5: manufacturer; m = 1, 2: device.

Here M_ijk is a randomized effect depending on session, level, and subject, _jlm is an effect depending on level, manufacturer and device, and _ijklm is a measurement error with the expected value 0 and standard deviation _jl, depending on level and manufacturer.

The Wilcoxon signed rank sum tests with Bonferroni-Holm adjustment were applied to calculate P values for differences of devices and differences between manufacturers. See also Appendix A.

The question regarding which variable was responsible for the increasing errors in sO₂ measurements with respect to levels 97, 85, and 75 was answered by analysis of variance proportion. The procedure for analyzing the proportion of the variance of cHHb and cO₂Hb that contributed to the variance of the sO₂ measurements is listed in Appendix B.

RESULTS

Each of the test systems was used to analyze n = 72 samples. The distribution of the absolute values for the individual manufacturers is given in Figure 1. The significances listed for level 97 between the mean values of the two Devices A and B from one manufacturer, compared with the mean value from all other companies were masked by the increasing variances at levels 85 and 75.

View larger version (12K): [in this window] [in a new window]   Figure 1. Mean and sd of the absolute values, summarized for Sessions 1 and 2. *P < 0.05; test between Group 1 (mean of Devices A and B) versus Group 2 (mean of the other companies).

The measured differences, pairwise, between Devices A and B, as a measure for the error of the hemoximeters within a series increased clearly and significantly with all manufacturers between levels 97 and 85. This effect was even stronger between level 97 and 75 (Fig. 2).

View larger version (15K): [in this window] [in a new window]   Figure 2. Mean and sd of the differences of the devices (A–B) of each company, ordered to level, summarized for Sessions 1 and 2. P < 0.05; test between Devices A and B.

Summarized for all samples, the differences of the Devices A and B were recorded as means and standard deviations for each manufacturer in Table 3, completed with the values for cHHb and cO₂Hb as well as the sums of cO₂Hb and cHHb representing the denominator of the formula sO₂ = cO₂Hb/(cO₂Hb + cHHb).

The measurement of cCOHb showed no dependence on sO₂ level or session, but revealed significant differences between the manufacturers, as well as between the Devices A and B (Table 3). Overall, the absolute measured values for cCOHb were scattered over a broad range between 0% and 4%.

The measurement of cMetHb also showed no dependence with regard to sO₂ level and session, but here, just as with cCOHb, significant differences were seen between the manufacturers as well as between the Devices A and B (Table 3). Overall, the scattering of the measured values was restricted to a narrow range of between 0% and 1% cMetHb.

The variances associated with the measurement of cHHb were disproportionately responsible for the increasing differences between Devices A and B (Table 4).

DISCUSSION

For all manufacturers, the differences of the sO₂ values, measured with identical devices of a series, increase as saturation falls. For the analysis of sO₂, one can therefore assume that a measurement error also exists even for the "gold standard" of hemoximetry, and that this will influence pulse oximeter calibration.

For the absolute values, significant differences between the instrument manufacturers already occurred at level 97, an effect that was masked by the increasing variance at levels 85 and 75. The variance for the measurement of cHHb can be identified as an important cause underlying the error. The measurement errors for cMetHb were significant, but restricted to a range of 0%–1%. However, there were significant variances in the measurement error of cCOHb between 0% and 4%.

A standardized production of blood samples with a defined saturation level between 0% and 100% can only be achieved in isolated cases using a tonometrically based gravimetric procedure (2). A primary calibration can only occur in the factory before the devices are shipped to the customer. Any further calibration is then based merely on the application of aqueous sample materials; therefore, a fundamental determination of the measurement error is not possible. In particular, the error due to an inadequate hemolysis of the cellular substances in the blood is lost when calibrating with the aqueous samples.

As a possible means for approaching the unknown values for the true sO₂ levels, laboratories involved in calibrating pulse oximeters use several hemoximeters of an identical design from one or several companies, and then compute mean values (12) from them. This assumption is based, however, on a randomly distributed error that becomes minimized upon averaging.

The study presented here was based on the fact that all test devices received randomized blood from a population sample. In order to test for reproducibility, the breathe down procedure was carried out twice for each test subject (Sessions 1 and 2). In the statistical analysis, an effect based on differences between Sessions 1 and 2 could be excluded.

Considering the results presented, we can also assume a marked error within the reference devices. This is an effect that was already clearly identified by Bland and Altman, a fact which led them to establish their own evaluation procedure (10). A modified presentation, related only to the values obtained from the reference devices (11), can therefore be recommended only with reservations.

The algorithms and corrective procedures established by the manufacturers represent a further gray zone as regards to the reporting of measurements within the reference systems. As a single example, the number of wavelengths used in the devices investigated clearly varies between the different manufacturers (Table 2).

The variable sO₂ reported by the hemoximeters refers to the functional oxygen saturation. The calculation is based on hemoglobin that can bind oxygen (functional sO₂ = cO₂Hb/(cO₂Hb + cHHb) and is directly comparable to the value reported by the pulse oximeters. A new pulse oximeter (12) is now also able to measure cMetHb and cCOHb so that, in principle, a measurement of the fractional oxygen saturation (fractional sO₂ = cO₂Hb/(cMetHb + cCOHb + cO₂Hb + cHHb) is also possible by pulse oximetry. In a paper by Barker et al. (12), a Bland-Altman calculated difference (bias) of –1.12% and a precision of ±2.19% was reported for the cCOHb pulse oximetric value. This represents an order of magnitude that was also measured in the present test with the reference devices.

A fundamental problem is the lack of uniformity in the nomenclature used by the different manufacturers for the reported variables. Only the measurement of functional oxygen-saturation (sO₂ in %) was applied uniformly both for the hemoximeter and the pulse oximeter. The term "fractional oxygen saturation" will find increasing use due to the application of the new pulse oximeters in clinical settings. Errors arising from different spellings and definitions are more or less inevitable. In Germany, a consensus-building conference involving the leading manufacturers of blood gas analysis devices has already taken place, the results of which were published in the "Qualitest Consensus" (13).

At the time when the testing was performed, the devices examined were state-of-the-art. New developments in oximeter testing and standardization are desirable. A standardized test procedure for hemoximeters is important for the variables sO₂, cO₂Hb, cHHb, cCOHb, and cMetHb, similar to those already established for the measurement of ctHb (14).

CONCLUSIONS

Errors in hemoximeter determination of sO₂ depend both on the manufacturer of the hemoximeter as well as the sO₂ range. The variable cHHb disproportionately contributes to the measurement error for sO₂. Further, the measurement error for cCOHb is related to the absolute values and the variance is clearly greater than is the case for cMetHb, a fact relevant to calculating the fractional saturation.

Finally, we strongly recommended the Bland and Altman procedure as the preferred analysis method for reporting and presenting hemoximeter errors.

APPENDIX A: STATISTICAL TESTS AND CONFIDENCE INTERVALS

Estimators and Confidence Intervals for Single Standard Deviations _jl

We assume that the differences

are distributed according to N(_ijkl1 – _ijkl2, 2_jl²), apart from a small fraction of outliers, whereby _ijkl1 – _ijkl2 is the systematic difference between two identical devices. (Note that the effect M_ijk disappears here.) Denoting the available differences D_ijkl for given level j and manufacturer l with D₁, D₂,...,D_n, conventional parametric confidence intervals for _jl are then given by

Here _m;β² denotes the β-quantile of the ² distribution with m degrees of freedom.

To safeguard against outliers and obtain a robust estimate for _jl, we deleted the smallest and largest differences D_a. For this purpose, one has to replace the ² quantiles in Eq. [A1] with new quantiles taking into account this trimming. The latter have been determined using extensive Monte-Carlo simulations. Using the 50% quantile, we also calculated a robust point estimator for _jl, i.e., a "trimmed version’’ of

Confidence Intervals for Ratios _cl/_bl

For 1 b < c 3, a standard (1–) -confidence interval for the ratio _cl/_bl is given by

Here F_{m[1], m[2];β} denotes the β-quantile of Fisher’s F distribution with m(1) and m(2) degrees of freedom, and

is the usual estimator of _jl based on all n(j) available differences D_a^jl = D_ijkl.

Again we modified this classical procedure by means of trimmed samples and corresponding surrogates for the F quantiles.

Statistical Inference About the Device Differences _ijkl1 – _ijkl2

The null hypothesis that "_ijkl1 – _ijkl2 = 0", i.e., there is no systematic difference between the two devices of manufacturer l at level j, was tested via Wilcoxon’s signed rank test. Using Wilcoxon’s signed rank test rather than a corresponding t-test has the advantage that the results are less sensitive to outliers, and one could even relax the model assumptions considerably. Since we are testing 15 null hypotheses simultaneously (5 manufacturers, 3 levels), we adjusted our P values via the Bonferroni–Holm procedure (15).

With D₁, D₂,..., D_n as in "Estimators and Confidence Intervals for Single Standard Deviations _jl" section, a (1–)-confidence interval for _ijkl1 – _ijkl2 consists of all numbers such that the Wilcoxon signed rank statistic W(D₁ – , D₂ – , ..., D_n – ) lies within the thresholds for the two-sided Wilcoxon’s signed rank test at level . In other words, we determine the set of all values such that the latter test does not reject the null hypothesis "_ijkl1 – _ijkl2 = ".

Comparing Manufacturers

To test whether the devices of manufacturer l are significantly different from the other manufacturer’s devices at a certain level j, we considered the means

over two identical devices with _jl = (_jl1 + _jl2)/2 describing the average effect of manufacturer l at level j and _ijkl = (_ijkl1 + _ijkl2)/2 having mean zero and standard deviation

Now we considered the differences

where the bar stands for averaging over the remaining four manufacturers. With these differences we tested the 15 null hypotheses "_ijkl = _ijkl" via Wilcoxon’s signed rank test plus Bonferroni–Holm adjustment, analogously as in "Statistical Inference About the Device Differences _ijkl1 – _ijkl2" section.

APPENDIX B: VARIANCE PROPORTIONS

Our aim is to quantify the contribution of errors in cO₂Hb and cHHb measurements to the total error in measuring sO₂. To this end, we model the former measurements as cO₂Hb = µ₁ + ₁ and cHHb = µ₂ + ₂ with true concentrations µ₁, µ₂ and independent random errors ₁, ₂ having mean zero and standard deviations ₁, ₂. Assuming that ₁ + ₂ < < µ₁ + µ₂, we may expand sO₂ as follows:

The second summand on the right hand side has variance

Thus, the variance proportion (VP) of cO₂Hb, i.e., the contribution to the overall variance of sO₂ is essentially equal to

To obtain an estimate for this variance proportion, we plug in classical estimates for the unknown means µ_i and standard deviations _i based on all measurements from a certain level. That means, we average over all test subjects, sessions, manufacturers, and devices. This leads to P_cO2Hb = ₂²₁²/₂²₁² + ₁²₂².

Footnotes

Accepted for publication April 19, 2007.

Supported by the participating manufacturers.

Hartmut Gehring, MD, is the head of the Laboratory for Controlled Hypoxemia Studies (CHS). Testing and calibrating of the pulse oximeter is the essential task of this laboratory.

The study with the volunteers was performed at the request of, and financially supported by, a pulse oximeter manufacturer.

REFERENCES

1. ISO 9919. Medical electrical equipment—particular requirements for the basic safety and essential performance of pulse oximeter equipment for medical use. 2005
2. Zander R, Schaffartzik W. Qualitest Häm-Oxymeter. Georg Thieme Publisher (ISSN 1434–0143) 1999;5:0–8
3. Dumas C, Wahr JA, Tremper KK. Clinical evaluation of a prototype motion artifact resistant pulse oximeter in the recovery room. Anesth Analg 1996;83:269–72[Abstract]
4. Senn O, Clarenbach CF, Kaplan V, Maggiorini M, Bloch KE. Monitoring carbon dioxide tension and arterial oxygen saturation by a single earlobe sensor in patients with critical illness or sleep apnea. Chest 2005;128:1291–6[Web of Science][Medline]
5. Van de Louw A, Cracco C, Cerf C, Harf A, Duvaldestin P, Lemaire F, Brochard L. Accuracy of pulse oximetry in the intensive care unit. Intensive Care Med 2001;27:1606–13[Web of Science][Medline]
6. Berkenbosch JW, Tobias JD. Comparison of a new forehead reflectance pulse oximeter sensor with a conventional digit sensor in pediatric patients. Respir Care 2006;51:726–31[Web of Science][Medline]
7. Kugelman A, Wasserman Y, Mor F, Goldinov L, Geller Y, Bader D. Reflectance pulse oximetry from core body in neonates and infants: comparison to arterial blood gas oxygen saturation and to transmission pulse oximetry. J Perinatol 2004;24:366–71[Medline]
8. MacLeod DB. The desaturation response time of finger pulse oximeters during mild hypothermia. Anaesthesia 2005;60:65–71[Web of Science][Medline]
9. Hummler HD, Englelmann A, Pohlandt F, Högel J, Franz AR. Accuracy of pulse oximetry readings in an animal model of low perfusion caused by emerging pneumonia and sepsis. Intensive Care Med 2004;30:709–13[Web of Science][Medline]
10. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986;1:307–10[Web of Science][Medline]
11. Bickler PE, Feiner JR, Severinghaus JW. Effects of skin pigmentation on pulse oximeter accuracy at low saturation. Anesthesiology 2005;102:715–9[Web of Science][Medline]
12. Barker SJ, Curry J, Redford D, Morgan S. Measurement of carboxyhemoglobin and methemoglobin by pulse oximetry: a human volunteer study. Anesthesiology 2006;105:892–7[Web of Science][Medline]
13. Zander R. Qualitest Consensus. Georg Thieme Publisher (ISSN 1434–0143) 2005;8:0–7
14. Gehring H, Hornberger C, Dibbelt L, Roth-Isigkeit A, Gerlach K, Schumacher J, Schmucker P. Accuracy of point-of-care-testing (POCT) for determining hemoglobin concentrations. Acta Anaesthesiol Scand 2002;46:980–6[Web of Science][Medline]
15. Holm S. A simple sequentially rejective multiple test procedure. Scand J Statistics 1979;6:65–70

This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Email this article to a colleague Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Request Permissions Citing Articles Citing Articles via Google Scholar Google Scholar Articles by Gehring, H. Articles by Dibbelt, L. Search for Related Content PubMed PubMed Citation Articles by Gehring, H. Articles by Dibbelt, L.