Interrater reliability of emergency medicine residents in measurement of optic nerve sheath diameter with computed tomography

Abstract

OBJECTIVES: Many studies have recently been conducted on measuring optic nerve sheath diameter (ONSD) with computed tomography (CT). However, no studies focused on the interrater reliability in ONSD measurements with CT yet. Our first aim was to investigate the interrater reliability of the emergency medicine residents in the measurement of the ONSD with CT. Our secondary aim was to evaluate the interrater reliability and agreement of the emergency medicine residents and neuroradiologist measurements, which is the gold standard.

METHODS: Twelve residents (six seniors and six juniors) and a neuroradiologist measured ONSD in twenty different CTs in axial and sagittal planes. The interrater reliability was calculated by the intraclass correlation coefficient (ICC), and the level of agreement in categorical variables was calculated by kappa (κ) analysis.

RESULTS: We found that the interrater reliability level of all residents was “good” (ICC: 0.824), for seniors was “good” (ICC: 0.824), and for juniors was “moderate” (ICC: 0.748) in the measurement of ONSD. ICC was 0.812 for axial, and 0.783 for sagittal plane measurements. The interrater reliability between residents and the neuroradiologist measurements was “good” (ICC 0.891), and the agreement was found to be “good” (κ: 0.688; P < 0.001). The sensitivity of residents in detecting increased ONSD was 78%, and specificity was 90.8%.

CONCLUSION: The ONSD measurements with CT performed by the residents are reliable in themselves, and they are compatible with the gold standard measurements.

Introduction

The optic nerve is the extension of the brain with the sheath around it, and the optic nerve sheath diameter (ONSD) is widened in conditions that lead to increased intracranial pressure.[1],[2] In recent years, it has been shown that there is a positive linear relationship between the ONSD and intracranial pressure and that the increased ONSD and poor prognosis are correlated.[3],[4]

ONSD measurements are usually performed by ultrasonography (US) or computed tomography (CT). In ONSD measurements with US, significant differences can be seen between raters, since the measurement skill varies from person to person, therefore many studies have been published investigating the interrater reliability (IRR) of ONSD measurements using US. In recent years, many studies on measuring ONSD with CT have been published. However, there is no study focusing on IRR in ONSD measurements by CT.

In this study, our first aim was to investigate the IRR level of the measurement of ONSD in brain CT by emergency medicine residents. Our secondary goals are (1) whether the resident seniority and the measured cross-section affect IRR; (2) to investigate the agreement and diagnostic accuracy by comparing emergency medicine residents with neuroradiologist measurements, which is the gold standard.

Material and Methods

This is an observational, cross-sectional study. The study included 12 residents and one neuroradiologist working in a level 3 emergency medicine department. Ten widened (≥5.5 mm) and 10 normal (<5.5 mm) ONSD images were selected by the neuroradiologist among 3.533 CT images of patients who admitted to Dokuz Eylül University Hospital between January 1,2018 and December 31, 2018.

Six senior (postgraduate year 4) and six junior (postgraduate year 1) resident volunteers were included in the study. Residents were paired with each other, and 66 groups of two residents were created. Two brain CTs were assigned to each group with the cluster-type sampling method, one with normal and one with increased ONSD. Twelve residents made a total of 352 measurements, in axial and sagittal planes, in a total of 176 CT images. All measurements were completed within 3 weeks. Residents were given standardized 1-h training by the neuroradiologist participating in the study before the measurement.

All the CT scans were obtained by the 160-slice CT scanner (Toshiba^® CT Aquilion Prime, Toshiba^® Medical Systems, Japan) with 3 mm section thickness. The evaluation of the images and measurements were performed on Sectra^® Workstation IDS7 (version 20.2.10.3376, Sweden). Measurements were made from the left eye, in the axial and sagittal planes, 3 mm behind the optic disc, using a 5-fold magnification using a caliper. Evaluators were blinded to measurements of other raters and pathological findings in the brain parenchyma. In order to accurately measure the optic nerve adipose tissue, measurements were made in the abdominal view window. [Figure 1] shows the measurements of the optic nerve sheath from the left eye axial (a) and sagittal (b) planes. The neuroradiologist made all ONSD measurements, and these measurements were accepted as the gold standard. All measurements were recorded separately on data collection forms. Our study was started after the approval of Ethics Committee for Clinical Studies of Dokuz Eylül University Faculty of Medicine (Decision No: 2019/19-40, date: July 31, 2019).

Figure 1. (a) Axial plane measurement, (b) sagittal plane measurement of optic nerve sheath diameter

SPSS 24.0 (IBM^® Corporation, Armonk, New York, United States) program was used to analyze the data. The normality of data was analyzed by the Kolmogorov–Smirnov test. The variables with normal and abnormal distribution presented as mean with standard deviation and median with interquartile range (IQR), respectively. Reliability between measurements was shown with the intraclass correlation coefficient (ICC). ICC <0.5 was interpreted as “poor,” 0.5–0.74 as “moderate,” 0.75–0.89 as “good,” and ≥ 0.90 as “perfect” in terms of reliability. The measurement results of ONSD were dichotomized as normal (<5.5 mm) and enlarged (≥5.5 mm). Kappa analysis was performed to determine the agreement between residents and neuroradiologist. Kappa value was evaluated ≤ 0.2 as “slight,” 0.21–0.4 as “fair,” 0.41–0.6 as “moderate,” 0.61–0.8 as “substantial,” and >0.80 as a “perfect” fit. The sensitivity, specificity, and accuracy of residents in detecting widened ONSD were calculated. All analyses were evaluated within a 95% confidence interval (CI). P < 0.05 was considered to be significant.

Results

Nine of the residents participating into study were male, and the median age was 28 years (IQR: 26–28 years). The mean of the measurements made by the residents on the axial plane in CT images with normal ONSD was 5.2 ± 0.57 mm, and the mean on the sagittal plane was 4.9 ± 0.65 mm. On CT images with widened ONSD, the mean of the measurements on the axial plane was 6.5 ± 0.72 mm, and the mean on the sagittal plane was 6.4 ± 0.85 mm. We found ICC as 0.798 in the ONSD measurement by emergency medicine residents. The measurements and ICC according to the seniority of the residents and cross-section are shown in [Table 1].

Table 1. The interrater reliability of emergency medicine resident measurements of optic nerve sheath diameter

We found IRR of residents and neuroradiologist measurements was 0.891 [Table 2]. In the dichotomized data, the interrater agreement between residents and neuroradiologist measurements as “good” (κ: 0.688; P < 0.001). The sensitivity of residents in detecting increased ONSD was 78%, specificity was 90.8%, and accuracy was 84.3% [Table 3]. ONSD measurements of the neuroradiologist and residents on axial and sagittal planes are shown in [Supplement Table 1].

Table 2. The interrater reliability between the measurements of emergency medicine residents and neuroradiologist

Table 3. The agreement between measurements of emergency medicine resident and neuroradiologist in the dichotomized data

Discussion

Many studies investigating ONSD measurement with CT have published with limited data on reliability between measurements, although there is no study focused on this subject. We aimed to determine the IRR in the measurement of ONSD with CT by emergency medicine residents and for the reliability and agreement between residents and neuroradiologist measurements that are accepted as the gold standard.

In addition to sensitivity, specificity, and accuracy, the consistent measurement of evaluators is the essential parameter that affect the reliability of the measurement method. In other words, as well as accurate measurement, to get similar results in repeating measurements is one of the crucial factors for the acceptance of the measurement method in daily practice. Even if the measurement method is well standardized, the differences between the skill level of the evaluators and the specific training they receive may lead to significant difference between the measurement. Reliability between evaluators is an important issue, as ONSD measurements with CT are now beginning to be included in the practice of the emergency medicine. In this regard, the fact that the IRR level of emergency medicine residents is “good” suggests that ONSD measurements can be performed by residents in ED successfully. We also found the IRR (ICC: 0.891) and interrater agreement (κ: 0.688) for ONSD measurements between residents and neuroradiologist as “;good.”

We have shown that the IRR level of senior residents' measurements is higher than that of juniors (ICC 0.824 vs. 0.748). It is an expected result that the senior residents' skills in detecting the optic nerve correctly in the CT and making accurate measurements of ONSD are high. Oberfoell et al. investigated the IRR level of residents on ONSD measurements by US. It was shown that in the measurements performed by 51 residents. The ICC of senior residents was higher, although the level of IRR of senior (ICC 0.44) and junior residents (ICC: 0.40) was weak.[5] In our study, while the IRR between senior residents and the neuroradiologist measurements was “excellent” (ICC: 0.904), the IRR level of junior residents and the neuroradiologist was “good” (ICC: 0.877). As a result, the IRR level of senior residents was higher both among themselves and with the neuroradiologist on ONSD measurements.

Limited number of studies revealed IRR in ONSD measurements using CT, but the primary end points of these studies were not evaluating the IRR between measurements. Sekhon et al. had been investigated the association between ONSD widening in CT and mortality, only the two evaluators' measurements were compared, and the IRR level of evaluators was found to be “perfect” (ICC: 0.92).[6] In a study investigating the relationship between ONSD and early neurological outcome after cardiac arrest, Hwan Kim et al. reported the IRR level of two emergency medicine specialists as “good” (ICC: 0.833).[7] In another study conducted by Yesilaras et al. to determine the diagnostic and prognostic value of ONSD measurement by CT in patients with spontaneous subarachnoid hemorrhage, IRR level of four emergency medicine specialists was found to be “good” (ICC: 0.84).[8] However, since the primary aims of these studies were not to find the IRR level of raters in ONSD measurements with CT, a limited number of evaluators were included into analysis. The strength of our study is that including 12 emergency medicine residents. The characteristics of the studies calculated IRR in ONSD measurement with CT are shown in [Table 4].

Table 4. The characteristics of the studies focused on interrater reliability in optic nerve sheath diameter measurement with computed tomography

In studies comparing ONSD measurements with US, the IRR ranged between 0.39 and 0.96, whereas it was between 0.798 and 0.91 with CT.[5],[6],[7],[8],[10],[11],[12],[13],[14] It can be expected that the IRR levels will be lower with US rather than with CT. US examinations can be affected by many factors that depend on the US device, and skill of the physician, such as proper handling of the probe, image optimization, measuring from the appropriate section, viewing the optic nerve at an appropriate angle. CT has a several advantages for ONSD measurements such as minimizing artifacts, image reforming ability during the evaluation, easier optic nerve location with comparative images, and magnification of images.

In our study, the measurements in accordance with CT planes, we found the emergency medicine residents' IRR levels to be “good” among themselves and the neuroradiologist both in axial and sagittal planes. We have shown that measurements in axial and sagittal planes do not affect the measurement results. In other studies, IRR levels of measurement in axial and sagittal sections were similar.[6],[11],[12],[15]

Limitations

In our study, the residents evaluated CT images in the department of radiology instead of chaotic environment of the emergency department may have caused the results to be more reliable. Our study was conducted in a single center; it restricts the generalization of the results. A single neuroradiologist was used as the gold standard; however, intrarater reliability for the neuroradiologist was not assessed.

The mean of ONSD was very close to the cutoff limit (5.2 ± 0.57 mm for the axial plane and 4.9 ± 0.65 mm for the sagittal plane) in the selected CT for normal ONSD that may lead to low sensitivity in detecting. If CT images with a smaller diameter of optic nerve had been enrolled in the study for the normal ONSD, the sensitivity would be higher even if there were a few millimeters of deviations in the measurements.

The measurements made by a single neuroradiologist will not reflect the measurements made by all radiologists. In addition, a radiologist does not evaluate ONSD in their daily practice. Therefore, comparing diagnostic abilities of emergency physicians and radiologists has a limited value in ONSD measurement.

Conclusion

ONSD measurements performed by senior residents on CT are highly reliable, and it is well compatible with the gold standard measurements.

Supplement. Table 1: Measurements of the neuroradiologist and emergency medicine residents in axial and sagittal planes

Ethical approval was obtained by the Dokuz Eylul University Faculty of Medicine Ethics Committee for Non-Interventional Clinical Research. (Decision No: 2019/19–40, Date: July 31, 2019).

Irem Olcay Yilmaz: Conceptualization, methodology, data curation, supervision, and editing. Ersin Aksay: Conceptualization, methodology, writing-original draft preparation, supervision, reviewing and editing. Basak Bayram: Conceptualization, methodology, formal analysis, and editing. Nese Colak Oray: Conceptualization, methodology, formal analysis, and editing. Nuri Karabay: Conceptualization, methodology, data curation, and editing.

None Declared.

None.

References

1. Liu D, Kahn M. Measurement and relationship of subarachnoid pressure of the optic nerve to intracranial pressures in fresh cadavers. Am J Ophthalmol 1993;116:548‐56.
2. Ohle R, McIsaac SM, Woo MY, Perry JJ. Sonography of the optic nerve sheath diameter for detection of raised intracranial pressure compared to computed tomography: A systematic review and meta‐analysis. J Ultrasound Med 2015;34:1285‐94.
3. Balestreri M, Czosnyka M, Hutchinson P, Steiner LA, Hiler M, Smielewski P, et al. Impact of intracranial pressure and cerebral perfusion pressure on severe disability and mortality after head injury. Neurocrit Care 2006;4:8‐13.
4. Waheed S, Baig MA, Siddiqui E, Jamil D, Bashar M, Feroze A. Prognostic significance of optic nerve sheath diameter on computed tomography scan with severity of blunt traumatic brain injury in the emergency department. J Pak Med Assoc 2018;68:268‐71.
5. Oberfoell S, Murphy D, French A, Trent S, Richards D. Inter‐rater reliability of sonographic optic nerve sheath diameter measurements by emergency medicine physicians. J Ultrasound Med 2017;36:1579‐84.
6. Sekhon MS, McBeth P, Zou J, Qiao L, Kolmodin L, Henderson WR, et al. Association between optic nerve sheath diameter and mortality in patients with severe traumatic brain injury. Neurocrit Care 2014;21:245‐52.
7. HwanKimY,HoLeeJ,KunHongC,WonChoK,HoonYeoJ, Ju Kang M, et al. Feasibility of optic nerve sheath diameter measured on initial brain computed tomography as an early neurologic outcome predictor after cardiac arrest. Acad Emerg Med 2014;21:1121‐8.
8. Yesilaras M, Kilic TY, Yesilaras S, Atilla OD, Öncel D, Çamlar M. The diagnostic and prognostic value of the optic nerve sheath diameter on CT for diagnosis spontaneous subarachnoid hemorrhage. Am J Emerg Med 2017;35:1408‐13.
9. Legrand A, Jeanjean P, Delanghe F, Peltier J, Lecat B, Dupont H. Estimation of optic nerve sheath diameter on an initial brain computed tomography scan can contribute prognostic information in traumatic brain injury patients. Crit Care 2013;17:R61.
10. Shah S, Kimberly H, Marill K, Noble VE. Ultrasound techniques to measure the optic nerve sheath: Is a specialized probe necessary? Med Sci Monit 2009;15:MT63‐8.
11. Johnson GG, Zeiler FA, Unger B, Hansen G, Karakitsos D, Gillman LM. Estimating the accuracy of optic nerve sheath diameter measurement using a pocket‐sized, handheld ultrasound on a simulation model. Crit Ultrasound J 2016;8:18.
12. Woster CM, Zwank MD, Pasquarella JR, Wewerka SS, Anderson JP, Greupner JT, et al. Placement of a cervical collar increases the optic nerve sheath diameter in healthy adults. Am J Emerg Med 2018;36:430‐4.
13. Jeon JP, Lee SU, Kim SE, Kang SH, Yang JS, Choi HJ, et al. Correlation of optic nerve sheath diameter with directly measured intracranial pressure in Korean adults using bedside ultrasonography. PLoS One 2017;12:e0183170.
14. Hassen GW, Bruck I, Donahue J, Mason B, Sweeney B, Saab W, et al. Accuracy of optic nerve sheath diameter measurement by emergency physicians using bedside ultrasound. J Emerg Med 2015;48:450‐7.
15. Martin M, Lobo D, Bitot V, Couffin S, Escalard S, Mounier R, et al. Prediction of early intracranial hypertension after severe traumatic brain injury: A prospective study. World Neurosurg 2019;127:e1242‐e1248.

We would like to thank Murat Yeşilaras, MD, for his contributions to the article and data analysis.