Has the performance of radiography for detecting pneumonia been overstated? A study in the December edition of the American Journal of Roentgenology suggests that x-ray's sensitivity and specificity drop when uncertain impressions are included.
Radiography's performance as measured by sensitivity and specificity for diagnosing pneumonia varied greatly based on whether language indicating uncertainty was included in the radiology report, according to a research team led by Dr. Alexander Makhnevich of Northwell Health and the Donald and Barbara Zucker School of Medicine at Hofstra/Northwell in Hempstead, NY (AJR, December 2019, Vol. 213:6, pp. 1207-1212).
The researchers found that when more uncertain language was used, the sensitivity and specificity of radiography declined -- to an extent not found with hedging of CT reports. They believe that previous studies on radiography's value for diagnosing pneumonia -- most of which have excluded uncertain findings -- could be overstating its utility.
Uncertain value
Makhnevich and colleagues noted that chest radiography is used as a first-line modality in the diagnosis of pneumonia, which kills 450,000 Americans a year and costs $10.5 billion in hospital costs. CT is considered to be the reference standard.
But despite widespread use of x-ray for pneumonia, doubts persist about its effectiveness. Previous estimates of the modality's sensitivity have ranged from 32% to 78%, with specificity estimates ranging from 59% to 94%.
Those estimates are based on what the authors called a "binary reference standard" -- a diagnosis that's either positive or negative -- to evaluate radiography's performance statistically. But Makhnevich's team points out that in the real world, radiologists frequently hedge when struggling to interpret uncertain findings.
"Uncertain results present a diagnostic dilemma for clinicians, especially when the clinical presentation is ambiguous," the study authors wrote.
The researchers, therefore, decided to reevaluate radiography's performance for pneumonia by including cases in which a radiologist's interpretation was uncertain. They analyzed radiography exams performed from 2015 to 2016 at two tertiary care centers.
After some patients were excluded, the researchers ended up with 2,411 exams in their study population. Of these, 7.3% were positive for pneumonia, 60.9% were negative, and 31.8% were uncertain.
To assess the impact of uncertainty, they measured x-ray's performance in five clinical scenarios.
- Scenario 1: All uncertain x-ray and CT exams were considered to be positive for pneumonia.
- Scenario 2: All uncertain x-ray studies were considered positive, but all uncertain CT studies were classified as negative.
- Scenario 3: All uncertain x-ray studies were negative, but all uncertain CT studies were positive.
- Scenario 4: All uncertain x-ray and CT studies were considered negative.
- Scenario 5: All uncertain x-ray and CT studies were excluded from the analysis.
| Impact of uncertainty on radiography performance for pneumonia | |||||
| Scenario 1 | Scenario 2 | Scenario 3 | Scenario 4 | Scenario 5 | |
| Sensitivity | 51.9% | 59.6% | 13.4% | 19.6% | 32.7% |
| Specificity | 71.3% | 67.1% | 97.7% | 96.4% | 96.8% |
The researchers believe the findings indicate that radiography's performance varies greatly depending on whether uncertain findings are included in the mix. When they aren't -- as indicated in scenario 5 -- x-ray works about as well as indicated in the literature.
But uncertain findings do occur in the real world. And when they are included, radiography's performance dives considerably, with either sharply lower specificity, as in scenarios 1 and 2, or much lower sensitivity, as in scenarios 3 and 4.
"Pneumonia can present atypically, so including patients with only the classic symptoms of pneumonia misses a proportion of cases that are already difficult to diagnose," the authors wrote. "In this retrospective study, we found that uncertain results not only are prevalent but also affect the statistical measures of performance of chest radiography and result in wide-ranging values."
So what's to be done? Makhnevich and colleagues recommended future studies to more closely analyze cases classified as uncertain to determine the true accuracy of chest radiography for pneumonia. They also advised the use of their approach to analyze other "nonbinary" tests for pathologic conditions.
"Uncertain chest radiograph results affect the statistical measures of performance of chest radiography to a much greater degree than uncertain CT results," the authors concluded. "The poor statistical measures of performance of chest radiography for identifying pneumonia may translate into both the underdiagnosis and overdiagnosis of pneumonia."
