Chest x-rays may have had limited usefulness in diagnosing COVID-19 cases early in the pandemic, despite widespread use of the technology, according to a study published in the February issue of Clinical Imaging.
In a reader study, a group led by Dr. Judy Gichoya of Emory University in Atlanta evaluated radiologist performance in diagnosing COVID-19 on chest x-ray images acquired at the time patients presented in emergency departments. They found poor agreement among 10 readers from four academic institutions and that radiologist performance was ultimately nonspecific for diagnosing COVID-19.
"This performance does not improve even when the clinical history is provided, and improves only slightly with >5 years of experience among the readers," the authors wrote.
The findings may inform the design of patient triage workflows for COVID-19 in emergency departments, especially in both U.S. and international underserved regions where resources remain limited, according to the authors.
Early in the pandemic, institutions struggled to get their hands on real-time reverse transcription polymerase chain reaction (RT-PCR) tests for COVID-19, which led to heavy reliance on imaging for diagnosis. Hospitals in Italy and Britain adopted chest x-ray as a first-line screening tool, citing it as a cheap and portable alternative to CT, as it eliminated the need to sanitize CT rooms and after patient scanning.
In the U.S., guidelines recommended the use of chest radiography for monitoring rather than diagnosis of COVID-19. Yet chest x-ray remains frequently used to screen for disease severity and patient triage pending the availability of final RT-PCR test results, the authors wrote.
Diagnostic difficulty has only increased as the COVID-19 pandemic has evolved, and in this context, the researchers aimed to better delineate emergency radiologist performance on U.S. data.
The researchers culled RT-PCR results, clinical history, and chest x-rays of all patients tested between March 2020 and June 2020 across four academic medical centers. They included 984 RT-PCR-positive and 1,043 RT-PCR-negative radiographs. One hundred cases were read by all radiologists and 1,927 cases were read by two radiologists.
Each radiologist chose the single best label per case out of the following: Normal, COVID-19, Other -- Infectious, Other -- Noninfectious, Nondiagnostic, or Endotracheal Tube. Cases labeled with endotracheal tube (246) or nondiagnostic (54) were excluded, and the remaining cases were analyzed for label distribution, clinical history, and interreader agreement.
(A) RT-PCR-negative case in which both readers agreed the case was normal. (B) RT-PCR-negative case in which readers disagreed, with one reader labeling COVID-19 and the other labeling Other - Infectious. (C) RT-PCR-positive case in which both readers agreed on COVID-19. (D) RT-PCR-positive case in which readers disagreed, with one reader labeling COVID-19 and the other labeling Other - Noninfectious. Image courtesy of Clinical Imaging.A total of 1,727 radiographs (732 RT-PCR positive, 995 RT-PCR negative) were included from 1,594 patients (51.2% male, 48.8% female, age 59 ± 19 years).
For 89 cases read by all readers, there was poor agreement for RT-PCR positive (Fleiss score 0.36) and negative (Fleiss score 0.46) exams, the researchers found. Agreement between two readers on 1,638 cases was 54.2% for RT-PCR-positive cases and 71.4% for negative cases.
Agreement was highest for RT-PCR-negative cases labeled as normal (50.4%, n = 479). In addition, reader performance did not improve with clinical history or time between chest x-ray and RT-PCR results, according to the findings.
"Emergency radiologists have nonspecific performance for interpreting COVID-19 on [chest x-ray] with a high rate of disagreement," the researchers stated.
Overall, the results are consistent with the known literature on the diagnostic capabilities of chest x-ray for the diagnosis of COVID-19 pneumonia, the researchers stated. They also reinforce expert panel recommendations against routine use of chest radiography as a COVID-19 screening test, instead highlighting it as a tool to monitor disease severity, they wrote.
The authors noted study limitations, namely that they were unable to classify patients by disease severity using available data and that it is possible that stratifying radiologists' performance by disease severity could demonstrate improved performance for more severe cases.
In conclusion, the authors speculated on the implications of their findings in artificial intelligence (AI) research using chest x-ray datasets from COVID-19 patients. They stated they plan to release the annotated chest x-ray dataset from their study to aid in such evaluations.
"As image-based algorithm development increases for COVID-19 diagnosis, we urge caution in interpreting the results of these algorithms and their comparison to radiologist performance," they wrote.
