A free commercial artificial intelligence (AI) software application can perform comparably to radiologists in triaging suspected COVID-19 cases on chest radiographs, according to research published online May 8 in Radiology.
A team of researchers from the Netherlands and AI software developer Thirona reported that their software yielded an area under the curve (AUC) of 0.81 for detecting COVID-19 on a test of nearly 500 cases. It also performed similarly to six experienced radiologists.
"The tool is made available pro bono on the manufacturer's website, to be of benefit in public health surveillance and response systems worldwide and may provide support for radiologists and clinicians in chest radiograph assessment as part of a COVID-19 triage process," wrote the team of authors led by Keelin Murphy, PhD, of Radboud University Medical Center in Nijmegen.
Based on the previously released CAD4TB v6 deep-learning software for detecting tuberculosis on chest x-rays, CAD4COVID-Xray was developed and validated by Thirona using 24,678 chest x-ray images. It was then tested on 454 continuously acquired chest x-ray exams in patients suspected to have COVID-19 pneumonia at the Jeroen Bosch Hospital. Of these 454 patients, 223 had positive reverse transcription polymerase chain reaction (RT-PCR) assay findings for the SAR-CoV-2 virus, while 231 had negative results.
The test set was also independently and blindly evaluated by six readers, including a chest radiologist with five years of experience, a chest radiologist with six years of experience, a chest radiologist with 20 years of experience, a chest radiologist with more than 20 years of experience, a radiologist with 24 years of experience, and a chest radiologist with more than 30 years of experience.
In addition to providing an AUC of 0.81, the software also outperformed each reader at their highest possible sensitivities (p < 0.001). Only one reader was able to significantly outperform the AI system (p = 0.04) at the lowest sensitivity setting of 65%.
Frontal chest x-ray (left) of a 74-year-old male with positive RT-PCR test for SARS-COV-2 viral infection. The AI system's heat map (right) overlaid on the image showing the pneumonia-related features. The AI system score for this subject is 99.8. All images courtesy of Radiology.At an 80% sensitivity setting, the CAD4COVID-Xray had a positive predictive value (PPV) of 68% and a negative predictive value (NPV) of 81%, compared with a 58% PPV and 72% NPV achieved by the reader consensus. At a 75% sensitivity setting, the AI software produced 77% PPV and 76% NPV, compared with 72% PPV and 76% NPV for the reader consensus.
"The results achieved by the AI system compared to radiologist readings are noteworthy given the fact that the presentation of COVID-19 pneumonia on [chest x-ray] can be highly variable ranging from peripheral opacifications only to diffuse opacifications making differentiation from other diseases challenging," the authors wrote.
A larger training set of radiographs is needed to improve the performance of the algorithm, according to the researchers. Incorporating clinical and laboratory findings could also help.
"In future work, the role of AI in management or triage of patients in the COVID-19 pandemic should be investigated, taking all related patient information and the experience level of the healthcare professionals interpreting the radiographs into account," the authors wrote.
