An artificial intelligence (AI) algorithm can't be relied upon -- yet -- to detect pulmonary embolism (PE) on CT pulmonary angiography (CTPA) in clinical practice, according to research from Germany published August 6 in RöFo.
Researchers led by Dr. Katharina Müller-Peltzer of Albert Ludwig University of Freiburg in Germany retrospectively assessed the performance of a commercial AI-based computer-aided detection (CAD) software application in over 1,000 CTPA exams for detecting PE. They found that the software missed over one-third of patients who had an embolism and also generated a significant number of false positives.
"Applied to a population at a large emergency room, the tested commercial computer-aided detection algorithm faced relevant performance challenges that need to be addressed in future development projects," the authors wrote.
The researchers evaluated the performance of the CAD software in a retrospective assessment of all CTPA studies acquired for suspicion of PE at the institution between January 1, 2017, and January 1, 2020.
All CTPA exams were performed on a Somatom Force dual-source CT scanner (Siemens Healthineers) with a gantry rotation time of 0.25 s/rotation, a pitch of 1.9, a collimation of 192 × 0.6 mm, and a slice thickness of 0.75 mm. The Imeron-400 (Bracco Imaging) contrast agent was used in 2014 and 2015, while Ultravist-370 (Bayer) was utilized in 2016 and 2017.
The automated image analysis was conducted using syngo.via software version VB20A (Siemens) and the CAD analysis was compared to gold-standard interpretation conducted in consensus by two radiologists with five and 15 years of emergency radiology experience, respectively. An embolism identified multiple times by the software and by the gold standard interpretation was considered to be a true-positive result, according to the researchers.
Gold-standard interpretations were detected in 182 of the 1,229 patients included in the study, with 106 of these having multiple embolisms. A total of 504 embolisms were reported.
The software produced 3,331 findings, of which only 258 (8%) were true-positive results. The remaining 3,073 (92%) results were false-positive findings, according to the researchers. An average of 2.25 false-positive findings were produced per study.
| Performance of AI software for detecting pulmonary embolism on CTPA | |
| Sensitivity (per patient) | 118/182 (64.8%) |
| Sensitivity (for all embolisms) | 258/504 (51.2%) |
| Sensitivity (embolisms on segmental level) | 118/219 (53.9%) |
| Sensitivity (embolisms on subsegmental level) | 116/234 (49.6%) |
| Sensitivity (embolisms on lobe level) | 24/51 (47.1%) |
| Specificity | 21.6% |
| Positive predictive value | 12.6% |
| Negative predictive value | 77.9% |
Delving further into the results, the researchers found no correlation between the number of false-positive findings and the exam contrast-to-noise ratio. The most common underlying reasons for false positives were soft tissue (61%) and pulmonary veins (24.1%).
"Evaluation of the commercial PE-CAD system based on a large patient population in an emergency setting showed a need to optimize the software, particularly in relation to false-negative findings," the authors wrote. "The study highlights the role of the system as supporting analysis software that cannot (yet) replace visual assessment by radiologists."
