Radiologists can improve their assessment of indeterminate lung nodules on CT exams with help from artificial intelligence (AI)-based computer-aided diagnosis (CADx) software, according to research published online May 24 in Radiology.
In a retrospective study, a multinational team of researchers led by first author Dr. Roger Yim and senior author Dr. Anil Vachani of the University of Pennsylvania retrospectively evaluated the performance of six radiologists and six pulmonologists in assessing the malignancy risk of indeterminate pulmonary nodules on 300 cases.
They found that the use of a commercial AI-based application led to significantly improved accuracy and interobserver agreement for all readers.
"The level of improvement suggests that this tool has the potential to change how we judge cancer versus benign and hopefully improve how we manage patients," said Vachani in a statement from the RSNA.
Although AI-based CADx models have shown in prior studies that they can discriminate pulmonary nodules more accurately than both the commonly used Brock and Mayo risk prediction models, the practical clinical utility of these tools depends on their capability to provide additional information to clinicians during interpretation of chest CT exams, according to the researchers.
Following up on a successful pilot study, the researchers sought to perform a multicenter clinical validation of the Virtual Nodule Clinic 2.0 software (Optellum). They collected 300 screening and diagnostic chest CT scans -- each containing one 5 mm to 30 mm indeterminate pulmonary nodule -- from two institutions in the U.S. and four in the U.K.
Examples of axial CT images of pulmonary nodules included in the multicenter reader study. (A) Malignant nodule with a lung cancer prediction score of 10 on the AI software. (B) Benign nodule with a lung cancer prediction score of 2. Images and caption courtesy of the RSNA.On each of the 300 randomly ordered cases, the readers first estimated malignancy risk of nodules on a 100-point scale and selected a management option. Next, they viewed the AI software's risk malignancy score on a scale of 1 (lowest risk) to 10 (highest risk) and were asked to provide an updated risk estimate and management recommendation.
Each of the readers experienced a significant improvement in their area under the curve (AUC) for estimating malignancy risk.
| Impact of AI-based CADx on reader assessment of indeterminate pulmonary nodules | ||
| Readers without AI-based CADx | Readers with CADx | |
| Area under the curve (AUC) | 0.82 | 0.89 |
| Average sensitivity at 5% malignancy risk threshold | 94.1% | 97.9% |
| Average sensitivity at 65% malignancy risk threshold | 52.3% | 63.1% |
"The average sensitivity and specificity of pulmonologists and radiologists improved with CAD at both the very low (5%) and high (65%) malignancy risk thresholds, suggesting that CAD may have a meaningful impact on pulmonary nodule management decisions," the authors wrote.
Furthermore, interobserver agreement also improved significantly, increasing from a Fleiss kappa statistic of 0.50 to 0.71 for the 5% malignancy risk category and from 0.54 in the 65% malignancy risk category to 0.71. In addition, overall interobserver agreement for management recommendations was also significantly enhanced from the use of AI, climbing from a Fleiss κ of 0.44 without assistance to reach 0.52 with use of the software.
All differences were statistically significant. The researchers are now designing prospective trials for the software.
"These results are crucial for bringing AI-based tools closer to clinical implementation for risk stratification and patient management," said Dr. Masahiro Yanagawa, PhD, of Osaka University in Japan in an accompanying editorial. "Prospective research is necessary to validate these results in clinical practice."
