The combination of multiple artificial intelligence (AI) algorithms working alongside a human reader showed the best performance for interpreting screening mammograms in a new study, published online March 2 in JAMA Network Open.
The findings further confirm the promise of AI for improving mammography screening, wrote a team led by Thomas Schaffter, PhD, of Sage Bionetworks in Seattle.
"While no single AI algorithm outperformed U.S. community radiologist benchmarks, an ensemble of AI algorithms combined with single-radiologist assessment was associated with improved overall mammography performance," the group wrote.
Since mammography screening interpretation can be subjective, researchers are exploring whether AI can improve interpretation accuracy. The researchers sought to evaluate whether AI could overcome human limitations in reading mammograms by hosting the Digital Mammography DREAM Challenge between November 2016 and November 2017, a competition to assess whether AI algorithms can perform comparably to, or even outperform, radiologists.
More than 1,000 competitors participated. Data included 144,231 screening mammograms acquired at Kaiser Permanente Washington (of which 952 indicated cancer) and 166,578 examinations acquired at the Karolinska Institute in Stockholm, Sweden (of which 780 indicated cancer).
Schaffter's group assessed the accuracy for breast cancer detection of the AI algorithms submitted to the contest using area under the curve (AUC) and algorithm specificity, and they compared this performance with radiologists' specificity (radiologist sensitivity was set at a benchmark of 85.9% for the U.S. and 83.9% for Sweden).
The group found that, although no one AI algorithm outperformed community radiologist benchmarks -- and including clinical data and prior mammograms didn't improve AI performance -- combining the best-performing AI algorithms with radiologist assessment increased AUC and specificity compared with radiologists alone.
| Comparison of AI, radiologist, and AI plus radiologist mammogram assessment | |||
| Performance measure | Top-performing single AI algorithm | Radiologist assessment | Top-performing AI algorithms plus radiologist assessment |
| AUC | |||
| U.S. | 0.858 | 0.895 | 0.942 |
| Sweden | 0.903 | 0.923 | 0.942 |
| Specificity | |||
| U.S. | 66.2% | 90.5% | 92% |
| Sweden | 88% | 96.7% | 98.5% |
"While no single AI algorithm outperformed radiologists, an ensemble of AI algorithms combined with radiologist assessment in a single-reader screening environment improved overall accuracy," the group wrote.
Although AI for mammography interpretation shows promise, radiologists remain crucial -- especially since the technology continues to need development in terms of how it will be implemented, Dr. Claudia Mello-Thoms, PhD, from the University of Iowa in Iowa City wrote in an accompanying editorial.
"This has been the largest effort to date ... to develop AI that could aid radiologists in the reading of screening mammograms. For the most part, it showed that AI is not there yet," she wrote. "Studies like this strongly suggest that radiologists will be masters of their domain for quite some time, as the task of image interpretation is significantly more complex than radiologists get credit for."
