SAN FRANCISCO - One thing seems certain as lung computer-aided detection (CAD) systems move increasingly into clinical use: radiologists find more potentially malignant nodules with CAD than without it. But could CAD also be a case of too much of a good thing?
As radiologists are bombarded each year with more studies, more nodules, and more thin-section MDCT data that actually make all those lung lesions harder to see, the benefits of CAD seem increasingly obvious. Or are they?
Could too much of a good thing be worse than nothing? Preliminary research by Dr. Geoffrey Rubin and colleagues from Stanford University in Stanford, CA, suggests that after a while at the workstation, radiologists tend to accept increasingly dubious CAD findings.
The group's previous study showed that CAD finds more nodules than radiologists do, but that multiple readers still find nodules that CAD and the other readers miss.
"The literature is replete with examples of radiologists' folly in our presumed ability to detect lung nodules, and performance can be strikingly poor in a diverse set of circumstances," Rubin said on Thursday at Stanford's International Symposium on Multidetector-Row CT. "So it's important to bear in mind we can benefit from aids to detecting lung nodules."
There are two principal ways to do that: Lung CAD can be deployed as part of a dedicated CT screening program with a great potential for saving lives. Still, lung cancer screening with CAD -- and CT lung cancer screening itself -- remains controversial, with benefits as yet unproven.
The second CAD approach is easier to justify: Use lung CAD as a second reader on all routine diagnostic CT scans to see if it might pick up an important finding on a scan that was already medically indicated for another purpose. Or CAD could be used for primary review.
"The problem with using it for primary review is that if we turn the CAD on first and look at all the nodules we're probably less inclined to search for any nodules the CAD might have missed," Rubin said. "So in that setting CAD really must detect everything, and I don't think we've yet seen a detector that can do that confidently. So right now the more conservative approach, and the one the FDA is more likely to embrace short-term, is one where CAD assists a radiologist after they have done a primary reading."
The team's research on lung CAD (SNO-CAD, developed at Stanford) and radiologists' interactions with CAD systems has yielded some interesting results. In a study published last year, 20 thin-section chest CT scans were read independently by three readers, then CAD, and then by two thoracic radiologists who determined the reference standards after reviewing all the detections.
CAD found more nodules than any of the readers, but each reader also found nodules not seen by the CAD or the other readers. "There were true-positive nodules that were uniquely detected by each of the three readers, so they all had something to bring to the table," Rubin said.
The mean sensitivity for individual readings was 50% (range, 41% to 60%), while double reading increased the sensitivity to 63% (range, 56% to 67%). With CAD the mean sensitivity was increased to 76% (range, 73% to 78%) -- making a reader with CAD better than any two readers together (Radiology, January 2005, Vol. 234:1, pp. 274-283).
"What was particularly interesting is that the amount of time the reader spent on reading the average case was quite variable ... and didn't necessarily correlate with their relative performance," Rubin said.
In addition, the Radiology study was based on the assumption that radiologists would accept all true-positive CAD findings and reject true negatives. Which has led to a new avenue of research.
In the group's latest study, Rubin, along with his co-investigators David Paik, PhD, Dr. Justus Roos, and Sandy Napel, Ph.D., examined whether this assumption is true. The datasets, three radiologist readers, and CAD system used in the study were identical to those employed in the 2005 Radiology study, while the retrospective readings for the two studies were separated by three years’ time.
The three radiologists were asked to sequentially assess the CAD detections that were ordered by a ranking of the CAD algorithm. "The more likely nodule based on CAD was presented first, and then the less likely, sequentially," Rubin said. The readers did a free search before unveiling the CAD results.
The study found that radiologists do not reject all false-positive detections. And that wasn't all; something funny happened on the way to the results. The results also showed that false positives rose sharply with increasing exposure to CAD.
"We developed a 'knee' in the curve -- in other words, each (radiologist) started to deviate toward higher false-positive acceptance with a reduction in sensitivity after some period of time," Rubin said.
On average, 90 seconds (range, 60-120 seconds) after the interaction with the CAD began, radiologists were more likely to accept false positives at a higher rate than they accepted true positives.
 |
| Chart plots the overall sensitivity of lung nodule detection against the number of false positives the three radiologist readers accepted incorrectly. At each diagnostic confidence level (5 = highest confidence), sensitivity increases slowed significantly an average of 90 seconds after interaction with CAD began, while the number of false positives accepted as true positives rose. Image courtesy of Dr. Geoffrey Rubin. |
"This is a very striking finding, one we hadn't really anticipated as we broke the data down on a per-radiologist basis," Rubin said. "We observed it in all three radiologists. We observed it in the different size thresholds as well."
At this point the team is seeking to better understand the morphology behind the false-positive markings, and assess whether radiologists can be trained to avoid the downhill spiral after a minute and a half with CAD.
Can CAD be too much of good thing?
"It appears as though ... radiologists given enough time will relax their standards when CAD is showing them lung nodules, and begin to accept false positives," Rubin said.
By Eric Barnes
AuntMinnie.com staff writer
June 16, 2006
Related Reading
CAD improves lung nodule detection, June 1, 2006
Part II: Automated CT lung nodule assessment advances, May 16, 2006
Part I: Automated CT lung nodule assessment advances, April 17, 2006
Trimming CT dose decreases lung CAD performance, January 12, 2006
Management strategies evolve as CT finds more lung nodules, August 19, 2005
Copyright © 2006 AuntMinnie.com
