The advantages of computer-aided detection (CAD) software can be sharply reduced if readers focus their attention primarily on areas of images with CAD marks, researchers from Brigham and Women's Hospital and Harvard Medical School have found.
In a study published in the October issue of Academic Radiology, the Boston researchers determined that readers did not search images as completely when using CAD as they did without the software. They spent much more time on areas that were marked by CAD, and less time searching other areas for other possible targets.
"In our experiment, the benefit for detection of marked targets was roughly balanced by the cost of unmarked targets, yielding no net benefit of our CAD even though our artificial CAD system was designed to be comparable to a good commercial system," lead author Trafton Drew, PhD, told AuntMinnie.com.
The researchers became interested in exploring CAD due to a number of recent findings, such as a study published in 2011 by Fenton and colleagues that suggested CAD might not improve overall performance in the clinic, Drew said.
"From a signal detection perspective, any time you add information to a task, you should enhance performance, yet evidence for benefits due to CAD is mixed," he said. "Even those studies that do find that CAD aids performance typically find effects that are smaller than we would predict."
In an attempt to understand what appears to be suboptimal performance when using CAD, the research team employed eye tracking that followed readers' eye movements when using the technology (Academic Radiology, October 2012, Vol. 19:10, pp. 1260-1267).
Finding a target
A total of 47 observers with an average age of 24.3 (range, 18-54) participated in two studies in which they were asked to search for a target, the letter T. The T targets were embedded in a field of "noise" that included letter Ls, which functioned as distractors. The observers' eye movements were monitored using the EyeLink 1000 tower system (SR Research) at a sampling rate of 1,000 Hz.
The observers were told to click on the T when they found one, and to click on an "absent" button if they couldn't find the letter. Half of the trials contained one target. The observers also provided a confidence rating at the conclusion of each trial.
Half of the observers in both studies completed the experiment without CAD, while the other half searched the same trials with the aid of the researcher's artificial CAD software that marked 75% of all targets and 10% of nontargets. Those using CAD were instructed to use the marks to help them find the target, but they were also told that CAD would sometimes miss the target or mark a distractor, according to the authors.
In the first study, the CAD system's primary function was to tell observers where the target might be, while in the second study, CAD provided information about target identity. In the second study, the appearance of the T targets and L distractors was also manipulated to make the letters more similar to each other, according to the researchers. The opacity of the background noise was decreased, however, to make the items easier to find.
"Experiment 1 had high noise and low similarity between targets and distractors, whereas experiment 2 had lower noise and higher similarity between targets and distractors," the authors wrote. "Thus, the targets in experiment 1 were difficult to detect but easy to 'diagnose.' Here CAD would aid detection (CADe). The targets in experiment 2 were easy to detect and hard to identify. In this case, CAD would aid diagnosis (CADx)."
In the first experiment, the researchers found a significant enhancement of observer sensitivity (t(22) = 4.74, p < 0.001) from CAD. But these gains came at a cost: Targets not marked by the CAD system were more often missed (t(22) = 7.02, p < 0.001) than equivalent targets in blocks of the experiment that did not use CAD.
Meanwhile, the second experiment found no behavioral benefit, but also no significant cost regarding sensitivity to unmarked targets (t(22) = 0.6, p = NS).
Drawing attention away
The researchers also discovered that CAD produced reliable changes in eye movements in both experiments. A lower total percentage of the search area was examined by CAD observers (t(48) = 3.05, p < 0.005) than by the observers without CAD (t(50) = 7.31, p < 0.001).
 |
| Eye-tracking heat maps show that observers did not search images with CAD marks as completely as those without CAD. Image courtesy of Trafton Drew, PhD. |
The data suggest CAD marks can pull attention away from areas that are not marked by the CAD system, leading to incomplete search, Drew said.
"This sets up a troubling situation when the CAD system fails to mark a target," he said. "Now the observer is less likely to find the target than they would have been without CAD."
While first noting that the study was an artificial task, the researchers hope they have uncovered a basic effect that can be applied to CAD usage in the clinic, Drew said.
He believes the problem could be ameliorated if CAD is used as a "second reader," activated only after a thorough first read by the radiologist.
"Informal conversations with mammographers suggest that the usage of CAD varies widely, with some clinicians deploying the CAD system prior to a complete search," Drew said. "This may lead to significant underperformance with CAD."
