Lung CAD shows limited sensitivity gains, finds nodules radiologists missed

Dec 12, 2005

Two studies presented at the recent RSNA meeting in Chicago underscored the clinical utility of computer-aided detection (CAD) for finding significant lung nodules with CT. They also served to highlight the limitations of current-generation lung CAD systems, most of which are based on common shapes of solid nodules.

The development of faster, easier detection systems for suspicious lung lesions will be critical for the advancement of resource-efficient screening programs for smokers and former smokers. To date only two CT-based lung CAD applications, LungChecker CAD (R2 Technology, Sunnyvale, CA), and LUNG CAR 1.1 (Medicsight, London) have received FDA approval. But other companies -- including the major multimodality vendors -- have all been working on nodule detection schemes.

The studies presented at the RSNA show evaluated the LungCare investigational lung CAD system (Siemens Medical Solutions, Malvern, PA). The studies had different goals and methodologies, but both research groups found incremental gains in sensitivity for detecting significant nodules with CT using the prototype CAD system. One study suggests that radiologists may search for lesions less diligently when CAD is used concurrently.

Multicenter trial shows CAD benefits

In the first presentation, lung cancer expert Dr. David Naidich, professor of radiology at New York University (NYU) in New York City, discussed his group's multicenter, multireader trial to evaluate the prototype evaluation system.

Within the limitations of a retrospective study, the team aimed to evaluate the CAD system's performance as comprehensively as possible, comparing CAD to the findings of 17 experienced general radiologists reading thin-section multidetector-row CT, Naidich said. The cohort did not consist of screening patients; they had already been referred for suspicion of a lung nodule on CT or for staging.

"The goal was to create as difficult a template as possible for our readers," Naidich said.

The study's unique contribution and primary end point was lobar sensitivity, an effort to learn more about the clinical significance of found nodules than sensitivity and specificity alone could provide. The primary analysis was directed at detecting shifts in the multiple reader, multiple case receiver operating characteristic (MRMC ROC) curve (A_z) with the use of CAD versus no CAD. Secondary end points included sensitivity; per-patient results; size, location, and density of nodules; and their relationship to lung structures, Naidich said.

After excluding four patients, the group examined 196 consecutive chest CT cases referred for routine assessment of clinically or radiographically known or suspected solitary pulmonary nodules obtained from four institutions: the NYU Medical Center, the Cleveland Clinic Foundation in Ohio, the University of Maryland Medical Center in Baltimore, and the University of Pennsylvania.

All patients were imaged using four- or 16-detector-row scanners using thin-section imaging (1.25- or 1-mm slices reconstructed at 1 mm or 0.75 mm, respectively) with low-dose exams "specifically excluded" from the cohort, Naidich said.

The images were processed centrally and distributed to 17 board-certified general radiologists, who came to New York City for CAD training and then reading.

The investigators took care to separate the CAD results from the readers' impressions, "including the 'always-never rule' -- the always being that the first read had to be the clinical read, only followed by the CAD read, and there was never a time when the CAD markings would overlap those that were actually detected by the readers," Naidich said.

The 17 readers were also asked to document whether the nodule was "actionable" on a scale of 1 to 10. Their results were consolidated centrally and sent on to two panels consisting of five expert thoracic radiologist readers each, who confirmed the nodules' "actionability" by consensus of three of the five expert readers, further characterizing the nodules as calcified or noncalcified, and measuring them manually (the CAD's automated volumetric measurement feature was switched off for the study). The experts' findings constituted the reference standard for the study.

"The experts were required to evaluate the candidate nodules separately without knowledge of whether they were identified by the radiologist or the CAD," Naidich explained. "They were not asked to reread the CADS, only to look at the markings that were sent to them."

From a total of 3,752 candidate nodules identified by the general radiologists and CAD, 2,025 were subsequently validated as true positives by the expert readers, and 863 were deemed actionable.

In all, 181 of 196 patients had at least one true-positive nodule 3 mm or larger (the cutoff size). The expert readers deemed 65% of the nodules as actionable; only 15 patients had truly normal findings for a nodule prevalence of 92.3%, not surprising for high-risk population of long-term smokers and former smokers, and for CT data examined by so many readers, Naidich said.

All of the readers showed improvement with CAD. Compared to the primary analysis without it, CAD produced a 0.78% upward shift in sensitivity. The overall increased lobar sensitivity for nodules 5 mm and larger was 74% with CAD versus 68% without it. In all, 1,320 nodules were detected in 525 of 980 (54%) of potentially evaluable lobes. Approximately 40%, or 397 of 980 lobes, had at least one actionable nodule.

In aggregate, eight readers missed nine to 10 nodules, and 2.9% of nodules detected by CAD were not seen by any of the 17 radiologist readers. A mean of 2.3 false positives per patient was recorded, Naidich said.

The main limitation was the retrospective nature of the study, meaning that clinical correlation was limited, and that histological ground truth could be established to determine which detected nodules were actually cancer. In addition, the CAD algorithm was designed only for solid nodules; part-solid and ground-glass nodules were excluded. Still, CAD increased the detection of actionable nodules for all readers.

Concurrent CAD saves time but offers little else

In another study presented during the RSNA session, Dr. Florian Beyer and colleagues from the University Hospital of Muenster in Germany evaluated the Siemens LungCare package to determine its effect on sensitivity and reading time.

"Use of CAD as a second reader for chest CT is time-consuming, but increases radiologists' sensitivity for the detection of pulmonary nodules," the group wrote in its abstract. "Concurrent reading of radiologist and CAD -- that is, display of CAD results simultaneously to the primary reading by the radiologist -- was reported to save time, but the effect on sensitivity is unknown."

The study compared sensitivity and reading time when CAD was used simultaneously and as a second reader. Fifty MDCT scans representing randomly selected pathologies were examined by CAD as a concurrent reader, the authors wrote.

In all, 535 nodules were found, with 180 deemed actionable. The mean sensitivity for all four readers was 0.68 for reading without CAD, 0.68 for concurrent reading, and 0.75 for second reading. There was no difference between reading without CAD and concurrent reading (p = 0.47), but the differences between sensitivity of concurrent and second reading (p < 0.001) and between reading without CAD and second reading (p < 0.001) were highly significant, Beyer wrote in an e-mail to AuntMinnie.com.

Mean reading time was shorter for concurrent reading (274 seconds) than for reading without CAD (294 seconds), demonstrating a significant difference on a Student's paired t-test (p = 0.04). The second reading took significantly longer (337 seconds) than concurrent reading (p < 0.001) and reading without CAD (p< 0.001). Analysis of the detection of nodules smaller than 4 mm showed similar statistical significances.

There was no statistically significant difference between reading without CAD and the use of CAD as concurrent reader, Beyer wrote. This result is surprising at first, he added, but might be explained by two effects.

"The first effect is that CAD used as a concurrent reader leads to a less thorough analysis of the CT scans by the radiologist," a negative effect, Beyer wrote, that was demonstrated by the shorter reading time with concurrent CAD use (295 seconds without CAD versus 274 seconds with CAD versus 34 seconds using CAD as a second reader).

The second effect is that CAD detected additional nodules even when it was used as a concurrent reader. Both effects, when CAD was used as a concurrent reader, showed sensitivities similar to reading without CAD, Beyer added.

Concurrent use of CAD in primary reading may result in less careful examination by radiologists. In one session in which CAD was being used as a concurrent reader, LungCare software marked a nodule in the periphery of the apical lower lobe of the left lung (red circle). However, two radiologists missed the centrally located nodule located on the same slice (arrow), while only one radiologist missed the same nodule when CAD was used as a second reader. Image courtesy of Dr. Florian Beyer.

CAD can either speed up reading times or increase sensitivity for pulmonary nodule detection when used as a second reader -- but it cannot do both, the group concluded.

"We think when the highest sensitivity is needed, a CAD system should be used as a second reader," Beyer wrote.

By Eric Barnes
AuntMinnie.com staff writer
December 13, 2005

CAD hunts down missed lung lesions, September 15, 2005

Despite flaws, lung CAD earns second-reader role, September 5, 2005

Management strategies evolve as CT finds more lung nodules, August 19, 2005

CARS panel discusses the future of CAD, June 28, 2005