Despite flaws, lung CAD earns second-reader role

Sep 4, 2005

In a new study from Canada, a popular lung CAD application proved its value as a second reader of low-dose thoracic CT studies, detecting lung nodules that had been missed in a double reading by radiologists. Yet the application also produced large numbers of false positives, and some troubling false negatives.

"We think lung cancer screening is probably one of the best indications for CAD," said Dr. Heidi Roberts, associate professor of radiology at the University of Toronto in Ontario. "You have very thin slices and a lot of slices -- more than 250 on average in our patients. And it's a very repetitive task because you're always looking for the same thing -- lung nodules in otherwise healthy lungs."

Consistency is a key benefit of automated detection, of course. Hours after a human reader has begun to tire, CAD can still tear through data like the Energizer Bunny. The Toronto group's study also reaffirms the hypothesis that CAD, like the bunny, marches to its own drummer, detecting a different set of nodules than the ones the radiologists found.

At the same time the application found only one of five lung cancers, all of which were eventually detected by the radiologists. The CAD misses included a small cancer of a size and shape the algorithm was designed to detect, according to Roberts and an accompanying abstract co-written with her colleagues Dr. Demetris Patsios, Michael Kucharczyk, Dr. Narinder Paul, and Dr. Timothy Roberts.

The school's University Health Network in Toronto is a participant in the International Early Lung Cancer Action Program (I-ELCAP) trial, which screens smokers and former smokers age 55 and older with a smoking history of at least 10 pack-years. Roberts presented the team's research at the 2005 Computer Assisted Radiology and Surgery (CARS) meeting in Berlin.

The retrospective study examined CT data from 250 consecutive lung screening participants, whose CT studies had been acquired on a multislice CT scanner (GE Healthcare, Chalfont St. Giles, U.K.) at 1.25-mm collimation, 60 mAs, and 140 kVp.

Each scan was read independently by two experienced thoracic radiologists, whose consensus reading served as the reference standard. Then the data was read by ImageChecker (R2 Technology, Sunnyvale, CA), the only lung CAD application to date that has received premarket approval from the Food and Drug Administration.

ImageChecker looks for focal densities of approximately spherical shape, with borders that are smooth, lobulated, or spiculated, and have a CT density approximating that of soft tissue. Based on this information, the system highlights all findings that exceed a defined likelihood threshold, according to the researchers.

"We restricted this study to nodules that were 5 mm and larger for several reasons," Roberts said. "The software vendor says the algorithm does work for nodules 4 mm and larger, but clinical radiologists are more interested in what's clinically important."

And for that the group relied on a recent study showing that pulmonary nodules 5 mm or larger in diameter were far likelier to be malignant than smaller masses (Henschke et al, Radiology, April 2004, Vol. 231:1, pp. 164-168).

Following review by the radiologists and the software, CAD findings were divided into four groups:

True-positive entities (TPEs) -- solid densities that were correctly defined by the CAD but rejected as nodules based on overall appearance
True-positive nodules (TPNs) -- solid nodules found by CAD and confirmed by the radiologists upon reassessment, whether or not it was identified on the initial double reading
False-positive findings (FPs) -- entities identified by CAD that were subsequently dismissed
False-negative findings (FNs) -- nodules found in the initial double reading, but missed by CAD

"There were marks that really looked like nodules," Roberts said of the unusual TPE classification that was applied to 41 findings. "But our radiologists know a little more. It's actually not a nodule. And there is more to the diagnosis of a nodule than this solid spherical shape. But to have it called false-positive also would not have been quite fair."

A 5-mm nodule in right middle lobe, representing a true-positive CAD finding. All images courtesy of Dr. Heidi Roberts.

The TPE designation was applied mostly to pleural densities that had to be deemed pleural plaques, pleural or subpleural thickening, fibrosis, or atelectases, but were correctly defined by CAD as focal nodular densities, she said.

Above: A 7-mm nodule in right middle lobe, representing a true-positive CAD finding. Below: A 5-mm nodule in the lower right lobe, representing a true-positive CAD finding.

In some cases of interstitial lung disease, for example, "it's definitely a nodule, but it's not a nodule that needs to be followed up in a screening study," Roberts said.

In all, 64,389 slices were analyzed in the study, representing an average of 258 slices for each patient. CAD marked 1,136 findings, of which 196 were eliminated because they were outside the lungs. An additional 29 solid and 27 part-solid nodules, and 124 calcified nodules, were eliminated because they were outside the algorithm's nodule detection criteria, according to Roberts.

Oblong nodule of 9 mm in left upper lobe was a true-positive CAD finding.

"We had to exclude a few," she said. "Some had to be redefined by the manufacturer. We should definitely exclude everything that's calcified, and the algorithm has not been trained to (reject) calcified nodules." Nonsolid and part-solid nodules were excluded, as were findings outside the lungs.

Solid nodule of 6 mm in right lower lobe was a false negative, found by the radiologist but not marked by CAD.

The radiologists had found 83/104 nodules that were detected by the combined CAD and radiologist reading, representing a sensitivity of 80% for the original double radiologist reading. CAD had marked a total of 856 entries, including 117 (14%) that were true positives (TPNs + TPEs). Seventy-six were true-positive nodules (TPNs), and 739 (86%) were false positives (FPs).

Twenty-eight nodules (27%) were missed by the CAD (FNs) but found by the radiologists in the initial double read, for a CAD sensitivity (TPNs) of 73% (76/104), Roberts said.

The false-positive detections represented atelectases (35%), normal lung fissures (14%), normal vessels (13%), pleural abnormalities (11%), mediastinal vessels or structures (9%), apical fibrosis (6%), osteophytes (2%), pericardial fat (2%), and mucoid impaction (1%).

Most (13/28) false negatives were pleura-based, and were more prevalent in the right lung (n = 19) than the left lung (n = 9), with an average size of 9 mm.

Five cancers were found in the cohort of 250 patients, including one detected by CAD, four bronchoalveolar-cell carcinomas excluded from the analysis due to their nonsolid consistency, and one pleura-based carcinoma approximately 10 mm that CAD missed.

"We think that lung nodule detection can be improved (with the use of CAD) in this particular subset of a lung cancer screening population," Roberts concluded. "There are a lot of false positives, but it's very easy to dismiss (them) just by scrolling though a couple of times."

On the other hand, she said, the prevalence of false negatives is cause for concern. "Because if I as a radiologist have to read the study again completely to make sure that no nodule is missed, the utility of the CAD turns out to be much lower than if I just have to look at a few entities that turn out to be false positives."

The good news is that ImageChecker found nodules that would otherwise have been overlooked. "The ones that were missed by the radiologist happened to have been found by the CAD, which makes it an important second reader," she said.

By Eric Barnes
AuntMinnie.com staff writer
September 5, 2005

Part II: Computer-aided detection identifying new targets, July 15, 2005

CARS panel discusses the future of CAD, June 28, 2005

3D diminution index spots lung nodules attached to blood vessels, June 27, 2005