Backed by a study that shows high sensitivity and specificity in patients, a new computer-aided detection (CAD) scheme for virtual colonoscopy (also known as CT colonography or CTC) was approved last week by the U.S. Food and Drug Administration (FDA).
The 510(k) clearance granted to VeraLook from iCAD of Nashua, NH, is a rare one for the FDA, which promised a year ago to speed up its near-moribund approval process for computer-aided detection products.
Used as a second reader, VeraLook identifies polyps in images produced from a virtual colonoscopy study, then highlights its findings after the radiologist's initial review to help identify additional potential polyps that the human reader may have missed.
VeraLook has been commercially available in Europe and Canada since early 2009 and is expected to be available in the U.S. "very soon," said Robert Van Uitert, PhD, director of product management for iCAD. "VeraLook will be packaged and sold with 3D visualization workstations, and iCAD is in various stages of integration and commercial relationships with each of the major players in the CTC review workstation market."
Underpinning the approval was a study led by Abraham Dachman, MD, and colleagues from several centers in the U.S.
"As with breast and prostate cancer detection, computer-aided detection for virtual colonoscopy will help us find more cancers at an earlier stage," Dachman said in a statement accompanying the FDA clearance.
The study of patients with colonoscopy-proven VC cases aimed to assess the effect of using CAD in second-read mode on readers' accuracy in interpreting CT colonography images. The results were recently published online in Radiology (July 28, 2010).
"Because many polyps are often visible in retrospect, CAD is expected to help readers improve sensitivity," Dachman and colleagues wrote in Radiology. "In standalone studies, CAD has been shown to be sensitive for detection of polyps, but reader studies are critical to demonstrate the practical value of CAD because readers may accept or reject CAD marks, and there is a potential for sensitivity and specificity to improve or deteriorate."
Earlier studies of virtual colonoscopy CAD software were all limited -- by their small number of patients, small number of readers, or focus on a narrow question such as cost-effectiveness. The present study, which included 19 experienced readers, delivered a statistically significant improvement in overall reader performance, Dachman told AuntMinnie.com.
Dachman, a professor of radiology and director of abdominal and CT imaging at the University of Chicago, led the study in cooperation with several colleagues including Nancy Obuchowski, PhD, from the University of Chicago; Jeffrey Hoffmeister, MD, from the Cleveland Clinic; J. Louis Hinshaw, MD, from the University of Wisconsin, Madison; Michael Frew, MD, from the Walter Reed Army Medical Center in Bethesda, MD; and Ronald Summers, MD, PhD, and Bruce Hillman, MD, from the National Institutes of Health in Bethesda, MD.
Dachman disclosed that he is a paid consultant to iCAD, but he also said that he owns no stock in the firm, is not a board member, and had full control of the study data.
Two studies of effectiveness
The CAD software was first "trained" on a separate cohort of 437 cases and then tested on 355 cases (184 with positive findings and 171 with negative findings). In this standalone study, CAD was 94.4% sensitive for patients with at least one polyp ≥ 10 mm and 90.2% sensitive for patients with small polyps between 6 and 10 mm, the authors reported. The average number of false marks for all cases with positive findings was 5.07, and for negative cases it was 4.08.
Next, a cohort of 100 colonoscopy-proven cases was used: In 52 patients with findings positive for polyps, 74 polyps of 6 mm or larger were found in 65 colonic segments; in 48 patients with findings negative for polyps, no polyps were observed.
"The 100 cases were from screening CTC examinations, but were chosen to enrich the number of patients with polyps such that the ratio of small to large polyps was 3:2," Dachman wrote in an e-mail to AuntMinnie.com.
The 19 readers interpreted each case at two different times, both with and without CAD assistance. The study team assessed the effects of CAD in segment-level and patient-level receiver operator characteristics (ROC) curve analyses.
The final sample of 52 positive patients included 35 with polyps ≥ 6 mm and no polyps ≥ 10 mm, as well as 17 patients with polyps ≥ 10 mm (four with one and one with two synchronous 6- to 9-mm polyps) and 48 negative patients.
The study team measured several performance parameters including per patient and per polyp for the overall findings, for small (6- to 9-mm) polyps, for polyps 10 mm or larger, for adenomas, and for polyps of various morphologies, the authors wrote. The statistical analysis employed segment-level ROC curves.
"There is no single standard metric for comparing results in clinical CAD trials, which is why it is important to present the results by using several different metrics," Dachman and colleagues wrote in Radiology. "There are results of remarkably few similar studies about the use of CAD with which to compare our results, partly because most clinical studies about CAD included much smaller cohorts of patients and readers and methods that focused on a specific question, such as CAD reading mode, CAD for training, or use in different or special viewing methods."
The polyps ranged in size from 6 mm to 20 mm. Among the 52 patients with polyps, there were a total of 74 polyps in 65 colon segments. Fifty-three of the polyps were sessile, 13 were pedunculated, and eight were flat. There were 36 patients with adenomatous polyps with 47 adenomatous polyps in 44 colon segments. Thirty-two of the adenomatous polyps were small (6-9 mm) and 15 were large (≥ 10mm), the group reported.
The VeraLook CAD scheme was used in second-reader mode, meaning that the study was read without the aid of CAD before the CAD marks were revealed.
Multiple readers
Nineteen board-certified radiologists were recruited to test the software, including 11 from academic environments and eight from community practice, Dachman said. The readers had been interpreting 20 to 300 cases per year in clinical practice prior to participating in the research and, thus, represent adequately trained readers, with varying degrees of experience.
The methodology was rigorous in its definition of the standard of truth compared to optical colonoscopy, Dachman said. For a finding to be counted, a panel of experts had to agree on the polyp location, size, and pixel-by-pixel painting of each polyp compared to the CAD marking.
Readers' sensitivities and specificities were estimated using confidence scores of > 51 as positive and scores < 51 as negative; the readers were aware of this cutoff point. For each reader, sensitivity and specificity were calculated with and without CAD at the polyp, colonic segment, and patient levels. C-RADS scores without and with CAD were also compared as a means of assessing the impact of using CAD on patient management, the authors wrote.
According to the results, 13 (68%) of 19 readers demonstrated higher accuracy using CAD, as measured with the segment-level area under the ROC curve (AUC). The readers' average segment-level AUC with CAD (0.758) was significantly greater (p = 0.15) than the average AUC in the unassisted read (p = 0.737).
|
||||||||||||||||||||
| Sensitivity was higher per segment, per patient, and per polyp with CAD versus unassisted reads. | ||||||||||||||||||||
Average reader sensitivity also improved with CAD by more than 0.08 for small adenomas. Fifteen of the 19 readers showed improvement in sensitivity for detecting adenomas with improvement as high as 0.21. Use of CAD reduced specificity of readers by 0.025 (p = 0.050).
 |
| Pedunculated polyp is detected by VeraLook CAD product in 2D (below) and 3D virtual colonoscopy (bottom) images. |
 |
 |
CAD-aided readers
The use of CAD resulted in a statistically significant improvement in overall reader performance, Dachman said.
"Some of the key metrics to take away from this study include the readers' average sensitivity for detecting polyps ≥ 6 mm, which improved by 11.8%, and for polyps 6-9 mm, which improved by 19.9%," Dachman said. "The drop in specificity by only 2.7% and the increase in reading time by an average of 4.5 minutes is consistent with prior small CAD trials and still maintains the benefit of the CAD read."
The use of CAD always reduces specificity to some degree because some polyp candidates marked by the CAD system can't be distinguished from true polyps even by expert readers, the authors noted. In the case of this study, the per-patient drop in specificity, from 0.929 to 0.904, with CAD is lower than previous studies, and it "suggests that the CAD system we tested has a good set of classifier features to reduce false-positive marks," the team wrote.
"Improvement in ROC performance is an indicator that the trade-off in specificity is worthwhile," they wrote. "Since 13 (68%) of 19 readers demonstrated higher accuracy, as measured by the segment-level AUC, with a CAD-assisted read than with an unassisted read (three of 19 reaching significance), the CAD system is shown to be helpful, albeit not all readers will see a large change."
The results would be strengthened with use of a larger cohort, and with the inclusion of more patients without polyps, the authors noted. Another limitation was the lack of testing for the readers, although they were experienced, potentially contributing to low performance in sensitivity for small polyps. Sensitivity for small polyps was lower than in previous studies, suggesting the need for a thorough review of the causes.
In patients with polyps, reader assessment changed from false negative to true positive in 14.14% (95% confidence interval: 10.0%-18.7%) and from true positive to false negative in 7.9% (95% confidence interval: 4.9%-11.9%) with use of CAD. The estimated net gain in true positive assessment was 6.2%.
"The use of CAD results in a significant improvement in overall reader performance," Dachman and colleagues wrote. "The CAD system improves reader sensitivity when sensitivity is measured at the segment, patient, and polyp levels for small polyps and adenomas."
CAD improved the sensitivity for small adenomas the most, they wrote. Although the use of CAD reduced specificity and increased reading time, the reduction in specificity was small.
By Eric Barnes
AuntMinnie.com staff writer
August 10, 2010
Related Reading
iCAD gets FDA nod for VC CAD, August 5, 2010
Minimal-prep VC gets new boost from dual-energy CT, July 21, 2010
VC CAD application does well alone and as reader aid, February 15, 2010
New VC CAD algorithm reveals submucosal colon cancers, February 4, 2010
VC screening effective in Medicare-age screening cohort, January 27, 2010
Copyright © 2010 AuntMinnie.com
