CAD prototype improves VC for inexperienced readers

Apr 23, 2006

A pilot virtual colonoscopy study using mostly inexperienced readers found that computer-aided detection (CAD) significantly improved sensitivity for colorectal polyps while increasing the false-positive rate for some readers.

A total of 50 datasets from four centers in the U.S. and Europe were used in the multicenter study led by Dr. Mark Baker of the Cleveland Clinic Foundation in Ohio.

Of these, 20 cases were used to train the prototype CAD system. The remaining 30 datasets, which included 24 positive (at least one polyp greater than 5 mm) and six negative cases (total polyps 33), were used for testing purposes, randomly ordered into three groups of 10, according to Baker.

Most of the seven readers had no experience in VC before training on the investigational CAD system (ColonCAD, Siemens Medical Solutions, Malvern, PA), while three had limited experience. Baker was the most experienced, having read 150 cases previously, he said in a presentation at the 2005 RSNA meeting in Chicago.

Following multiple training sessions led by an application specialist, the readers examined the 30 test cases in random order over a four-week period, when they had time to do four to five cases in a sitting, Baker said. For the test, the trained readers evaluated the VC dataset, marking and scoring the polyps before and after CAD.

"The findings were located per segment and marked using either 2D or 3D," he said. "We assigned a score of 1-5 -- 1 being definitely negative and 5 definitely positive.... The CAD results were then presented to readers ... (and) potential polyps were identified and marked in yellow. If (CAD) corresponded to a position the physician had marked, the mark was discarded and recorded. If it was a new mark, (it was) assessed using the 1-5 scale."

The results showed that without CAD, per-patient polyp detection sensitivity ranged from 58.3% to 83.3% (mean 73.8%, standard error [SE] = 5.8%). With CAD, per-patient sensitivity rose to 75.0% to 91.7% (mean 84.5%, SE = 4.1%), demonstrating significant improvement (p = 0.012, 95% CI for difference [-18.3% to -3.1%]).

Without CAD, the false-positive rate per patient ranged from 0% to 83.3%, mean 35.7% (SE = 2.4%). With CAD the false-positive rate per patient was 0% to 83.3% (SE = 38.1%), mean 38.1% (SE = 2.5%) (p = 1.00).

Without CAD, the sensitivity of polyp detection per segment was 0.536 to 0.786, mean = 0.699 (SE = 0.072). With CAD, the sensitivity of polyp detection per segment was 0.714 to 0.893, mean = 0.806 (SE = 0.057) (significantly improved; p = 0.018, 95% CI for difference [-0.180, -0.034]).

"There was only one reader where the false-positive rate per patient went up," Baker said. For all other readers the difference was not statistically significant per segment.

Per-segment sensitivity for polyps without CAD was 53.6% to 78.6%, mean 69.9% (SE = 7.2%). With CAD per-segment sensitivity was 71.4% to 89.3%, mean 80.6% (SE = 5.7%), demonstrating significant improvement (p = 0.018, 95% CI for difference [-0.180, -0.034]).

"Sensitivity without CAD varied from 58% to about 83%," Baker said. Sensitivity with CAD varied from about 75% to 92%. None are statistically significant, but when you look at the overall clustered data with seven readers it goes from 74% to 85%, and this is statistically significant," he said.

The group had intended to use an ROC analysis instead of McNemar's test, but the lack of some evaluation scores (no one marked 2, and few marked 3, for example) rendered this method inadvisable, Baker said.

The CAD evaluation added about five minutes to the evaluations of the last 10 cases, and more for the first 20, showing that there is a learning curve that continues even after completing training, Baker said. "It looks like we get faster and better looking at CAD marks," he said.

"For the patient results, there is a significant improvement in reader sensitivity for CAD; false positives increased for one reader," he said. "For the per-segment data, there was an overall increase in sensitivity with a significant increase in the false-positive rate" (p = 0.004, 95% CI for difference [-0.026, -0.007]).

"CAD definitely improved reader sensitivity at the expense of some increase in false-positive rates in some readers," Baker concluded. "It does not add substantial time to interpretation. I think it's going to be an essential adjunct to primary interpretation, particularly in clinical practice with experienced readers."

The readers were not told of the high prevalence of positive cases in the test set, Baker said in response to a question from the moderator.

By Eric Barnes
AuntMinnie.com staff writer
April 24, 2006

Colon CAD helps expert readers, March 16, 2006

AuntMinnieTV: NIH researchers add CAD to virtual colonoscopy, February 3, 2006

VC CAD found equivalent to colonoscopy in screening population, December 21, 2005

Colon CAD: VC's extra eyes face new challenges, August 5, 2005