Thursday, November 30 | 9:30 a.m. - 9:40 a.m. | R3-SSIN07-1 | N227B
In this presentation, researchers will reveal how they were able to improve AI algorithm performance by utilizing local data.
During this Thursday morning session, neuroradiologist Stuart R. Pomerantz, MD, of Massachusetts General Hospital, will present a novel method for calibrating AI algorithm performance to “local conditions” and the results of using it on an intracranial hemorrhage (ICH) CT detection algorithm previously developed at his institution. The calibrated model provided a formal positive predictive value (PPV) guarantee of at least 85% and negative predictive value (NPV) of at least 95%, with only 7% of data points abstained.
The study showed that conformal uncertainty quantification effectively calibrated the AI model to a set of patients several years removed in time from the initial algorithm training, enabling high-confidence PPV and NPV compared with a model that cannot perform abstention. To minimize typical receiver operating characteristic tradeoffs, the methodology optimized both PPV and NPV by allowing abstention on cases for which the model had high uncertainty as determined by a conformal prediction statistical analysis using a hold-out set of recent cases for ex-post-facto calibration.
Conformal methodology was applied to the results of their ICH detection algorithm on consecutive CT exams (9,122 exams over a six-month period) and compared with ground truth from AI-naïve final reports. A deep-learning feature extractor was used on the array of per-image probabilities of each case with decision-tree training for probabilistic binary classification, according to Pomerantz.
By providing rigorous and mathematically provable guarantees for high accuracy on the local population, the method can improve confidence in AI system reliability across diverse settings, time periods, and use-case targets in the radiology workflow pipeline, according to Pomerantz.
As Pomerantz will explain, conformal uncertainty quantification provides a straightforward nonheuristic approach to calibrating an AI algorithm's performance to local conditions. The novel method presented could enhance prediction confidence and promise better decision-making and risk management in one own's setting.
