Any organization deploying algorithms in high-impact areas needs to track the risks of stakeholder harm. The statement comes from the CEO of a risk consulting and algorithmic auditing firm and fellow authors in a story on auditing algorithmic risk in the MIT Sloan Management Review.
It's a compelling concept, auditing algorithmic risk; however, if we were truly ready to audit such risk, it may not have been readily apparent during a November 20 meeting of the U.S. Food and Drug Administration Digital Health Advisory Committee (DHAC), where stakeholders spent time in a YouTube broadcasted public meeting discussing regulation of generative AI and use of narrow AI tools.
"I looked far and wide and do not believe there is a single health system in the U.S. that is capable of validating an AI algorithm that's put into place in a clinical care system," stated FDA Commissioner Robert Califf, MD, to open the FDA DHAC meeting. Califf is no stranger to medical technology.
Prior to serving at the FDA, Califf was head of medical strategy and senior advisor at Alphabet Inc. He contributed to strategy and policy for Alphabet's health subsidiaries Verily Life Sciences and Google Health. As the FDA is currently undergoing a restructuring, Califf and others are watching what the Trump administration might dole out in changes.
AI everywhere
In drugs and biologics, almost every application has AI somewhere in the development process, Califf said during the DHAC meeting. Perhaps in contrast, AI for routine use in clinical care presents a range of complexities and variables that experts at the American College of Radiology (ACR), for example, are trying wrangle right now.
Bernardo Bizzo, MD, PhD, a diagnostic radiologist and senior director of Mass General Brigham's AI business, was among those who brought perspective on fieldwork toward AI quality and safety. Bizzo serves as associate chief science officer at the ACR Data Science Institute.
For the DHAC, Bizzo highlighted the ACR's national AI quality assurance program, ARCH-AI, which stands for ACR Recognized Center for Healthcare-AI. Under ARCH-AI is Assess-AI, Bizzo explained.
Assess-AI is a registry through the ACR that provides real-world monitoring of imaging-based AI models deployed in a clinical workflow. It enables the comparison of local performance metrics to national benchmarks and facilities with similar characteristics (region, facility type, trauma level, urban/rural, Bizzo explained.
Bizzo added that the program is currently piloting at 15 sites with four AI vendors and platforms engaged for two FDA-cleared clinical use cases, intracranial hemorrhage and pulmonary embolism on CT cases.
Narrow AI tools characterize the current state, Bizzo said, but the future state is generative AI. Members of the audience, though, had questions about determining clinically safe thresholds for use cases, and determining specific elements and metrics to monitor for safety and efficacy. Bizzo said those discussions are ongoing.
Episode 10
With all this as food for thought, we bring you this episode of Keeping Up With the Radiologists. Tessa Cook, MD, PhD, director of Penn Medicine's Center for Practice Transformation in Radiology which develops and pilots innovation initiatives, and Saurabh (Harry) Jha, MD, are joined by Hugh Harvey, MD, managing director of Hardian Health consultants.
Harvey is among those working to bring software as medical device (SaMD) and SaMD systems to market. The latest involves University College London (UCL) spin-out Guildford Street Laboratories and its blood-based Parkinson's disease test, PD Predict, which Hardian announced received FDA Breakthrough Device designation in November. The diagnostic test measures multiple biomarkers using machine-learning analysis.
Regulatory spectrum
"The question with any regulation is are we overregulating, are we under-regulating, what is an appropriate amount of regulation, and what is the purpose of regulation with medical devices and with artificial intelligence," Jha poses at the outset of the episode. Inside, there ensues some banter about Europe versus the U.S. and who might be doing better by its people, but that is for listeners to decide.
"Regulation is a spectrum between complete overregulation, i.e., banning things, to zero regulation to let chaos run rampant," Harvey said. "And I think anyone who works in the medical sector would agree some regulation is required for things that will impact patients and hospital care and general practice. If we can all agree on that basic assumption that there needs to be some checks and balances -- it can't be a complete free for all -- we can't outright ban new technology because we're scared of it."
Where do we want to put the needle on the pressure gauge of regulation? Listen to the panel discuss: CE marking versus FDA 510(k) and where getting regulatory clearance is more difficult, according to Harvey. Listen for the fundamental reason why there hasn't been a single Class-3 autonomous AI system regulatory approval yet. Listen for where the "cutting edge" is today in comparison to algorithm development a decade ago.
More impressions
{06:21:23} European AI legislation
{06:45:23} Decision to pilot an AI algorithm
{08:57:10} Evidence and rigor
{10:34:12} Asking for standards
{14:34:21} Transparency
{17:45:25} Algorithms are the new drugs
{20:20:22} Risks that present themselves through software
{22:14:19} Trust
{25:37:14} AI inputs keep changing
{32:53:20} Hospital quality management systems
{35:49:19} Managing AI in the real world
{37:28:03} AI vs radiologists study design
{41:23:15} Lacking large trials
{50:20:19} Promise of AI failing
{54:05:01} Efficiency gains
Guests:
Hugh Harvey, MD, is managing director of Hardian Health consultants, a U.K.-based clinical digital consultancy focused on leveraging technology into healthcare markets through clinical strategy, scientific validation, regulation, health economics, and intellectual property.
Hosts:
Saurabh (Harry) Jha, MD, is an associate professor of radiology at the Hospital of the University of Pennsylvania. Jha obtained a master’s degree in health policy research from the Leonard Davis Institute at the University of Pennsylvania. He earned his medical degree from the United Medical and Dental Schools of Guy’s, King’s, and St. Thomas’ Hospitals. Jha developed Value of Imaging, a set of radiology educational resources.
Tessa Cook, MD, PhD, is a cardiovascular radiologist and board-certified clinical informaticist at Penn Medicine. Cook serves as director of the 3-D and Advanced Imaging Laboratory at Penn Medicine and director of Penn Medicine's Center for Practice Transformation in Radiology which develops and pilots innovation initiatives looking for improving care delivery in radiology. That includes AI.
This episode of Keeping Up With the Radiologists is brought to you by AuntMinnie.com in collaboration with Penn Radiology. The series is also available on Spotify, YouTube Music, and Apple Podcasts. Check back for new episodes!
