What's taking so long for artificial intelligence (AI) algorithms to move into routine clinical use in radiology? Several regulatory bottlenecks could be holding AI back, according to a September 19 talk at the Conference on Machine Intelligence in Medical Imaging (C-MIMI).
In a keynote presentation that shared the perspective of the U.S. Food and Drug Administration (FDA) on AI/machine-learning (ML) software-as-medical devices, Ravi Samala, PhD, highlighted four driving forces that have led to unique regulatory challenges for the agency:
- Emerging applications -- algorithms for identifying new pattern observations in human physiology, treatment response monitoring, and more
- Novel specialty areas for AI
- The unique nature of medical data -- characterized by a low disease prevalence and lack of or difficulty in obtaining truth data, wide variations in imaging technologies across manufacturers, and variations based on sex, age, ethnicity, imaging modality, and disease categories
- Advances in algorithms -- new techniques such as generative adversarial networks (GANs) for data generation, reinforcement learning, continuous learning, and transformers
"Based on these observations, our team has come up with five AI/ML gaps in the regulatory science research that could help the agency's mission of giving patients in the U.S. first access to software as medical devices," said Samala, a staff fellow of the FDA's Division of Imaging, Diagnostics, and Software Reliability (DIDSR) in the agency's Center for Devices and Radiological Health (CDRH) Office of Science and Engineering Laboratories.
Research gaps
These research gaps relate to the following:
- Data size
- New and multimodality data
- Computer-aided triage applications
- Quantitative imaging
- Adaptive algorithms
Data size issues include challenges associated with scarcity of medical imaging data, data augmentation techniques, transfer learning, and GANs, Samala said.
"There's a strong need to fundamentally understand the limitations of smaller datasets, as well as to develop techniques to maximize information and enhance AI/ML training," he said at the meeting, held by the Society for Imaging Informatics in Medicine (SIIM).
The FDA is working to address this problem with efforts such as its Virtual Imaging Clinical Trials for Regulatory Evaluation (VICTRE) project, for example, according to Samala.
New assessment paradigms
The second regulatory research gap relates to the development of new types of AI software that utilize multiple types of data sources, including radiology, physiology, pathology, patient demographics, and other electronic health record data, according to Samala. These new types of algorithms often require novel assessment paradigms -- including both clinical and nonclinical testing -- and also measurement of safety and effectiveness when used in new types of applications.
"Although most of the devices currently on the market are diagnostic in nature, we are seeing more and more devices that are prognostic -- trying to do treatment response prediction, risk assessment in therapy -- which requires different assessment metrics as well as different standards," he said.
The FDA is working to identify the evaluation approaches that can assess the performance of these types of devices, he said.
Computer-aided triage
Computer-aided triage has been one of the fastest growing areas of AI, with more than 30 AI software applications receiving clearance over the past two years, Samala said. There's a critical need, however, for estimation of time savings in the clinical environment for these types of algorithms.
Samala highlighted new research that has been accepted for presentation at the upcoming RSNA 2021 meeting. The study will show an evaluation method of clinical effectiveness based on querying theory for time-saving analysis of computer-aided triage and notification devices.
Quantitative imaging and radiomics represents another gap in regulatory science. There's a need for well-characterized quantitative features and for a clear reference standard, Samala said.
The FDA is also working on a number of projects in this area, including an effort involving the use of radiomics features to assess bone fragility.
Adaptive algorithms
Another active research topic relates to adaptive algorithms, i.e., continuously learning algorithms that change their behavior through a defined learning process. There's a need for a validation testing framework and a need to account for uncertainty in the reference standard, according to Samala.
"Developing a clear, science-based framework to develop this type of regulatory flexibility is a major challenge," he said.
The FDA is working on one project that utilizes online calibration for continuous learning systems, for example
"What we're trying to see is if there are methods that can be used to calibrate these continuous learning systems so that it can ... maintain a good risk/benefit profile," he said.
Challenges, opportunities
Several emerging applications and novel areas for AI continue to emerge, and this has, in turn, led to new classes of AI software categories such as computer-assisted triage and computer-assisted acquisition and optimization, Samala said. The number of AI applications involving risk assessment and patient prognosis are also expected to increase.
"And these new classes of devices require new performance assessment methods," he concluded. "That's creating new challenges as well as opportunities in the research space."
The FDA has several initiatives in the pipeline to address these AI challenges, but it also continues to rely heavily on research and contributions from communities such as SIIM to help the agency in its regulatory activities, Samala said.
