Commercial AI devices that lack adequate clinical validation may pose risks for patient care – and unfortunately, that’s true for almost half of the tools cleared so far by the U.S. Food and Drug Administration (FDA), says a group at the University of North Carolina in Chapel Hill.
In an August 26 commentary in Nature Medicine, the group discussed an analysis they performed of AI device clearances to date and called on the FDA and AI developers to publish more clinical validation data and prioritize prospective studies.
“Although AI device manufacturers boast of the credibility of their technology with FDA authorization, clearance does not mean that the devices have been properly evaluated for clinical effectiveness using real patient data,” said first author Sammy Chouffani El Fassi, a medical student, in a news release.
While AI is booming, its implementation in medicine has raised concerns about patient harm, liability, patient privacy, device accuracy, scientific acceptability, and lack of explainability, otherwise known as the "black box" problem, according to the authors.
These concerns underscore the need for transparency in how AI technology is validated, they added.
To shed light on the issue, the researchers looked at 521 AI or machine-learning device authorizations. They found that 144 were retrospectively validated, 148 were prospectively validated, and 22 were validated using randomized controlled trials. Most notably, 226 of 521 (43%) lacked published clinical validation data.
Notably, the greatest number of authorizations (392, or 75%) were for radiology devices, the authors found. The most common device function was for use with PACS (123 devices, or 24%) and the vast majority (512, or 98.3%) of authorizations were for class II devices, which are of intermediate risk to patients and subject to special FDA controls, the authors wrote.
Furthermore, the researchers found that the latest draft guidance, published by the FDA in September 2023, does not clearly distinguish between different types of clinical validation studies in its recommendations to manufacturers.
“We shared our findings with directors at the FDA who oversee medical device regulation, and we expect our work will inform their regulatory decision-making,” Chouffani El Fassi said.
For its part, in an article published earlier this year in npj Digital Medicine, the FDA noted it is currently sketching out details to help improve transparency in AI product information.
Ultimately, Chouffani El Fassi and colleagues wrote that AI has practically limitless applications, ranging from auto-drafting patient messages in MyChart to improving tumor removal accuracy during breast cancer surgery. Yet to facilitate trust in the technology, FDA-cleared devices need to demonstrate clinical validity, they wrote.
“We also hope that our publication will inspire researchers and universities globally to conduct clinical validation studies on medical AI to improve the safety and effectiveness of these technologies,” Chouffani El Fassi added.
The full commentary can be found here.
