An artificial intelligence (AI) algorithm trained on a large, multinational dataset can be highly accurate for identifying COVID-19 pneumonia on chest CT scans, according to research published online August 14 in Nature Communications. It can also be highly specific for non-COVID-19-related pneumonias.
Researchers from the U.S. National Institutes of Health (NIH) in Bethesda, MD, graphics processing unit technology developer Nvidia, and several other institutions around the world trained a series of deep-learning algorithms on chest CT scans that included COVID-19 cases from four hospitals in three countries. In testing on an independent and diverse dataset, their best-performing model yielded over 90% accuracy for classifying COVID-19 pneumonia.
"While CT imaging may not necessarily be actively used in the diagnosis and screening for COVID-19, this deep learning-based AI approach may serve as a standardized and objective tool to assist the assessment of imaging findings of COVID-19 and may potentially be useful as a research tool, clinical trial response metric, or perhaps as a complementary test tool in very specific limited populations or for recurrent outbreaks settings," wrote the team, led by first author Stephanie Harmon, PhD, and senior author Dr. Baris Turkbey of the National Cancer Institute.
Other authors contributing equally to the work included Dr. Thomas Sanford of the State University of New York -- Upstate Medical Center in Syracuse, NY; Dr. Peng An of Xiangyang No. 1 People's Hospital Affiliated to Hubei University of Medicine in Xiangyang, China; and Dr. Bradford Wood of the NIH Clinical Center.
Although a number of studies have reported that AI could produce up to 95% accuracy in detecting COVID-19 on chest CT, implementation of these algorithms at other institutions have been hampered by the tendency for AI to overfit to the training data, according to the researchers. As a result, the researchers in the current study aimed to maximize the potential for generalizability by utilizing a diverse, multinational dataset, which included COVID-19 patients from four hospitals in China, Italy, and Japan.
In addition to including exams performed for COVID-19 screening, the cohort also included studies acquired on the same day as an initial positive real-time reverse transcription polymerase chain reaction (RT-PCR) test, as well as for advanced disease in hospitalized inpatients.
The dataset also included normal control cases from chest CTs performed for oncology, emergency, and pneumonia-related indications at two other hospitals. A publicly available dataset was also used.
"Furthermore, the inclusion of patients undergoing routine clinical CT scans for a variety of indications including acute care, trauma, oncology, and various inpatient settings was designed to expose the algorithm to diverse clinical presentations," they wrote.
After initially developing a lung segmentation algorithm, the researchers then evaluated several different classification models. Training and validation was performed on 1,280 patients; testing was performed on an independent test set of 1,337 patients. The best results were achieved by a 3D classification model that utilized the entire lung region.
| Performance on independent test set for classification of COVID-19 on chest CT | |
| Area under the curve (AUC) | 0.949 |
| Sensitivity | 84% |
| Specificity | 93% |
| Accuracy | 90.8% |
The researchers noted that the algorithm's performance was highest in patient cohorts from centers utilizing CT earlier in the diagnostic pathway; poorer detection sensitivity was found when using CT in cases with advanced pneumonia.
"Further work regarding the diagnostic utility of this algorithm in the setting of early vs. advanced COVID-19 related pneumonia is warranted," they wrote.
Although the model's sensitivity moderately declined when applied across multinational cohorts, it did appear to generalize well, given the variability in CT acquisition and clinical considerations, according to the authors. Prospective validation is now needed, however.
