An artificial intelligence (AI) algorithm yielded a high level of accuracy for distinguishing between cases of COVID-19, influenza A/B, nonviral community-acquired pneumonia, and healthy subjects on CT exams, Chinese researchers reported in an article published online October 9 in Nature Communications.
What's more, their model outperformed five experienced radiologists who participated in a reader study, and the algorithm showed good generalizability when applied to external test sets, according to co-first authors Cheng Jin, PhD, and Weixiang Chen of Tsinghua University in Beijing, and Dr. Yukun Cao of Tongji Medical College in Wuhan.
"Radiologists could perform an individualized diagnosis of COVID-19 with the AI system, adding [a] new driving force for fighting the global spread of [the] outbreak," the authors wrote.
They trained and tested their deep-learning algorithm using 4,260 CT scans retrospectively gathered from 3,177 subjects from three centers in Wuhan. Of these studies, 2,529 were COVID-19 scans, 1,338 were cases of community-acquired pneumonia, 135 were influenza A/B studies, and 258 were normal.
After being trained on 3,263 scans from 2,688 subjects, the algorithm was then tested on 3,199 studies from 2,688 subjects. These included 1,229 healthy subjects, 668 with nonviral community-acquired pneumonia, 42 with influenza A/B, and 751 with COVID-19.
Jin and colleagues also applied their algorithm to two external test sets: the China Consortium of Chest CT Image Investigation (CC-CCII) test cohort of 2,539 and MosMedData test set of 254 nonpneumonia cases, and 857 COVID-19 cases from Russia.
Next, they compared the performance of the AI algorithm with radiologists by performing reader studies for the following three different diagnostic tasks on three different patient cohorts:
- Diagnosing pneumonia: 100 subjects, including 50 without pneumonia, 25 with community-acquired pneumonia, and 25 with COVID-19
- Distinguishing community-acquired pneumonia and COVID-19: 100 subjects, including 50 with community-acquired pneumonia and 50 with COVID-19
- Distinguishing influenza and COVID-19: 50 subjects, including 20 with influenza A/B and 30 with COVID-19
These studies were interpreted by five radiologists from Wuhan Union Hospital who had all interpreted over 400 CT scans of COVID-19 patients over a three-month period. The radiologists, who had an average of eight years of experience in imaging diagnosis of pulmonary diseases, were aware of the tasks being evaluated in the reader studies, according to the researchers.
| AI algorithm performance for lung diseases | |
| Overall test cohort | Area under the curve (AUC) = 0.978 |
| External test set #1 (CC-CCII database) | AUC = 0.930 |
| External test set #2 (MosMedData database) | AUC = 0.933 |
| Reader study #1 (distinguishing pneumonia vs. nonpneumonia) | AUC = 0.987 |
| Reader study #2 (distinguishing COVID-19 vs. community-acquired pneumonia) | AUC = 0.973 |
| Reader study #3 (distinguishing influenza A/B vs. COVID-19) | AUC = 0.959 |
The team noted that it took the AI system an average of 2.73 seconds to analyze each study, compared with an average of 6.5 minutes by the radiologists. Although the algorithm performed slightly worse in distinguishing between pneumonia and nonpneumonia, it outperformed the radiologists for the more challenging tasks of distinguishing between community-acquired pneumonia and COVID-19, as well as between COVID-19 and influenza, according to the group.
Of the 26 errors made by radiologists in distinguishing between COVID-19 and community-acquired pneumonia, 23 (88.5%) were correctly classified by the AI system. Similarly, 20 (86.9%) of the 23 mistakes made by radiologists in distinguishing between influenza and COVID-19 were correctly categorized by the AI software, the researchers noted.
"[These results mean] that the AI system can be used as an effective independent reader to provide reference suggestions," the authors wrote. "Besides, with a [high] sensitivity setting, it can screen out suspicious patients for radiologists to confirm; with a high accuracy setting, it can give possible diagnosis error warnings made by radiologists."
They also said that applying radiomics analysis to the AI results could potentially lead to the discovery of new biomarkers for COVID-19.
