How much variability is there in the performance of radiologists when detecting liver metastases on abdominal CT scans? Perhaps not as much as one might expect, a group of researchers has found.
Abdominal imaging subspecialization does help, but there's not as much variation in sensitivity across a range of readers as one might expect, wrote a team led by Scott Hsieh, PhD, of Mayo Clinic in Rochester, MN. The results of the study were published October 4 in Radiology.
"We found higher [accuracy] for abdominal subspecialists than for trainees or nonabdominal subspecialists, but we found no evidence of difference in sensitivity among groups," the authors noted.
It's no secret that the performance of radiologists can vary in detecting abnormalities on imaging exams. Interpretation differences can be affected by fatigue, experience (or lack thereof), or even the navigation tools a department's PACS workstation offers for interpreting the dense amounts of data CT produces.
Hsieh and colleagues sought to pin down what causes interreader variability for identifying liver metastasis on abdominal CT imaging. They explored factors such as reader experience, navigation patterns like eye movements and how a reader uses the workstation, and whether the amount of eye gaze time on missed metastases affects performance.
The study included 25 readers who interpreted 40 contrast-enhanced liver CT studies. Of the radiologists, nine were abdominal imaging specialists, five were generalists or other types of specialists, and 11 were senior residents or fellows.
Eight of the studies the readers interpreted were normal and 32 were abnormal. The radiologists flagged any liver metastases they found and rated their confidence level that lesions were in fact malignant on a scale of 0 to 100, and the workstation used to assess the images tracked the navigation and eye movements of the readers with eye-tracker software (Popup Calibration).
Hsieh and colleagues then assessed reader performance using the area under the jackknife alternative free-response receiver operator characteristic (JAFROC-1) curve (a measure that allows for quantitative analysis of the interpretation performance of radiologists, with 1 being the reference and values closer to 1 indicating better performance).
Abdominal radiologists had higher performance for diagnosing lesions as malignant, with a mean JAFROC-1 curve of 0.77 compared with a mean of 0.71 for residents and a mean of 0.69 for nonabdominal subspecialists. But the sensitivity differences between the readers were not statistically significant.
The researchers also found the following:
- Image navigation variables associated with higher sensitivity included longer interpretation time (p = 0.003), as well as greater use of liver display windows and assessment of coronal images on the workstation (p < 0.001).
- Eye gaze time was at least 0.5 for 71% of missed metastases and two seconds for 40% of missed metastases.
Although the study doesn't seem to suggest that there's a significant problem with the ability of a range of radiologists to identify liver metastases on contrast-enhanced CT, it's still worth trying to reduce reader variability, the authors noted. They suggested that new reconstruction algorithms or artificial intelligence (AI) tools specifically developed to help clinicians identify liver metastases could help, as could targeted training.
"Defining a more effective training program from [the] insights [of our study] is a subject of future work," the group concluded.
