Learning to produce a concise and accurate radiology report is a crucial element of a radiologist's education. But how can a radiology resident's reporting skills be objectively measured? Researchers from Thomas Jefferson University (TJU) Hospitals believe they have found a way.
By calculating the changes in text characters between a preliminary report from the radiology resident and the final report from the attending radiologist using a metric called the Levenshtein percent, a TJU team found an improvement over the course of a radiology resident's postgraduate training experience.
"Analysis of the Levenshtein percent for individual trainees may be a useful metric for program directors to evaluate a trainee's growth in reporting skills during training," said Dr. Kevin Pan, a radiology resident at TJU. Pan presented the research during a session at the recent Society for Imaging Informatics in Medicine (SIIM) annual meeting in Grapevine, TX.
Radiology residency programs are tasked with educating trainees to practice as competent and independent radiologists, Pan said. Currently, there are several metrics to monitor the educational progress of diagnostic radiology trainees, including the American Board of Radiology certifying exam, the American College of Radiology in-service exam, faculty evaluations, and the newly proposed Accreditation Council for Graduate Medical Education (ACGME) milestones.
Effective communication by the radiologist to the clinician through a well-formulated report is an essential measure of practice competency; however, no objective measure currently exists to evaluate a trainee's ability to create a succinct and accurate radiology report, Pan said.
Through the process of mentoring and feedback, the quality of a trainee's reports should more closely approximate that of the attending physician over time, Pan said. Therefore, the researchers proposed that the difference between the preliminary and final report should diminish for a trainee over time.
Quantitative metric
As a result, the TJU team endeavored to create a quantitative metric to evaluate the extent of changes that attending radiologists make to reports drafted by trainees during report finalization. The group also sought to establish rates of change during training for reports that were of higher complexity.
Pan and colleagues extracted all preliminary and final radiology report pairs transcribed for MRI exams from the institution's RIS for a seven-year period. These report pairs were then anonymized and exported to a MySQL database.
For inclusion in the study, resident trainees were required to have completed both their radiology residency and fellowship at TJU. Six trainees met the criteria and were included in the investigation, while the rest of the reports were excluded.
The researchers used open-source software to calculate two proposed metrics: Levenshtein distance (LD) and Levenshtein percent (LP). Levenshtein distance was used to measure the number of characters required to convert a preliminary report into the final version. To normalize for report length, they then calculated the Levenshtein percent, which is LD divided by final report character length.
The researchers sorted individual report LP values chronologically for each trainee, and they calculated the average LP for each consecutive batch of 100 examinations for all exams performed. They then analyzed the results using Pearson correlation analysis.
Of the 8,912 exams included in the study from the six trainees, the overall average LP was 10.6%.
| Changes to radiology reports, by resident trainee | |
| Trainee | Average LP (95% confidence interval) |
| 1 | 9.44% (8.75% to 10.13%) |
| 2 | 9.93% (9.21% to 10.64%) |
| 3 | 11.82% (10.91% to 12.74%) |
| 4 | 11.54% (10.35% to 12.73%) |
| 5 | 10.54% (9.74% to 11.33%) |
| 6 | 13.66% (12.78% to 14.55%) |
Examining trends, the researchers noted that a relative and sharp decrease in the number of changes to reports as measured by average LP occurred about halfway through the training cycle, although there was considerable variation in average LP among trainees.
"You can see that as the trainees [produced] more and more reports, they have a decrease in the extent of changes for their examinations between their preliminary and final reports," Pan said.
| Correlation between consecutive batches and average LP for each trainee | |
| Trainee | Pearson correlation (p-value) |
| 1 | -0.63 (p < 0.015) |
| 2 | -0.799 (p < 0.00004) |
| 3 | -0.709 (p < 0.003) |
| 4 | -0.777 (p < 0.008) |
| 5 | -0.642 (p < 0.004) |
| 6 | -0.722 (p < 0.004) |
Through residency training, the extent of edits made by attending radiologists should decrease as the trainee develops better reporting skills. And, in fact, the study found a decrease in the extent of changes for a given trainee, independent of variations between specific attending radiologists, Pan said.
"The question is now whether that's a combination of idiosyncrasies for the reporting process, the specific division of neuroradiology, or the stylistic changes of the attending themselves," he said. "Or is this a broad improved competency in concisely communicating findings and diagnosis?"
Experience matters
Trainees with more experience were less likely to have edits made to their preliminary reports, Pan said. Fewer edits were made to preliminary reports after individuals had authored about 500 to 1,000 MRI reports.
Because the LP data demonstrated the expected trend of a decrease in the extent of edits over the training period, it may be a suitable metric for assessing the reporting skills of a specific trainee, Pan said.
A dashboard interface could be used to monitor LP -- and therefore each trainee's reporting skills -- over time, he said. Trainees with LP values that are not following the expected trend of a decrease in edits would be alerted.
Furthermore, validated normative values could be used to track future trainees as a means of identifying outliers earlier and potentially proposing remediation, according to Pan.
"Alternatively, trainees performing ahead of established normative values could be commended for their high levels of reporting skills," he noted.
Further analysis is needed to investigate this LP trend, using additional trainees, imaging modalities, and institutions, the researchers concluded.
