Stanford Health developed an in-house large language model pipeline that automatically detects critical results in pediatric radiology reports with 99.7% sensitivity and 97.9% specificity, eliminating the need for expensive commercial tools while improving workflow efficiency and patient safety.
- The LLM framework detected six critical result categories: increased intracranial pressure, non-accidental trauma, malpositioned feeding tubes, unsuspected pneumoperitoneum, portal venous gas, and tension pneumothorax
- Validation achieved 99.7% sensitivity and 97.9% specificity across 916 radiology reports
- During prospective deployment from August 2024 to July 2026, the system flagged 1,297 critical results from 182,047 finalized reports
- The locally deployed LLM (Llama 3.3, 70B) requires no external data transfer or commercial tooling, reducing costs and privacy risks
- Turnaround times and communication documentation remained stable throughout deployment, proving the system's reliability in clinical workflow
PHILADLEPHIA – A large language model (LLM) framework can automate the detection of critical results and turnaround time monitoring in pediatric radiology, according to findings presented at the Society for Imaging Informatics in Medicine (SIIM)-CAIMI 2026 meeting.
In her presentation, data scientist Pooja Pathak from Stanford Health in California shared her team’s findings showing how their in-house LLM approach is reliable in their institute, including achieving near-perfect sensitivity in validation settings.
“I think as a first-case establishment in automated detection of these critical results, our team’s approach was largely successful,” Pathak told AuntMinnie.
Critical results reporting and tracking are needed for quality assurance and workflow efficiency in radiology. And while commercial tools exist for adult patients, Pathak said it’s more difficult to develop equivalent pediatric approaches. This puts in-house development in the spotlight.
Pooja Pathak presents her team's findings at SIIM-CAIMI 2026 in Philadelphia.
The Pathak team developed and deployed its LLM-based framework for automated detection and turnaround time monitoring of critical results in a pediatric radiology workflow.
The team defined six critical result categories: increased intracranial pressure, non-accidental trauma, malpositioned feeding tube, unsuspected pneumoperitoneum, portal venous gas, and tension pneumothorax. It used a locally deployed LLM (Llama 3.3, 70B; Ollama) to process report impressions. The researchers validated the framework against radiologist-adjudicated labels on 916 reports across six categories (5,496 report-category decisions). The researchers deployed the system in August 2024, with the study including data collected between then and July 2026.
The dataset included 182,047 finalized reports. X-rays made up most of these reports at 67.5%, followed by ultrasound (16.0%) and MRI reports (9.4%). CT, fluoroscopy, and nuclear medicine made up the remaining reports. The patients had an average age of 10.8 years.
Validation yielded 351 true positives, 107 false positives, 1 false negative, and 5,037 true negatives. This corresponded to 99.7% sensitivity, 97.9% specificity, and a negative predictive value of 99.9%.
Prospective implementation led to 1,297 critical results flagged during the deployment period. These included the following critical results: 892 malpositioned feeding tubes, 158 portal venous gas, 109 tension pneumothorax, 89 increased intracranial pressure, 40 unsuspected pneumoperitoneum, and 9 non-accidental trauma cases.
Finally, turnaround times and communication documentation remained steady throughout deployment, Pathak reported. She said the results show how open-weight models on local infrastructure can continuously monitor every finalized report in-house, with no need for commercial tooling or external data transfer.
Pathak added that future directions include making the LLM pipeline more context-aware and using more updated models.
“At the time, Llama 3.3 was one of the better models,” she told AuntMinnie. “But since then, we’ve had so many frontier models. So, I think it would be interesting to evaluate how this pipeline would work with an in-focus model.”
