There may be flaws in a new statistical tool developed by the U.S. Centers for Medicare and Medicaid Services (CMS) that's designed to measure how often hospitals perform appropriate brain CT studies on emergency patients presenting with headaches, according to an article published online February 24 in Annals of Emergency Medicine.
Outpatient Measure 15 (OP-15) is one of a group of so-called imaging efficiency measures that CMS developed to determine appropriate use of imaging for seven clinical indications. CMS has indicated that it could base future Medicare reimbursement on the ability of outpatient facilities to meet efficiency levels as measured by OP-15.
But OP-15 may not be accurately measuring whether emergency departments are doing a good job of ordering appropriate CT studies, according to researchers from Harvard Medical School and other members of the American College of Emergency Physicians (ACEP). Adopting the tool on a widespread basis could have unintended consequences, such as failure to order indicated CT scans, in particular for older adults, they warn (Ann Emerg Med, February 24, 2012).
About OP-15
OP-15 is one of seven imaging efficiency measures included in CMS' Hospital Outpatient Quality Reporting (OQR) program for calendar year 2012. In this quality data reporting program, hospitals participating in CMS' Outpatient Prospective Payment System (OPPS) report data using standardized measures of care to receive payments. To receive the full annual payment, hospitals must submit data for 23 quality measures, 14 clinical performance measures, two Web-based structural measures, and seven imaging efficiency measures.
The seven imaging efficiency measures calculated from hospital Medicare fee-for-service claims data paid under OPPS are as follows:
- OP-8: MRI lumbar spine for low back pain
- OP-9: Mammography follow-up rates
- OP-10: Abdomen CT use of contrast material
- OP-11: Thorax CT use of contrast material
- OP-13: Cardiac imaging for preoperative risk assessment for noncardiac low-risk surgery (new in 2012)
- OP-14: Simultaneous use of brain CT and sinus CT (new in 2012)
- OP-15: Use of brain CT in the emergency department for atraumatic headache (new in 2012)
The category of brain CT for atraumatic headache as a new imaging efficiency measure had been chosen because of increasing use and practice pattern variation among physicians and hospitals ordering CT exams. Its criteria had been developed by an independent contractor for CMS.
The validation study
Because OP-15 had not been validated against chart review, and because the measure had not been endorsed by the National Quality Forum due to lack of supporting scientific evidence, a team of researchers decided to conduct a validation study. The study was led by Dr. Jeremiah Schuur, an emergency physician and assistant professor of emergency medicine at Harvard.
For the study, 21 hospitals retrospectively reviewed all brain CT exams ordered by emergency department physicians in 2009 to identify the ones ordered for patients presenting with atraumatic headaches. They then reviewed each patient's medical records to determine if the clinical indications for ordering a CT exam met the CMS criteria, and if they did not, whether the diagnosis of the patient merited performing the exam.
Most of the participating emergency departments were high-volume and located in urban and academic hospitals, but several of the hospitals were small, and one was in a rural setting. Fifteen hospitals had computerized physician order-entry (CPOE) systems for radiology in place; eight also offered integrated clinical decision-support systems. Almost half treated more than 50,000 emergency patients each year.
From the records submitted by the 21 emergency departments, the researchers reviewed the medical records of 748 emergency department visits by 732 individual patients for whom brain CT scans were considered inappropriate by CMS criteria. Of those scans, 83% should not have been labeled as inappropriate based on either ACEP clinical policy guidelines or expert consensus standards.
Specifically, 65.4% of brain CT exams identified as inappropriate by CMS were for patients with a "measure exclusion," or a clinical condition for which ordering a CT scan for atraumatic headache is clinically indicated. That meant that OP-15 was only 34.6% reliable.
A list of indications for a brain CT exam showed that the measure's validity was 47.5% among the 259 patient visits without measure exclusions. Overall, 623 of the 748 patient visits had either a measure exclusion or a consensus indication for a CT exam.
"OP-15 was only 17% accurate in assessing which patients should receive a CT scan," Schuur said. "Furthermore, hospitals' performance on the new measure as reported by CMS did not match the proportion of CT exams with a documented clinical indication."
The authors attributed the poor performance of OP-15 in part to the limitations of using administrative data in assessing appropriateness for this condition. The study results call into question the justification for the measure and challenge the integrity of CMS' measure development procedures, they wrote.
Recommendations for CMS
The authors strongly endorsed CMS' goal of reducing inappropriate or unnecessary CT scans ordered by emergency physicians. They offered three recommendations specifically related to the OP-15 imaging efficiency measure:
- Government and private insurance payors should partner with providers and researchers to develop efficiency measures based on evidence-based appropriateness rather than utilization.
- Policymakers should encourage further development, testing, and implementation of clinical decision-support systems.
- Policymakers should address the thorny issues of defensive medicine and what should be done when treating patients who demand advanced imaging procedures.
CMS needs to be aware that its well-meaning intentions could backfire, Schuur and colleagues also warned. Physicians could use different diagnostic codes or improve documentation for exclusions. Improving coding without altering ordering practices will not improve quality or cost, they noted. In addition, they expressed great concern that hospitals could pressure their emergency physicians not to order brain CT scans for Medicare patients with atraumatic headaches.
In conjunction with publication of the current study, ACEP issued a press release on February 23 criticizing the new imaging efficiency measure for being inaccurate. The organization also chided CMS for potentially pressuring physicians not to order otherwise appropriate CT scans in order to bolster their hospitals' performance on the CMS measure.
