Word is beginning to spread: The American Board of Radiology (ABR) is sending messages about exam integrity. What's that all about?
The ABR is totally serious about respect for the integrity of the examinations it gives. There are some very good reasons why.
The first is professionalism. The ABR, in partnership with the Accreditation Council for Graduate Medical Education (ACGME) and the American Board of Medical Specialties (ABMS), subscribes to the "six competencies" as the framework defining the qualified and competent physician specialist. While all six are important, when it comes to exam integrity, professionalism is the focus. Achieving a passing exam score that is based on sharing questions violates the integrity of the exams, and is de facto evidence that a candidate has not mastered the competency of professionalism.
The ABR's first responsibility as a board is to patients and the public. They have a right to expect that diplomates of the ABR have attained certification because of their own knowledge and understanding. An honest, level, and clean playing field for all examinees is the only credible approach.
Another reason the ABR is serious about the integrity of its exams is that they are resource-intensive to produce. Let us briefly review the process of producing an examination for you:
- For a typical ABR examination, item writers are selected for their expertise in the field. They serve voluntarily and have "day jobs" in addition to writing questions for the ABR. But they give their time to travel to the board office for orientation, receive training and assignments, and follow through with writing from home.
- At that point, ABR staff members enter the equation. The questions are reviewed by professional editors, and the images are put into the proper format by imaging specialists.
- Then exam developers, who serve in a project management role, present the questions for review to the entire committee of experts.
- Some questions do not survive, others need to be reworked, and some pass on to the next stage: selection according to an exam blueprint to produce a well-balanced exam, measuring important aspects of radiology practice.
- The items are then formatted for exam administration. Several iterations of quality assurance review are necessary before the exam is ready for delivery.
Each complete set of test questions, known as a form, costs an estimated $112,000 to produce. And that does not include any of the costs of actual administration, nor any value ascribed to the time of the hundreds of volunteers who write and review the questions. Three forms per year is the production level necessary to deliver the promised two opportunities per year to take the examination. Best testing practices include re-use of well-performing exam questions, because they add stability and help ensure equivalence of exam forms over time. Items that are widely shared fail to perform as intended, necessitating more question production by the ABR and its committees. This directly affects cost, with the end result of forcing an increase in candidate and diplomate fees.
After an exam is produced, another committee process that requires both staff and volunteer time and effort is implemented to establish a passing standard. All ABR examinations are "criterion referenced." That means that rather than being graded on a curve, with a percentage of candidates destined to fail, they are graded with regard to a standard representing the minimum level of content mastery required for safe and effective practice. Theoretically, all could pass -- or all could fail -- depending solely on performance with respect to content, not depending on performance with respect to others in the group of test takers.
Examination blueprints were mentioned above in the context of assembling a well-balanced exam that measures important aspects of the practice of diagnostic radiology, radiation oncology, and medical physics. The process of blueprint development by committees of experts in each discipline, organ system, or modality category is informed by a broad-based practice-analysis survey of current practitioners. This survey reveals those practice activities that are most frequently performed and most critical to patient outcomes across all types of practices and practice settings.
The ABR began conducting such practice analyses recently, with an initial study in 2006 and a repeat study in 2010. These studies assist the ABR item writing and reviewing committees to ensure that the items on the exam are not esoteric or trivial, but rather truly relevant to the most important aspects of contemporary practice. Additionally, examinee evaluations are solicited after every exam administration to gather perceptions regarding the practice relevance of the questions, as well as other aspects of a fair examination experience.
So why would any candidate for certification want to demean the value of the very examinations that have led to his or her certification by participating in the exchange of recalled test questions? The potential gain is certainly not worth the loss of integrity, credibility, and value of certification for our specialty! The ABR is prepared to take whatever steps are necessary to safeguard the integrity of its exams, including sanctions affecting certification, if necessary. A robust response to this appeal for professionalism by all stakeholders, individually and collectively, will launch the culture change that is needed.
The comments and observations expressed herein do not necessarily reflect the opinions of AuntMinnie.com, nor should they be construed as an endorsement or admonishment of any particular vendor, analyst, industry consultant, or consulting group.
