ABR policy on exam recalls: Please read carefully

There has been intense debate among residents regarding the recent policy changes at the American Board of Radiology (ABR) regarding the sharing of ABR exam items, also known as "recalls." While some residents have voiced opposition to the changes, they go to the heart of maintaining professionalism in radiology.

The ABR announced its new security policy on exam item sharing in an April 27, 2011, letter sent to radiology program directors and chairs. The policy prohibits examinees from giving or receiving confidential examination information at any time prior to, during, or after the administration of the examination or possessing study materials in any medium during the time period of the examination.

The rationale behind this policy is multifactorial:

• The new examinations are extremely resource-intensive to produce.
• Use of recalled questions is unnecessary because the new examination is practice-relevant and image-rich, and because study guides and sample questions have been posted online and will continue to be updated.
• Use of recalled questions renders the situations encountered on the examination not novel, but memorized; as such, the examination doesn't measure higher-level management skills.
• If the ABR had to produce examinations of entirely new content each year, the examination production expenses would double, resulting in higher examination fees.
• The public has the right to expect that diplomats of the ABR have attained that status because of their own knowledge, and not because of association with others who could tell them the answers to examination items or because they have access to other sources to discover the correct answers.

The ABR estimated that it costs more than $100,000 to produce each set of examination questions.¹ To protect the investment of time and effort in developing the tests, the ABR copyrights the test materials, including the questions. Ownership is recognized by federal intellectual property law. Each physician taking an ABR examination is given the ABR Examination Security Policy² that communicates the board's ownership of the questions. Each examinee signs a statement agreeing to abide by the ABR's policy.

Forum postings

The new policy has sparked debate among residents, such as in a forum posting on AuntMinnie.com in September 2011.³ The posting began with an entry citing an article in the American College of Radiology (ACR) Bulletin.¹ In the ACR article, Shields and Hoffman emphasized that providing, selling, buying, or using active test questions from the ABR examinations was unprofessional and illegal.

Negative comments on the policy included:

• "At the time I took boards it was felt that the writtens had little if anything to do with radiology other than anatomy in general and medical conditions in general. Most felt that the only way to study was to use old exams."
• "The recent drive against recalls by the ABR most likely means they'll be using the same questions, since implementing a fair testing strategy has been deemed too expensive."
• "They [ABR] probably SHOULD charge more for a fair, valid test that is rewritten every year."
• "I doubt that they [ABR] will have much success at trying to prosecute any test takers."

There were also positive comments:

• "The recertification exam is very fair, image-rich, and you can pass by actually opening up books and studying the books -- the way you're supposed to study for a fair test."
• "The new test pilot section that I took before the boards was certainly a lot better than the writtens, and I really don't think it's the kind of test that you would need recalls to prepare for."
• "All standardized tests reuse questions -- you basically have to for reproducibility purposes."

There is a scholarly rationale for reusing examination items, which was recognized by one AuntMinnie.com Forum participant. Question banks contribute to the validity and reliability of the assessment process by establishing a common language for discussing curriculum goals and objectives. The questions relate directly to individual tasks (skills and knowledge specific), which students are capable or incapable of demonstrating.⁴

Reusing questions also allows for the following:

• Statistical reliability of each examination from year to year⁵
• More predictable and stable examination performance as those items that perform below standards are removed
• Monitoring of passing standards to ensure that they do not vary arbitrarily across time

Examinees should not assume, as did one AuntMinnie.com Forum participant, that the ABR will not be able or willing to prosecute those who breach the ABR's security policy. The American Board of Internal Medicine (ABIM) recently took action against 139 physicians for requesting or otherwise misusing actual ABIM certification examination answers. The penalties included delaying candidates' opportunities to take the certification examination and suspending or revoking certification for those who already completed it.

Additionally, the ABIM filed civil lawsuits against five of the worst physician offenders, as well as a commercial review company that solicited and used examination questions in its courses.¹

These same penalties are available to the ABR. The ABR has said it will respond to violations of its security policy by one or more of the following actions:

• Letter of warning
• Invalidation of examination
• Denial of admission to the examination for a specified period and/or the requirement to retest
• Notification of others such as program director, department chair, and/or external authorities
• Judgment as unacceptable for certification
• Legal action for copyright violation

In my capacity as chair of the department of radiology at the University of Cincinnati (UC), I recently met with my residents and informed them of the ABR's policy, the rationale for the policy, and potential consequences if found to be in violation of the policy. I emphasized how disappointed I would be if anyone from the UC radiology residency program was found to be guilty of policy violation.

I told my residents that no matter how remote the chances of being caught may seem, they should strongly consider the risk of breaching the policy and whether they feel the risk outweighs the benefit. I advised them that in my opinion, it is not a risk worth taking. It can negatively affect the violator's career.

Furthermore, I strongly feel that when a resident agrees in writing to abide by the policy, any breach of that policy is unethical, unprofessional, and unacceptable.

Dr. Jannette Collins is the Ben Felson Professor and chair of radiology at the University of Cincinnati College of Medicine.

The comments and observations expressed herein do not necessarily reflect the opinions of AuntMinnie.com, nor should they be construed as an endorsement or admonishment of any particular vendor, analyst, industry consultant, or consulting group.

References

1. Shields B, Hoffman T. Studying or cheating? Misusing ABR exam questions will cost you. ACR Bulletin. July/August 2011:27-28.
2. American Board of Radiology. ABR examination security policy. Available at: http://theabr.org/forms/ABR_Exam_Security_Policy.pdf. Accessed on October 14, 2011.
3. AuntMinnie.com. Forum: Board recalls. Available at: http://www.auntminnie.com/forum/tm.aspx?d=1&m=318436. Accessed on October 14, 2011.
4. Littlejohn A. Reusing Online Resources: A Sustainable Approach to E-learning [e-book]. Sterling, VA: Psychology Press; 2003. Available from: Google books. Accessed on October 14, 2011.
5. College Board. Course and exam development. Available at: http://professionals.collegeboard.com/testing/ap/about/development. Accessed on October 14, 2011.