Second Take is a series of polemics edited by Drs. Saurabh Jha and Jason Itri discussing controversial issues in medical imaging, intended to offer both sides of an argument by deliberately taking a one-sided view of the subject. It is hoped that the audience will not only appreciate the duality of the argument but also the difficulty in resolving the issue. The views expressed by the authors are not necessarily the views held by them. Click here for Dr. Jha's rebuttal to the argument below.
On Friday, January 13, 2012, CNN published a report titled "Doctors cheated on exams," detailing the use of recalls in preparation for radiology board exams.1 The opening statement of the article reads, "For years, doctors around the country taking an exam to become board certified in radiology have cheated by memorizing test questions. ..."
The American Board of Radiology's (ABR) response detailed in the article can be summarized by the following statement by Executive Director Dr. Gary Becker: "Accumulating and studying from lists of questions on prior examinations constitutes unauthorized access, is inappropriate, unnecessary, intolerable, and illegal."
The CNN piece was based on the story of a whistleblower, Dr. Matthew Webb at the San Antonio Uniformed Services Health Education Consortium, who told CNN reporters, "Cheating is the ultimate betrayal of trust to patients, and it's also the most egregious and flagrant violation in academia. I got to where I was based on my own personal achievements, learning and educating myself. To have to take an exam up against others who have been cheating is ... unfathomable."
The use of recalls to prepare for board exams has been radiology's dirty little secret for more than a decade, and many have rushed to pass judgment on radiology residents worldwide.
But the board recalls conundrum, like many situations in life, is not as black and white as portrayed. It is human nature to pass judgment quickly with limited information, particularly when it has to do with cheating. This article will attempt to discuss the preceding statements regarding the use of recalls with an objective approach that suspends judgment, supplemented with the subjective opinions of a radiology "insider."
Are recalls inappropriate?
The first assertion I would like to challenge is that accumulating and studying from lists of questions on prior examinations is inappropriate. Entry into college is dependent on scoring at a certain level on either the SAT or ACT exams. Many preparation courses are available for a cost from commercial businesses such as Princeton Review and Kaplan Test Prep. Classes generally include classroom teaching supplemented with proctored "mock" exams that mimic the real exam, as well as online banks of questions.
At least half of my preparation for the SAT was based on reviewing banked questions that were designed to replicate the format and degree of difficulty of the real test. Surprisingly (at the time), several of the questions I encountered on the real exam were exactly the same as the test bank questions. These companies also offer preparation courses for the LSAT (law school), MCAT (medical school), GMAT (business school), GRE (graduate school), DAT (dental school), PCAT (pharmacy school), USMLE (medical licensing), and bar exam.
When I enrolled in a review course in preparation for the MCAT, our class instructor told us that he had sat for the MCAT five times to see the questions and better prepare us for the exam. In preparation for the USMLE Step 1 exam, I paid for access to an online question bank called USMLEWorld. Much like my experience with the SAT a decade earlier, I noticed several of my USMLE questions were exactly the same as questions I had seen on USMLEWorld.
Dr. Jason Itri, PhD, from the University of Pittsburgh.What I am leading you toward is the fundamental question of how these companies generate massive banks of test questions that are designed to mirror the questions you see on the actual exam. The simple answer is that many of the question writers have taken the exam themselves and replicate questions or write variations of questions they've seen. It's almost impossible not to write questions similar to what you've seen or been tested with because of the indelible imprint in your memory.
Some question writers take the exam specifically so they can create recall-type questions, as was the case with my MCAT instructor. The issue of test sharing arose in medicine when the American Board of Internal Medicine (ABIM) suspended 139 doctors for soliciting or sharing test questions used to certify doctors with an exam review company in 2010. A statement provided by the ABIM reads, "The test questions are copyrighted property of ABIM, and unauthorized dissemination of them is in violation of copyright law and professional ethical medical standards."2
Copyright issues aside, we can debate the appropriateness of creating and using "recalled" questions, but first it is critically important to realize that the answers to these questions are unknown. Questions and answers are remembered and replicated, but determining the correct answer requires research and expertise.
Some physicians, including program directors, have described recalled questions as study tools or guides rather than "cheat sheets." This is because the act of studying from recalls requires students to research topics and understand why one answer is correct and the other answers are incorrect. In gaining that knowledge, students become more proficient in the given subject.
There is a difference between this approach, which requires the student to gain knowledge, and memorizing that the answer to question 5 is A. Should it be considered cheating if the examinee masters the material by studying recalls rather than by reading books and attending lectures? The end point is the same; examinees have the knowledge necessary to pass the exam, demonstrating the requisite expertise to deliver excellent patient care. If the end point is not the same, then we have an entire generation of radiologists practicing in the U.S. who do not have the necessary knowledge to practice radiology competently.
Why are sample test questions used so extensively in test preparation for virtually every standardized test ranging from the SAT to the USMLE? It's because this is one of the most effective ways to study and learn material, as well as assess progress.
Many of us have had the experience of reading a book chapter in exhausting detail, taking extensive notes and rereading sections until we feel comfortable with the material. Then we take the exam and find out that the details we thought were important were not the same details tested on the exam. Is it fair to take away this highly effective study method from learners who have spent their entire adult lives studying for standardized exams using this approach?
At this point, it's in our DNA. If studying from test questions is such an effective way to learn material, then why doesn't Kaplan or Princeton Review offer board review courses for the radiology board exams? That will be addressed in the next section.
Are recalls unnecessary?
One of the main arguments defending the use of recalls is that radiology board exam questions are "so obscure, that unless you know that they like to ask questions about that topic, you're not going to study it because some of them are completely irrelevant to the modern practice of radiology."1
Webb, the resident who revealed the use of recalls in radiology to CNN, did not use recalls and admitted to failing the physics board exam. There are countless anecdotal stories of top residents who did not use recalls because of their superior knowledge, only to fail one of the board exams and subsequently retake it using recalls as a study tool. The ABR should release prior physics and written board exams to the public so that others can judge for themselves if the material is relevant to the practice of radiology.
I would be remiss if I did not provide my own anecdotal example, in which I was asked about the mechanism of Dulcolax on my written board exam. Dulcolax, also known as bisacodyl, is a stimulant laxative used to treat constipation or to clean out the intestine prior to a bowel examination or surgery. It's relevance to radiology remains a mystery to me, as I have never seen it prescribed by a radiologist, nor have I ever used it myself despite performing more bowel examinations than I care to admit.
Are recalls intolerable?
The ABR and residencies across the country have tolerated the use of recalls for years. Residency program directors and residents alike consider recalls a "necessary evil." There are several reasons, in addition to the ones described above, why it has been considered necessary to use recalls as study guides.
First, radiology board exams are considered "high-stakes testing," in which critical decisions about residents are made based on test scores. If you do not pass the board exams, you will not become a board-certified radiologist, and you will have to start looking for a new career. There is also a monetary fee associated with retaking a board exam. Finally, there is the humiliation of being one of the few, if not the only, resident in your class who did not pass a board exam.
These are conditions that make it more likely for individuals to do whatever it takes to pass an exam. By no means does this make it acceptable for residents to do whatever it takes. But there is another critical factor to take into consideration.
Groups that administer standardized examinations generally test validity of the exam in predicting performance, knowledge, or competency. The ACT publishes a technical manual that summarizes studies conducted of its validity in predicting freshman GPA, equating different high school GPAs, and measuring educational achievement.3
Various studies conducted over the lifetime of the SAT show a statistically significant increase in correlation of high school grades and freshman grades when the SAT is factored in. Frey and Detterman analyzed the correlation of SAT scores with intelligence test scores and found SAT scores to be highly correlated with general mental ability.4
According to the Accreditation Council for Graduate Medical Education's (ACGME) Advisory Committee on Educational Outcome Assessment, high-quality assessment methods (exams) should be tested for reliability, validity, ease of use, resources required, ease of interpretation, and educational impact. The radiology board exams do not undergo testing for reliability, validity, or educational impact -- arguably the most important aspects of a standardized exam.
There are few (if any) radiologists who would agree that radiology board exams accurately reflect a resident's competency or knowledge in the field of radiology. When you combine esoteric board questions with the high-stakes nature of the exams and the alarming lack of evidence to support reliability, validity, or educational impact, should it surprise anyone that these exams are not taken seriously, and residents do what's necessary to pass them?
Can I see a show of hands from all the residency program directors who have an unquestionably incompetent resident who manages to pass all the board exams? Let's be honest: Competency as a radiologist is in no way related to how well one answers multiple-choice questions that may or may not be relevant to the daily practice of radiology.
Are recalls illegal?
"All examination material is the sole property of the American Board of Radiology and is subject to copyright protection," according to the ABR Exam Security Policy. "You are expressly prohibited from disclosing, publishing, reproducing, or transmitting this exam, in whole or in part, in any form or by any means, verbal or written, electronic or mechanical, for any purpose."
Residents taking board exams are required to sign paperwork acknowledging their understanding of the Exam Security Policy, which makes the use of recalls "illegal" and "unauthorized." What can be called into question, however, is the assertion in the preceding policy section of the document that exam results always reflect examinee attainment or maintenance of the standard of knowledge, skill, and understanding essential to the practice of diagnostic radiology, radiation oncology, and medical physics.
As stated above, no correlation has been demonstrated between performance on radiology board exams and the attainment or maintenance of the standard of knowledge, skill, and understanding essential to the practice of diagnostic radiology.
Solutions
To solve the board recalls conundrum, the ABR will be administering a new image-rich exam that has been marketed as more clinically relevant than prior radiology board exams. The ABR has also made it exceedingly clear that the use of recalls is prohibited. However, the new board exam remains a high-stakes exam with a multiple-choice question format that once again lends itself to the creation of board recalls.
I'm a realist (and a nihilist); I have my doubts about whether or not we've seen the end of recalls. Unlike the Men in Black, we don't have cool pens that can erase memories with a momentary flash of light. Studying from test questions is an inbred trait, and nobody wants to fail a major exam.
I believe a better solution is to create new questions or change questions for each exam such that recalls cannot be used for anything more than study guides. Problem solved. One of the differences between standardized exams such as the SAT and radiology board exams is that previous exam questions are used to make up approximately 50% of radiology board exams. As noted by Dr. James Thrall, "if the exam is substantially changed from one year to the other, then the recalled questions really become in my view more of a learning resource."
In addition, the ABR should show correlation between performance on the board exams and attainment or maintenance of the standard of knowledge, skill, and understanding essential to the practice of diagnostic radiology. The ACGME has provided a framework to identify high-quality assessment, which requires evaluation of reliability, validity, and educational impact. This type of evaluation is undertaken for other standardized exams, and it is necessary to justify use of the exam to make critical decisions about examinees.
Conclusion
Claiming that the use of "recalls" constitutes cheating calls into question the integrity and honesty of nearly every radiologist who has trained in the past decade. However, taking a more objective view of the board recalls conundrum reveals several important mitigating factors that help us understand why this has happened, while also providing a better framework for how to ensure that residents en masse are not perceived as cheating on standardized exams in the future. This is critical to ensure public trust in our ability and commitment to providing excellent patient care.
Studying from test questions is an accepted and effective way to gain knowledge, an approach we have all used to study for standardized exams. We cannot make people forget the test questions they've seen. We also cannot ethically enforce "high-stakes testing" using an examination with unproven reliability, validity, and educational impact.
The only fair and practical solution is to create an examination that changes from year to year, such that recalled questions can only be used as a study guide, with the exam tested for reliability, validity, and educational impact. In this scenario, there are no opportunities for cheating the process that ensures trainees have attained or maintained the standard of knowledge, skill, and understanding essential to the practice of diagnostic radiology.
References
- Zamost S, Griffin D, Ansari A. Exclusive: Doctors cheated on exams. CNN. http://www.cnn.com/2012/01/13/health/prescription-for-cheating. Published January 13, 2012. Accessed June 18, 2013.
- American Board of Internal Medicine. ABIM sanctions physicians for ethical violations. http://www.abim.org/news/ABIM-sanctions-physicians-for-ethical-violations.aspx. Published June 9, 2010. Accessed June 18, 2013.
- ACT Technical Manual. 2007. http://www.act.org/aap/pdf/ACT_Technical_Manual.pdf. Accessed June 18, 2013.
- Frey MC, Detterman DK. Scholastic assessment or g? The relationship between the Scholastic Assessment Test and general cognitive ability. Psychol Sci. 2004;15(6):373-378.
Dr. Jason Itri, PhD, is currently an assistant professor in the department of radiology at the University of Pittsburgh Medical Center and serves as the director of quality and safety. Prior to taking this position, he trained in the diagnostic radiology residency program and completed an abdominal imaging fellowship at the Hospital of the University of Pennsylvania. He is actively involved with several national organizations, including the American College of Radiology, the American Board of Radiology, and the Accreditation Council for Graduate Medical Education. Over the past several years, Dr. Itri has received several awards related to advocacy and research in quality and safety, as well as for contributions to graduate medical education. His research interests include the culture of quality among hospital staff and trainees, radiologist performance, diagnostic error, process improvement, and leadership development.
The comments and observations expressed herein are those of the author and do not necessarily reflect the opinions of AuntMinnie.com.
