By BRYAN CARMODY
Recently, I was on The Accad and Koka Report to share my opinions on USMLE Step 1 scoring policy. (If you’re interested, you can listen to the episode on the show website or iTunes.)
Most of the topics we discussed were ones I’ve already dissected on this site. But there was an interesting moment in the show, right around the 37:30 mark, that raises an important point that is worthy of further analysis.
__
ANISH: There’s also the fact that nobody is twisting the arms of program directors to use [USMLE Step 1] scores, correct? Even in an era when you had clinical grades reported, there’s still seems to be value that PDs attach to these scores. . . There’s no regulatory agency that’s forcing PDs to do that. So if PDs want to use, you know, a number on a test to determine who should best make up their class, why are you against that?
BRYAN: I’m not necessarily against that if you make that as a reasoned decision. I would challenge a few things about it, though. I guess the first question is, what do you think is on USMLE Step 1 that is meaningful?
ANISH: Well – um – yeah…
BRYAN: What do you think is on that test that makes it a meaningful metric?
ANISH: I – I don’t- I don’t think that – I don’t know that memorizing… I don’t even remember what was on the USMLE. Was the Krebs Cycle on the USMLE Step 1?
__
I highlight this snippet not to pick on Anish – who was a gracious host, and despite our back-and-forth on Twitter, we actually agreed much more than we disagreed. And as a practicing clinician who is 15 years removed from the exam, I’m not surprised in the least that he doesn’t recall exactly what was on the test.
I highlight this exchange because it illuminates one of the central truths in the #USMLEPassFail debate, and that is this:
Physicians who took Step 1 more than 5 years ago honestly don’t have a clue about what is tested on the exam.
That’s not because the content has changed. It’s because the memories of minutiae fade over time, leaving behind the false memory of a test that was more useful than it really was.
I’m speaking from experience here.
CONFESSIONS OF A FORMER STEP 1 APOLOGIST
Believe it or not, I was once a USMLE apologist. If you’d asked me my opinion of Step 1 a few years ago, I would have said “Yeah, it’s not the best, but it’s the best we’ve got…” and parroted back many of the same arguments for a scored test that are still bandied about today.
My opinion started to change a few years ago when I started teaching some courses for our first- and second-year medical students. I wanted to be sure that the concepts and terminology I used were consistent with what they needed to know for the USMLE, so I reviewed a whole bunch of Step 1 questions using the NBME’s Computerized Assessment System (CAS).
As I answered question after question, three broad categories of questions emerged.
- CORE SCIENTIFIC CONCEPTS: Many questions I knew cold. These questions tested key aspects of physiology, pharmacology, or pathology that I use frequently to take care of patients.
- A LITTLE OFF THE BEATEN PATH: Another group of questions were more challenging. They asked about basic science concepts with only peripheral relevance to the practice of nephrology. Some of these I got right; some I got wrong. Most stirred up a dormant memory of facts that I might have known during my own Step 1 studying.
- BASIC SCIENCE TRIVIA: A final group of questions were completely foreign. Not only did I not know them, but I had no memory of there ever being a time in my life when I would have known them. For these, it was hard to even conceive of how such information could be clinically useful.
I’d like to share examples of these questions to you. However, to gain access to CAS, I signed a blood oath confidentiality agreement with the NBME that I would not disclose the questions within, and I am going to honor that.
However… the USMLE does provide some real USMLE Step 1 sample questions to the general public. I reviewed just the first block (questions 1-40) and highlighted some of the best below.
SAMPLE QUESTION 1
You might think that the makers of a mandatory medical licensing examination would be interested in determining whether doctors-to-be knew how to keep this poor guy from dying in the first place. Nah. For Step 1, let’s just figure out who knows how this cell got here.
(The correct answer, in case you were wondering, is B.)
_
SAMPLE QUESTION 2
What I love about this question is the way it lures you in with a totally-believable clinical vignette. Is it a question about burn care? Infection prevention? Indications for debridement? Nope. Hope you memorized your cytokines! If not, the correct answer is E.
_
SAMPLE QUESTION 3
Whoa. Not to be outdone by Sample Question 2, this question makes multiple head-fakes toward clinically relevant medical decision-making that might benefit a human being… before ending with cellular signaling pathways.
Look, I actually evaluate children suspected to have diabetes insipidus. Never once has it occurred to me to look up this factoid (much less carry it around in my head). But in case your diagnostic algorithm differs from my own, the correct answer is A.
SAMPLE QUESTION 4
Are you sensing a theme here?
As usual, the question stem starts off strong with a vignette on a clinically-relevant entity (RSV bronchiolitis), before diverting into Virology Jeopardy. Is it really the case that all doctors must know, as a condition of licensure, that RSV has a linear, negative-sense RNA genome?
Sigh.
I hopefully guessed answer choice E, thinking that perhaps the USMLE was trying to test a clinically-useful point which I had been heretofore unaware. They weren’t. The correct answer is D.
_
SAMPLE QUESTION 5
I have to at least applaud this question for its lack of pretense. No clinical foreplay at all. It gets straight into the basic science.
And in case it’s been a while since you walked into clinic and did a Southern blot on the digestion products of your patient’s cerebellum and pancreas, the correct answer is C.
HOW’D YOU DO?
Anybody go 5 for 5?
Let me save you a little time on your angry e-mails and address some of the inevitable objections to this little exercise.
YOU CHERRY-PICKED THOSE QUESTIONS.
Yes, I did. But bear in mind, these cherries were plucked from a 40 question block – so these 5 represent 12.5% of the total. (Plus, these are the exemplary questions that the NBME has chosen to release on the sample test. The questions I reviewed on CAS verged even farther into the realm of the obscure.)
And bear in mind, as Step 1 scores climb higher and higher, the questions that are used to discriminate between candidates at the upper end become more and more obscure. Knowing the answers to these kinds of questions may be the difference between a student being able to pursue a career in, say, orthopedic surgery – or not.
So students will study more and more to memorize facts that are less and less useful. And as this process repeats iteratively over residency application cycles, things will get worse and worse – until we demand that something be done about it.
THOSE AREN’T BAD QUESTIONS. I GOT THEM ALL RIGHT.
Cool. Actually, I got two of them right. But that doesn’t keep me from appreciating that the concepts being tested are almost certainly unnecessary for the safe and effective practice of medicine.
THIS MATERIAL ISN’T IRRELEVANT! WHY, JUST YESTERDAY I PRESCRIBED A VEGF INHIBITOR TO A PATIENT IN MY CLINIC.
So why not test the indications for using VEGF inhibitors, or their side effects? Just because the answer to some of these questions relates to something that’s useful doesn’t mean that they’re good questions.
THOSE QUESTIONS AREN’T IRRELEVANT FOR ALL PHYSICIANS. WHAT ABOUT THOSE WHO PURSUE CAREERS IN RESEARCH? SOUTHERN BLOTTING IS QUITE USEFUL!
Let me be clear – none of what I am pointing out here is intended to denigrate basic science or basic scientists. For those of you who work in research, hats off to you. Honestly. But I would put it to you that even those who have made it their life’s mission to understand the Janus kinase weren’t inspired to do so through their Step 1 preparation.
Here’s the thing: the USMLE Step 1 is a licensing examination. Medical licensure exists to protect the public.
And it’s very for me difficult to conceive of a situation in which a patient could be harmed by a physician’s inability to answer any of those questions above. (It’s even more difficult to try to imagine a case in which a patient would benefit by receiving care from a clinician armed with these nuggets of trivia.)
WE DON’T USE STEP 1 SCORES BECAUSE OF THE KNOWLEDGE IT TESTS. WE USE SCORES BECAUSE IT SHOWS US WHAT CANDIDATES HAVE THE BEST WORK ETHIC, TIME MANAGEMENT SKILLS, DETERMINATION, GRIT, ETC.
Then could we not measure those things while engaging students in an endeavor that actually leads to them being a better physician? If it’s really true that the content is irrelevant, why not just ask students to memorize digits of pi?
THE BOTTOM LINE
The “save the scored USMLE” coalition is a varied one.
There are students who perceive that they benefit from a scored test, or who would rather compete with their Step 1 score than by some other metric. There are graduates who worked their butts off to get a high score and have benefitted from it – and they view an attack on Step 1 scores as an attack on their own intelligence or work ethic. There are (cough cough) certain organizations who have a vested financial interest in maintaining a scored test. And there are well-meaning faculty and program directors who aren’t sold on Step 1, but buy into the NBME’s messaging about how anything that could possibly replace Step 1 will be worse.
Some of those objections I get; some, not so much.
If you support maintaining scored test, all I ask is this: know what you’re fighting for. Don’t rely on your own remote experience with Step 1 inform your knowledge about what’s being measured by the Step 1 score.
Dr. Carmody is a pediatric nephrologist and medical educator at Eastern Virginia Medical School. This post originally appeared on The Sheriff of Sodium here.
