Why does this change feel like a roll of the dice?

By: Ryan Kelly

On February 12th, it was announced that Step 1 of the United States Medical Licensing Exam (USMLE) will now be graded on a pass/fail basis, as opposed to a numerical score.

According to the official announcement, the change will be phased in over the next 11-24 months.

Step 1 assesses a medical student’s knowledge of the basic foundational sciences. In the past, it has been scored on a scale of 0-300, with 230 being an average score and 250+ being considered above average.

If you’ll be taking Step 1 before January 2022, this decision will have little to no impact on you. But if you’ll be taking Step 1 after January 1st, 2022, your Step 2 score, shelf exam, medical school, and grades have just become far more significant.

The Invitational Conference on USMLE Scoring (InCUS), which drew upon input from the Federation of State Medical Boards (FSMB) and the National Board of Medical Examiners (NBME), was rather vague in its published statement that describes the motivation behind the change:

“In reaching this decision, various score reporting options were considered, including maintaining the status quo (i.e., no change to score reporting). The views of all stakeholders were considered throughout the score reporting discussions. Specific consideration was placed on supporting the educational engagement and overall experience of medical students – and on increasing the dialogue about how multiple assessments of competency could best be utilized by stakeholders in medical regulation and medical education.

The FSMB and NBME believe that changing Step 1 score reporting to pass/fail can help reduce some of the current overemphasis on USMLE performance, while also retaining the ability of medical licensing authorities to use the exam for its primary purpose of medical licensure eligibility. The USMLE co-sponsors also believe that moving to pass/fail reporting of Step 1 while retaining a scored Step 2 CK represents a positive step toward system-wide change, while limiting large-scale disruption to the overall educational and licensing environment.”

Yeesh - that language is the definition of ‘boilerplate’ - what’s really going on here?

What are the pros and cons of this change, and which stakeholders will be most affected by it?

Pros of the New Pass/Fail Step 1 of USMLE

It’s important to note that many of the “pros” could also be “cons,” depending on your point of view or particular speculation regarding the change’s effects.

Pro - Better Student Wellness

In past articles, we’ve covered the epidemic of medical student burnout, and this problem is said to be at the heart of the change to Step 1.

Many believe that the intense pressure attached to one’s numerical Step 1 score has been causing unhealthy levels of anxiety and stress for medical students during their first and second years. This is thought to be particularly problematic, considering that much of the material covered in Step 1 may not be especially relevant to the practice of these future doctors.

By changing Step 1 to pass/fail, this stress will hypothetically be alleviated.

But some sources argue that Step 1 is not the ONLY cause of burnout, and that this accommodation could actually lead to a “victim mentality,” where students continuously look for new things to blame for their lack of success.

Others argue that the stress associated with Step 1 will now just transfer over to Step 2. Previously, students who did poorly on Step 1 had a second chance by improving on Step 2, but now they only have one chance to prove their test-taking abilities to their prospective residency programs.

Pro - Fewer Disparities and Greater Equity

The idea is that a more “holistic review” will even the playing field by forcing residency programs to account for non-numerical factors. It’s believed that a poor Step 1 score might plague otherwise-stellar candidates who simply struggle with standardized tests.

Plenty has been written about the social/political discrimination that’s embedded in standardized tests, and traditionally, minority students tend to perform at a lower average on these exams.

Preparation for Step 1 can also be expensive, so this change will theoretically make medical school slightly more affordable for students (especially those from low-income families who do not have the financial backing of their families).

Lastly, since Step 1 has historically been a filter for more competitive specialties, like orthopedic surgery or dermatology, the change could decrease the stratification and elitism that are perpetuated by the test and subsequent student ranking.

However, when we recently interviewed an M1, he expressed concern that this change would more likely harm “lower-tier” candidates. He believes that students in the top ~30 MD programs will be fine, since these programs are so concerned and proactive about making sure their students match into top residencies.

On the flip side, the Step 1 exam will no longer be a way for lower-tier candidates to distinguish themselves, and the lower stakes of the test might even encourage them to pursue less competitive specialties like family medicine (which they might have felt destined to pursue in the first place).      

Pro - Eliminates the “Parallel Curriculum” Problem

InCUS has stated that the pass/fail change will counteract the problem of a “parallel curriculum” that has essentially supplanted the traditional existing curricula at medical schools:

“Emphasis on maximizing USMLE scores has led medical school students to the perception of a ‘parallel curriculum’ to the curriculum of their medical schools. Students engage in their school’s curriculum while simultaneously preparing for what is perceived to be ‘other material’ important for achieving high scores on the USMLE assessments.”

Many students have abandoned aspects of their medical school curriculum that are less relevant to Step 1. Since a competitive Step 1 score is so crucial to their admission success, students often forego lectures or labs to make more time for independent studying and test preparation.

The thought here is that Step 1 is hurting students’ investment in their medical school community, cohort, and involvement, as well as overall well-roundedness.

Pro - Thwarts the Greed of the NBME

Although the NBME is a not-for-profit organization, it is largely believed to have a monopoly on the practice tests and preparation materials for Step 1.

The CEOs of the NBME and FSMB have expressed concern that medical students’ studying time might now be replaced by Netflix binging and Instagram posting, but some financial and admissions experts believe that they’re more concerned with their revenue and finances.

However, if you’re stoked about “sticking it to the man,” you shouldn’t get too excited. With these changes, we’re likely to see a mad scrambling for power and profits among disseminators of Step 2 preparation materials. If residency programs will now be looking towards Step 2 for their objective assessments, medical students will still be emptying their wallets, albeit into different pockets.

Cons of the New Pass/Fail Step 1 of USMLE

Con - More Nepotism

With the numerical Step 1 score being nulled, it’s argued that there will be a shift in focus towards more “social aspects” (loosely used term) of your residency application, namely your letters of recommendation and clerkship grades.

Letters of recommendation are a highly ranked criterion during the residency selection process, and now, there will be even more emphasis on students’ networking abilities, or in other words, their skills in schmoozing and “kissing up” to their superiors.

Clerkship grades were also highly ranked among selection factors, and these are very susceptible to subjectivity and favoritism, since they come directly from a personal assessment from a superior.

So, even though the intent of the change was to alleviate stress, medical students will now feel the pressure of this “social game” when developing their applications for residency programs. And for many students, particularly the more reserved and introverted ones, this could wind up being even more stressful.

In our interview with our M1 friend, he explained that students’ “pedigree” and connections (“who you know”) have always been major factors, but the change to Step 1 could make this facet even more cutthroat and competitive among applicants.    

Con - Hurts DO and International Students

As explained in the past section, residency program directors place a high emphasis on letters of recommendation, clinical grades, and research experience.

It’s no secret that students at top schools have more access to prestigious faculty and research opportunities. Since clinical grading differs across schools, Step 1 scores have been used as “objective” metrics that could aid in comparing candidates from different schools. The absence of Step 1 scores will likely exacerbate the advantages for students from top-tier schools.

If you can read between the lines, it’s easy to see why this is bad for DO and international students; a stellar Step 1 score used to be an opportunity for them to differentiate themselves and compensate for their “questionable” academic reputation, but now, that chance is gone.

Remember that “pro” about greater equity? It seems like the intentionality behind that could backfire and further stratify students.

Con - Fewer Objective Criteria for Residency Selection

A criticism of Step 1 was that residency program directors used it as an unfair shortcut to filter down applicants.

Programs will now need to find other “holistic” methods of screening applicants. While this might sound like a good thing, it is incredibly challenging in practice.

It is common for programs to receive ~100 applications for a single position. To screen all their applications without a score, programs will need to invest in more staff or increase their reliance on overworked faculty and residents.

As mentioned earlier, letters of recommendation and clinical grades are the furthest thing from objective. According to our M1 friend, it’s impossible to standardize clinical grades, both in-house and across schools. Plus, at some programs, the preceptors have a reputation for almost always giving out honors, whereas other programs have “hard quotas” to ensure differentiation.

Residency programs are fully aware of this subjectivity, so they might be at a loss when trying to assess applicants with any degree of fairness.

Currently, there aren’t any tests to measure soft skills like empathy, professionalism, and communication skills. Step 1 isn’t perfect, but it at least offered some standard to go from. Not anymore!

Con - Could Lead to Less Competent Physicians

Without the pressure of a Step 1 score, students might skimp on learning the basic science concepts that seemingly have little relevance to clinical medicine, which is what will be more objectively and numerically scored on Step 2.

Proponents of basic science are objecting to this change, since the newest clinical discoveries often stem from basic science. Plus, it’s more than possible that basic science knowledge could impact clinical care for a student’s future patients.

How will medical schools adjust to compensate for this change in students’ focus? How can they ensure that they’re producing competent, effective future doctors?

Many schools have transitioned to a systems-based curriculum to integrate the basic and clinical sciences, so it’s likely that more and more schools will copy this model.

Final Thoughts on the New Pass/Fail Step 1 of USMLE

In our interview with our M1 friend, he offered some final theories of speculation:

• We will have a Wild West dynamic for the next 2-3 years, especially for students like himself who will be in the last group to have a scored exam. There’s a possibility that programs will simultaneously be assessing students with a score and students without a score, so this middle-ground group might have their scores negated unilaterally.

• The Step 1 exam itself might change, and if so, it’s likely to get harder. When dental exams became pass/fail, the test-makers designed the exams to be more difficult in order to compensate for the pass/fail switch. It’s possible that this could happen to Step 1 as well.
• In the past, some students didn’t even take Step 2 until after residency applications started rolling in (because some programs didn’t require it for their applications). But now, those programs might prioritize it sooner. Overall, it seems like we’re moving towards a system that uses Step 2 as the new authoritative, objective measure of knowledge and test-taking ability.