Scrapping high pressure clinical exams for trainees makes it dangerously easy to become a specialist
Pressure from the Australian Medical Council for medical colleges to eliminate high-stakes exams for trainees puts doctors and patients at risk.
I write this because the Royal Australian and New Zealand College of Psychiatrists says its decision to eliminate the only test of practical skills required to become a psychiatrist under exam conditions — the ‘gold standard’ OSCE — was driven by advice from the regulator back in 2021.
The following year, the college introduced a replacement for the OSCE as an interim measure while in-person exams were prevented by pandemic measures.
It was called the Alternative Assessment Pathway (AAP) and it involved a portfolio review looking at each candidate’s performance across their three most recent in-training assessments.
Candidates who failed the portfolio review could still pass by presenting prepared cases to local consultants, a much lower bar than the 10-12 unseen stations of an OSCE.
What was notable was the pass rate for the AAP. It was 90%. This was in contrast to the OSCE pass rates during the four years before COVID struck, which ranged from 66% to 72%.
Despite this, the college has argued that the AAP is not more likely to certify trainees to practise as consultants before they are ready than the OSCE (so-called “false positives”).
I’m one of the authors of a paper published in Australian Psychiatry last month that questions that claim.
We attempted to identify the number of false positives based on three separate assumptions: that the AAP was a reliable test that few candidates fail (reliable/easy), an unreliable test that few candidates fail (unreliable/easy), or an unreliable test that many candidates fail (unreliable/hard).
In the best-case scenario, we found the false positive rate would reach 1.5% — the equivalent of six pre-competent candidates being assessed as meeting fellowship standards.
In the worst case, however, the false positive rate was 35% — the equivalent of 106 of the 305 candidates who passed the AAP being below standard.
To me, those numbers are concerning.
The advice from the AMC was given to all medical colleges. I’m sure it was well-intentioned. It was about reducing the pressure on trainees, speeding up the production of consultants, and increasing the fairness of medical training.
But it was also deeply flawed, given the council provided little to no evidence that eliminating high-stakes exams would achieve any of these goals, and no evidence at all that it would maintain or improve patient outcomes.
When the RANZCP announced that the OSCE was being retired, hundreds of psychiatrists protested in a petition to the college board. They warned of widespread fear within the specialty that it would lead to a deterioration of the skills of psychiatrists and a loss of the public’s confidence.
The petition included a plea not only for the return of the OSCE or an acceptable evidenced-based alternative, but also a return of the long-case examination abandoned in 2012 in favour of the OSCE.
Many psychiatrists link the end of the observed clinical interview with a deterioration in the basic clinical skills that underpin the speciality, such as interviewing and formulation.
For them, the loss of the OSCE is yet another step in the slow erosion in standards of psychiatric practice.
The attempt to move away from high-stakes exams in medical education — the RACGP has also recently replaced its version of the OSCE as part of its fellowship assessment — follows the broader fad in higher education that the application of time pressure in tests is unfair.
In this view, the ability to solve problems more quickly is not a sign of greater intelligence or capability.
In the US, where this idea developed in response to persistent differences in performance between various sociodemographic groups, there is a push to eliminate time-based pressures from the SAT, the test used to select university students.
The alternative to the OSCE being suggested by the RANZCP will not be finalised until 2025 at the earliest, but from the discussion we have heard so far, the suggestions are all based on multiple low-stakes activities performed across the registrar training period.
The petition to the RANZCP indicates that psychiatrists and other medical specialists fear that the end of high-stakes exams will result in disaster.
Senior doctors should be selected for their ability to operate at a high level when time is critical.
The reason is simple: all senior doctors in clinical settings by definition face the risk of sudden responsibility for life or death decisions and emergency interventions.
The ability to pass a high-stakes exam under time pressure is the most appropriate test of a doctor’s ability to effectively meet that fundamental responsibility we all have as specialists.
Dr Andrew Amos is a Queensland psychiatrist and chair of the Section of Rural Psychiatry at the RANZCP.
Read more:
