Kristen Panthagani, MD, PhD is a resident physician and Yale Emergency Scholar at Yale New Haven Hospital, completing a combined Emergency Medicine residency and research fellowship. She graduated from the Medical Scientist Training (MD/PhD) Program at Baylor College of Medicine in 2021, receiving a PhD in Genetics and Genomics in 2020 for her thesis work studying the human microbiome and the health impacts of Hurricane Harvey. Her research interests include population health, epidemiology, clinical informatics, communication and misinformation. During the pandemic, she developed an interest and science communication and education for the general public and founded the independent website ‘You Can Know Things,’ which helps explain the science of the pandemic in a way everybody can understand, with an emphasis on addressing misinformation with evidence-based medicine.
Summarising her thread for those with no access to Twitter:
You may have heard the shocking headline that 250,000 people die every year in the US due to misdiagnosis in the ER. You may be even more shocked to know that this statistic is extrapolated from the death of… just one man. in a Canadian ER. over a decade ago.
These shocking numbers are the results of a report published by the Agency for Health Care Research and Quality (AHRQ), reported in the NY Times this week:
E.R. Doctors Misdiagnose Patients With Unusual Symptoms
Doctors fail to recognize serious conditions like stroke and sepsis in tens of thousands of patients each year, according to a new study.
But as it turns out, the methods used to arrive at these estimates are very, very bad. Let’s go through them and see why…
The report goes through a lot of different outcomes and statistics, but let’s focus on the most eye-catching one: an estimated 250,000 deaths each year in the US due to misdiagnosis in the ER.
The report performs a review of the studies on this topic, and they find one high quality study (methods are strong), and some other not as high quality studies.
The high quality study looked at 503 patients discharged from two Canadian ERs in the late 2000s. In the study, they called people 14 days after discharge to see how they did (or if they were still in the hospital, they looked at their medical charts).
The study found that of the 503 patients, 1 patient had an unexpected death that was related to a delay in diagnosis by the ER physician (details below). That man had signs of an aortic dissection and for reasons we don’t know, the diagnosis was delayed for 7 hours.
This Canadian study is quite reasonable. But the way it was used in the AHRQ report was not.
The goal of the Canadian study was not to estimate the rate of misdiagnosis-related death in their EDs (and especially not in all the EDs across the US). They were looking at overall harms, and didn’t have a large enough study to estimate how often misdiagnosis leads to death.
This didn’t stop the AHRQ from misusing this single death to estimate a death rate across the entirety of the US. Dividing 1 death by 503 patients, they estimate the death rate to be 0.2%. They then multiple by total ER visits in the US: 130 million visits x 0.2% = ~250k deaths
I hope I don’t have to tell you how statistically terrible this is. This is not how epidemiology works.
But, there’s more. Whenever estimating rates from data like this, it’s standard to also calculate a confidence interval, which provides a statistical estimate of the uncertainty around your value.
In general, the smaller the sample size, the wider the confidence interval, and the more uncertain you are that your value is the correct one.
They do calculate their confidence interval (death rate could be anywhere from 0.005% to 1.1%), but then decide it is “implausibly wide” and… just guestimate a new one???
You… can’t do this. This is akin to changing up your p-values simply because they seem wrong to you. You can’t just change statistical parameters. Statistical parameters come from objective analysis of the data, not guestimates from the authors.
They do try to justify their guestimated confidence interval by citing other sources. But, not shockingly, these other sources don’t hold up either.
First, they reference a study of medicare data saying that for people older than 65 without a terminal illness who are discharged from the ED with a non-life threatening diagnosis, 0.12% of them die within 7 days (allegedly dying from ER misdiagnosis??)
They argue that number (0.12%) is inside their guestimated confidence interval, (0.1%-0.4%), providing evidence their confidence interval is right
But there are, of course, problems with this. If you look at the study they cited, many of these deaths were not misdiagnoses. The ER diagnosis matched the death diagnosis for a significant chunk of them.
For those patients whose ER diagnosis matched their death diagnosis, does that mean the ER doc didn’t do enough to help them? Possibly. But ER docs don’t have the power to admit everyone. Say someone has a mild COPD exacerbation, responds to meds, is back to normal…
It is very reasonable to send that person home, and the hospitalist would probably push back if the ER doc tried to admit them. That doesn’t mean the ER doc can guarantee the patient won’t have another serious COPD exacerbation 4 days later.
For those patients whose ER diagnoses did not match their death diagnosis, it’s not fair to assume the ER doc just “missed it” — it’s very possible for a person to not have a particular illness on Monday and then have that illness on Friday.
It’s not reasonable to expect ER docs to predict, in advance, every future illness their patients are going to have. Barring possession of a time machine, that is an impossible task.
It is only a misdiagnosis if the patient had the symptoms when they were seen in the ER, and the ER doc missed it. If the patient developed the symptoms the next day, there’s no way for the ER doc to predict that.
In short, this medicare data is not evidence of ER docs missing diagnoses. If you say it is, that puts a standard on ER docs that is impossible to meet (they must predict every death from any disease in the near future.)
Next, they try to support their guesstimated confidence interval by averaging results from two studies: one from Switzerland that looked at admitted patients, and one that looked at discharged patients in Spain. Since this thread is already very long, I’ll be brief.
The study in Spain looked at patients discharged from the ER who returned 3 days later, and found that 3 of them ended up dying. (They had a control group, but it wasn’t great.)
The Spanish study does not report whether these deaths were a result of preventable error or misdiagnosis, yet the report just assumes they were in their calculations.
But perhaps more importantly, this Spanish study has a very similar study design to three other studies that the AHRQ report ignored in their calculations, all three of which report VASTLY lower rates of death associated with ER misdiagnosis…
How did the AHRQ report justify ignoring these three studies? Because of their… study design.
Seems like a biased inclusion of studies to me?
The study in Switzerland looked at admitted patients and compared the diagnosis the ER doc gave them to the diagnosis listed at discharge (or death). They found that when the diagnosis differed, those patients had longer hospital stays and were more likely to die.
While there are merits to this study, it doesn’t do a dive into the medical records to discern whether the misdiagnosis caused a delay in care that resulted in these deaths. In other words, were these deaths avoidable if the diagnosis had been arrived at earlier?
It’s very possible some deaths could have been avoided, but we don’t have an estimate of how often that happened. Furthermore, sometimes people die from complications they develop in the hospital that weren’t present in the ER, so they are impossible for the ER doc to diagnose.
Second, as has been stated repeatedly, it’s not fair to take this single small study performed in Switzerland and extrapolate that as the rate of misdiagnoses-related death for every admitted patient across the entire US.
Finally, one of the main conclusions from this study was that misdiagnosis happened most often when patients’ symptoms were not typical for their disease. Atypical presentations are challenging, and they are something every ER doc strives to avoid missing.
points out in his newsletter, the flip side of this issue is being judicious with testing — overtesting can result in harms too.
Breaking: Government report on ER misdiagnoses has “fatal flaw,” internal analysis found.
A report which has made shock waves in the US medical world had flimsy methods on key outcomes, which peer reviewers and technical experts worried about prior to publication.
Jeremy Faust, MD Dec 17
Every day on shift, we have discussions of who to test and who not to. We hate missing diagnoses, but we also can’t test everyone for everything. Focusing only on misdiagnosis of atypical symptoms, and not considering the harms of over-testing, doesn’t paint a complete picture.
In conclusion, the catchy “250,000 people die from misdiagnosis in US ERs” is based on: -one Canadian study where one person died With a guesstimated confidence interval based on:
-medicare data that requires ER docs be clairvoyant -one small Swiss study (33 deaths) that didn’t look at details of medical records (and only included admitted patients) -one small Spanish study (3 deaths), that didn’t even report if deaths were related to misdiagnosis
Call me underwhelmed.
I know this thread is way too long so here it is as a blog post:
No, ER misdiagnoses are not killing 250,000 per year
- December 18, 2022
By Kristen Panthagani, MD, PhD