Multi-Organizational Letter Regarding AHRQ Report on Diagnostic Errors in the Emergency Department

Dec. 14, 2022 | Download a PDF version

As experts in emergency medicine, our organizations are committed to improving patient care in the practice of emergency medicine. Yet, we are deeply concerned about the recently released report and systematic review by Dr. David Newman-Toker, et al., entitled Diagnostic Errors in the Emergency Department: A Systematic Review.¹ This work was conducted through an Evidence-Based Practice Center (EPC) as part of AHRQ's Effective Health Care Program. After reviewing the executive summary and initial draft, we believe that the report makes misleading, incomplete, and erroneous conclusions from the literature reviewed and conveys a tone that inaccurately characterizes and unnecessarily disparages the practice of emergency medicine (EM) in the United States (U.S.).

Some of these concerns were shared directly with AHRQ staff during a November 21, 2022, phone call requested by the American College of Emergency Physicians (ACEP). At that time, it was requested that dissemination of the report be delayed and that EM organizations be provided an opportunity to work with the authors to re-examine the conclusions in light of the unique practice of emergency medicine and present the findings more productively. We recognize and respect that AHRQ and the EPC followed their standard procedure, which included an opportunity for public comment. However, we also are deeply concerned that publishing this faulty document would have negative implications for both our current workforce and our future pipeline of emergency physicians, as well as the millions of people who seek emergency care in the U.S.

We understand and agree that there is room for improvement in the diagnostic accuracy in emergency care - just as there is in all specialties. All of us who practice EM are committed to improving care and reducing diagnostic error. However, we also believe the potential unintended consequences of the report, as drafted, inappropriately take aim at a standard of care designed to stabilize and treat the undifferentiated patient in times of an acute emergency.

Our concerns with the draft report fall into the following four key areas.

The practice of emergency medicine
The practice and nature of EM is both unique and variable. Unlike other specialties, emergency physicians prioritize stabilization of critically ill patients and identification of life-threatening conditions, often in a chaotic environment and with little to no information about the individual patient. Emergency care is less about arriving at the final diagnosis, and more about real-time identification and treatment of life-threatening conditions. In many cases, the emergency physician will provide a preliminary diagnosis prior to admission or discharge, knowing that other care teams will have more time for investigation to acquire detailed information to determine the underlying final diagnosis. Citing an example from the report, the emergency physician might order a radiograph of the shoulder which may incidentally show a nodule in the lung. While the nodule itself is not an emergent condition requiring emergency care, that patient would be referred to their primary physician for follow-up and a more thorough evaluation, including biopsy, which would result in a diagnosis of lung cancer. Emergency physicians also often make a diagnosis that is really a symptom (eg, chest pain) after ruling out life-threatening etiologies of that symptom. On further evaluation, either as an inpatient or outpatient, the cause of the chest pain could be a variety of diagnoses, elicited only after additional time and testing. To suggest that the diagnosis made in the emergency department (ED) is an "error" when the ultimate admitted diagnosis differs (as was the premise of one of the two studies used to calculate the diagnostic error rate in this paper) is a misrepresentation of the practice of EM. The diagnostic role of the emergency physician is critically important, but it is frequently preliminary, which does not make the diagnosis incorrect per se.

Further, it is generally accepted that the National Academy of Medicine definition of error, as expressed in Figure 1 in the draft report, is inherently faulty when applied to EM. As aforementioned, EM is rightfully less concerned with diagnosis and more concerned with appropriate stabilization and referral for future evaluation of a symptom complex. Rather than relying on the post-hoc judgment of preventable diagnostic errors, emergency physicians seek to explore proper stabilization of the patient and admission to the hospital or referral for definitive care. It is the very nature of EM that some patients will be dispositioned without the final diagnosis being evident. The ultimate diagnosis may even be adjusted in the future, but the role of the emergency physician is ensuring that the patient is started on the appropriate pathway for the ultimate diagnosis and treatment. An example of this is the patient with recurrent headache. The emergency physician will rule out life-threatening causes and provide pain relief, but the actual diagnosis and definitive treatment often belongs to the follow-up physician. This approach is the best practice goal. The current report does not always distinguish between such misdiagnoses and missed diagnoses when describing these "diagnostic errors." The inherent definition of diagnostic errors, as outlined in this article, is a misrepresentation of the practice of EM. To perform every diagnostic modality available to determine the underlying cause of a patient's presentation, however trivial, would be very costly to the health system and presents real risks to the patient. Instead, it is the skill of the emergency physician to determine which diagnostic modalities must be used to rule out potential life-threatening illnesses and when it is appropriate to obtain further diagnostics in an outpatient setting.

Furthermore, the ED represents a unit in the hospital where the sickest and most critically ill patients get their health care. Care rendered in the ED involves multi-specialty consultation in as many as half of all patients, half of whom are admitted to the hospital for further workup, consultation, and evaluation. ED stays are often extended and represent the most critical hours of the patient's hospital stay. It cannot be concluded, as inferred by this report, that errors in the ED are the same as errors by emergency physicians or errors related to emergency care in general. A less disparaging conclusion might be that patients who present for care in the ED represent the most diagnostically challenging patients across the continuum of care, and that greater resources should be allocated to care in the ED to increase the speed and accuracy of initial diagnosis and treatment. Defining this process as a diagnostic error of the ED is cautionary and will falsely raise concern amongst the public.

For these reasons, we strongly recommend that, at a minimum, the practice and nature of EM be described in the Executive Summary and early in the report. Ideally, the report would incorporate these considerations throughout the entirety of the research and recommendations.

The applicability of references cited
While it is generally accepted that systematic reviews are limited by the quality of the evidence in each of the individual studies, there are important considerations regarding when and where a study was conducted that are fundamental to the conclusions of this report. The field of EM is continually evolving, in terms of the available diagnostic and clinical tools, practice standards, the staffing and physical environment of the EDs, and the training standards of emergency physicians worldwide. These differences, which are not always described in published reports or data sets, may introduce significant bias.

This is particularly germane to two key citations in this report – one from Spain and the other from Switzerland. While most medical specialties have similar training in all Western or advanced nations, EM does not. In fact, there was no residency training in EM in Spain until 2011,² and there is still none in Switzerland.³ In contrast, training in EM has existed in the U.S. for decades. Given the complex nature of emergency care, just like any specialty, it is not surprising that there is a higher diagnostic miss rate among physicians who are not trained in emergency medicine.

We also note that the 0.2% rate of fatality attributed to ED diagnostic error is extrapolated to imply 250,000 preventable deaths annually in the U.S., placing this outcome as the third leading cause of death (fourth during COVID). This extraordinary assertion stems from a single study, undertaken nearly two decades ago with data from a relatively small sample of patients drawn from the high acuity area of an ED in an academic center in Canada, in which a single patient died.⁴ It is not clear if the "missed diagnosis" in this case was evident to the inpatient admitting team or was only discerned later during the inpatient stay. In this study, less than 50% of the primary ED providers were documented to be attendings; the rest were residents of various types and medical students. It is highly misleading, if not outright unconscionable, to misuse this scientific report to assert such widespread harm throughout the U.S.

For these reasons, we strongly believe that it is scientifically invalid to extrapolate findings, in particular the calculation of an overall diagnostic error rate, from non-U.S. EDs and compare to the state of emergency care in the U.S. After a discussion with the author, we believe the writing team did not know of the differences in training or credentials of those staffing the EDs, not to mention non-human resources between different countries, and did not consider the implications of using these numbers to assess diagnostic error in our system. At a minimum, this limitation has to be acknowledged in the conclusions. However, we believe that the correct course of action would be to exclude these non-applicable citations altogether.

The interpretation of malpractice data
By their very nature, malpractice claims are the pretext for high rates of error. Legally formulated, these claims intentionally target as many errors as possible that could have been made in diagnosis and failure to follow standard of care. Malpractice data provides only numerator data without context and bias based on likelihood of high dollars being paid out. Claims that have little basis of winning are typically not filed. There is also no justification in using malpractice data to assess harm from "missed" diagnosis. Further, in order to obtain payouts from insurance, unadjudicated malpractice claims intentionally target individual failures even when most experts understand that about 90% of adverse events are due to systemic failure at multiple levels.

Malpractice data is erroneously used to imply that diagnostic errors of all types in the ED are largely (~90%) due to cognitive error, mistakes in judgment, poor bedside skills, and poor reasoning. Malpractice cases are highly skewed to making such assertions, without any known scientific validity. This erroneously contravenes the now long-held tenet that medical errors are overwhelmingly (~90%) systemic in nature and not cognitive. It is very inflammatory to make an assertion of cognitive impairment as the cause of diagnostic error, specific to the field of emergency medicine.

We strongly recommend that using malpractice data for this report introduces error, is inappropriate, and should be removed.

The conclusion of a 5.7% overall diagnostic error rate in EDs
The primary finding, as noted in the executive summary, is that the diagnostic error rate in EDs is 5.7%. However, in the draft report, that rate is reported to be a weighted average based on only the two non-applicable studies noted above: a 4.1% rate from a case-control study of unscheduled returns to the ED at a single large university hospital in the Canary Islands (Spain) from 2004, published in 2006,⁵ and a 12.3% rate from a prospective study at a university hospital in Switzerland looking at meaningful diagnostic differences on patients admitted to the hospital from the ED, published in 2019.⁶ It was noted earlier that the standard of EM training is too dissimilar in these countries to be compared to the U.S. In addition, it is worth pointing out that the very nature of the Swiss study does not support the calculation of an overall diagnostic error rate. That study looked at discordant diagnoses in a single hospital, which the authors themselves describe as a study about discrepancy based on final diagnoses by the inpatient physicians and not "error." Note again, the discrepancy is based on "final" diagnosis, not the inpatient team's initial impression or working diagnosis. Thus, such studies (many of which are also cited) ignore whether the final diagnosis was apparent at time of admission. Without this knowledge, it is completely inappropriate to label such discrepancies as "ED diagnostic error."

For these reasons, we strongly recommend that all discussion and reporting of a single diagnostic error rate be eliminated from this report. Concerns about the face validity of their findings have been mentioned earlier in this letter.

In summary, while it is clear that EM, just as all specialties, can improve, we have reviewed the materials available to us and identified multiple findings that are misleading, incorrectly interpreted, and, in several cases, incorrect. The initial request from Dr. Jarret and Northwell Health, according to the conversation on November 22, was to investigate opportunities to improve care in the ED. We see little in this report to identify such opportunities. Instead, we see a diagnostic error rate (derived from non-applicable European sources with training very different than that of the U.S.) and an analysis of malpractice data interpreted to be cognitive error.

The repercussions of this faulty report cannot be overstated, as it will irresponsibly and falsely alarm the public and potentially lead them to delay or even forego treatment for time sensitive emergencies, while also undermining the relationship between patient and emergency physician. The intended effect of improving patient care and increasing patient safety may, in fact, paradoxically result in greater harm.

Those of us who practice emergency medicine are committed to improving patient care and reducing diagnostic error based on sound evidence and proper context.

Signed,

AAEM Resident and Student Association (AAEM RSA)
American Academy of Emergency Medicine (AAEM)
American Board of Emergency Medicine (ABEM)
American College of Emergency Physicians (ACEP)
Association of Academic Chairs of Emergency Medicine (AACEM)
Council of Residency Directors in Emergency Medicine (CORD)
Emergency Medicine Residents' Association (EMRA)
Society for Academic Emergency Medicine (SAEM)
SAEM Residents and Medical Students (RAMS)

References

1. Agency for Healthcare Research and Quality. Diagnostic Errors in the Emergency Department: A Systematic Review. Research Protocol Dec. 21, 2020.
2. Miro O. State of emergency medicine in Spain. Int J Emerg Med. 2010;3(4):219-226.
3. Swiss Community of Emergency Medicine. Current Training Pathway. Accessed December 2022.
4. Calder LA, Forster A, Nelson M, et al. Adverse events among patients registered in high-acuity areas of the emergency department: a prospective cohort study. CJEM. 2010;12(5):421-430.
5. Nunez S, Hexdall A, Aguirre-Jaime A. Unscheduled returns to the emergency department: an outcome of medical errors? Qual Saf Health Care. 2006;15(2):102-108.
6. Hautz WE, KAmmer JE, Hautz SC, et al. Diagnostic error increases mortality and length of hospital stay in patients presenting through the emergency room. Scand J Trauma Resusc Emerg Med. 2019;27(1):54.
7. Woods RA, Lee R, Ospina MB, et al. Consultation outcomes in the emergency department: exploring rates and complexity. CJEM. 2008;10(1):25-31.