Epic sepsis model missed two-thirds of sepsis cases in external validation
What happenedJAMA Internal Medicine external validation of Epic's widely deployed sepsis prediction model found AUC 0.63 (vs claimed 0.76–0.83), missing 67% of sepsis cases while generating high alert volumes across hundreds of hospitals.
Failure modeProprietary model deployed at scale without independent external validation; training-label leakage (clinician suspicion baked into features); alert fatigue.
What might have helpedTreating independent external validation as a prerequisite before deployment could have provided an additional layer of scrutiny. Transparent performance reporting might have offered clearer insights into system behavior, while local recalibration could have better aligned the technology with specific environmental conditions. Monitoring real-world sensitivity, rather than focusing solely on the number of alerts fired, might have revealed nuances in performance that standard metrics overlook.
Partly mitigated Since thenEpic released a second-generation sepsis model built to be calibrated locally at each hospital. A multicenter prospective validation across four health systems found better discrimination than version 1 (AUROC 0.82 to 0.92).
Still openThe same study found low positive predictive value, high alert burden and wide variation between institutions, so each hospital still has to validate it locally.
LessonA better model did not remove the need for local validation: version 2 discriminates better overall, but accuracy and alert burden vary widely between hospitals.
Sources: pmc.ncbi.nlm.nih.gov · jamanetwork.com