documented ai-in-healthcare failures — what went wrong, why, and what might have helped, synthesised from public sources
Each safeguard is a pattern generalized from the incident library and from what was done afterwards: the fixes that shipped, the tools that were withdrawn, the laws that followed, and what each case suggests. They describe what the record shows, not what anyone must do. The tools at the end of the page are examples of how these patterns could be used in a review. For system-level failure (power, devices, degradation tiers) see FailSystems; for legislation see Clingantry.
Safeguard 1 · Independent validation and ongoing monitoring
Validation on a population like the one being served, by people outside the product team, with real-world performance still measured after go-live.
Models that looked good in vendor testing missed most real cases, hallucinated in approved deployments, or had error rates concealed from the people deploying them. 31 incidents point to it.
Learned from what was tried (4):
Ontario's Information and Privacy Commissioner issued guidance and a companion checklist for AI scribes on January 28, 2026, covering vendor assessment, contractual safeguards, ongoing monitoring and governance.
LessonProcurement vetting and privacy guidance did not stop approved scribes from writing wrong notes. The audit points to measuring accuracy in use and to clinician sign-off on each note.
Epic released a second-generation sepsis model built to be calibrated locally at each hospital. A multicenter prospective validation across four health systems found better discrimination than version 1 (AUROC 0.82 to 0.92).
LessonA better model did not remove the need for local validation: version 2 discriminates better overall, but accuracy and alert burden vary widely between hospitals.
Stanford Medicine apologised and dropped the formula. The new allocation ranked staff by clinical exposure: high-acuity settings (ED, ICU, COVID units) came first, then lower-risk hospital areas, then ambulatory and remote staff. Faculty who had received early doses waited until frontline residents and fellows were vaccinated. By 21 December more than 600 residents and fellows had been vaccinated, and on 22 December Stanford set up a public distribution dashboard.
LessonThe case suggests that a simple rules-based formula can go wrong in ways nobody sees until it runs on real staff lists. Testing it against the groups it is meant to protect before using it, and publishing results openly (as the dashboard later did), might have caught the problem sooner.
HHS's 2024 Section 1557 rule (45 CFR 92.210) requires hospitals, clinics, physician groups and payers that receive federal funding to make reasonable efforts to find patient care decision-support tools, including algorithms and AI, that use factors measuring race and other protected traits, and to reduce the risk of discrimination from them. The compliance deadline was 1 May 2025.
LessonThe case suggests that bias can enter through what a model is trained to predict, even when race is not an input at all. Rules that check only the input variables may miss that kind of bias; the study's authors found it by comparing outcomes for Black and white patients.
Appears in law or rule: Utah HB 452, mental health chatbots
Related checklist items: checklist local validation · checklist shadow mode
Safeguard 2 · Crisis escalation that holds up
Escalation to crisis resources and a person, based on risk that builds across a conversation or across sessions, in a way that is hard to talk around.
The most common failure in the library. Systems handled single risky messages but missed risk that built up over weeks, or were argued out of escalating by role-play, fiction or rapport. 20 incidents point to it.
Learned from what was tried (3):
OpenAI began routing conversations that show acute distress to reasoning models regardless of the model the user picked (announced September 2, rolled out September 29, 2025), launched parental controls that can notify a parent when a linked teen may be in crisis (September 2025, expanded July 2026), and added age prediction that applies under-18 safeguards automatically (January 2026).
LessonRouting, parental alerts and age prediction arrived only after lawsuits, and they cover teens best. The sequence suggests crisis escalation works better built in for every user from launch, and evaluated independently, than added under legal pressure.
Character.AI ended open-ended chat with characters for users under 18 on November 25, 2025, limiting teens to non-chat features, after announcing the change in October 2025.
LessonWhen open-ended companionship could not be made safe for a vulnerable group, removing that group's access was the response. It depends on age assurance, though, and does nothing for vulnerable adults.
After the death was reported, Chai Research added a crisis-intervention feature that shows helpful text and points users toward a suicide hotline when they raise unsafe topics. The co-founder told Vice the company 'worked around the clock' to add it.
LessonThe case suggests that a crisis banner added after a death does not stop a companion bot from giving harmful content when a user asks outright. A quick journalist test found that gap right away.
Appears in law or rule: California SB 243, companion chatbots · Utah HB 452, mental health chatbots
Related checklist items: checklist crisis stop · checklist no endless persona · checklist remembers risk
Safeguard 3 · Clinician review as a real checkpoint
A named clinician reviewing AI output before it becomes a decision, a note or a denial, with a quick way to override the model without penalty.
Review that existed on paper did not stop harm: denials issued on model output, scribe letters sent unread, triage scores that bedside staff could not overrule. 19 incidents point to it.
Learned from what was tried (5):
Australia's TGA moved to compliance action in August 2026 against scribes that perform medical-device functions without registration.
LessonRegulation targets what a scribe claims to do, not whether its notes are true. In this case only the clinician's read-and-sign step could have caught the fabricated content, and it was skipped.
Kinney Drugs scaled back its AI phone assistant after hundreds of complaints, reverting to a touch-tone phone system and making AI refill texts opt-in.
LessonIn hindsight, opt-in enrolment and a non-AI fallback could have been the launch design rather than the rollback. Rising complaint volume was an early warning that was handled as a service issue rather than a safety signal.
Ontario's Information and Privacy Commissioner issued guidance and a companion checklist for AI scribes on January 28, 2026, covering vendor assessment, contractual safeguards, ongoing monitoring and governance.
LessonProcurement vetting and privacy guidance did not stop approved scribes from writing wrong notes. The audit points to measuring accuracy in use and to clinician sign-off on each note.
On February 6, 2024, CMS issued an FAQ under its Contract Year 2024 Medicare Advantage rule. It says plans must base coverage decisions on the individual patient's circumstances, and that an algorithm that determines coverage from a larger data set instead of the patient's medical history, physician's recommendations or clinical notes is not compliant.
LessonThe rule now says an algorithm cannot decide coverage on its own, but without enforcement the remaining safeguards are the insurer's own individual clinician review and its monitoring of appeal overturns.
California's Physicians Make Decisions Act (SB 1120) took effect on 1 January 2025. It bars health plans and disability insurers from using AI, algorithms or other software to deny, delay or modify care on medical-necessity grounds; a licensed physician or other qualified professional must make that decision. The tools must also be applied fairly and equitably and are open to audit by the DMHC and the Department of Insurance.
LessonThe case suggests that simply having a physician sign off doesn't make the review meaningful. The California law requires a professional to decide but leaves open how closely they must actually look at each case, which may be where the next dispute arises.
Appears in law or rule: Illinois HB 1806, Wellness and Oversight for Psychological Resources Act
Related checklist items: checklist override · checklist contraindications · checklist draft until signed
Safeguard 4 · Firm limits on specific dangerous content
Refusing or rerouting a short list of content (self-harm methods and notes, controlled-substance dosing, claimed professional credentials, reassurance about red-flag symptoms) regardless of how the request is framed.
In each case the harm came from one concrete piece of content that a fixed rule would have blocked, regardless of conversational context. 18 incidents point to it.
Learned from what was tried (0):
No published fix or mitigation has put this into practice yet; it rests on the incidents where it was missing.
Related checklist items: checklist no dosing · checklist see someone
Safeguard 5 · Disclosure, consent and a non-AI option
Telling people when AI is involved, not presenting it as a licensed professional, and keeping a working non-AI pathway for anyone who declines.
Patients were auto-enrolled in AI tools they could not identify, denied care for refusing scribes, or steered by sources presented as neutral. 11 incidents point to it.
Learned from what was tried (3):
Kinney Drugs scaled back its AI phone assistant after hundreds of complaints, reverting to a touch-tone phone system and making AI refill texts opt-in.
LessonIn hindsight, opt-in enrolment and a non-AI fallback could have been the launch design rather than the rollback. Rising complaint volume was an early warning that was handled as a service issue rather than a safety signal.
Character.AI ended open-ended chat with characters for users under 18 on November 25, 2025, limiting teens to non-chat features, after announcing the change in October 2025.
LessonWhen open-ended companionship could not be made safe for a vulnerable group, removing that group's access was the response. It depends on age assurance, though, and does nothing for vulnerable adults.
Under a class settlement approved in 2016, Idaho's Department of Health and Welfare agreed to build a new budget system (a 'supports budget model') with input from the ACLU, participants and families. It also agreed to let people see their assessment documents, fund representatives for budget appeals, allow increases for documented health or safety needs, and keep each participant's highest past budget in place until the new model is ready.
LessonThe case suggests that a court order to rebuild an automated tool can take years longer than planned. The interim measures, such as disclosure, funded appeals and frozen budgets, did more for the people affected than the promised new model has so far.
Appears in law or rule: California SB 243, companion chatbots · Illinois HB 1806, Wellness and Oversight for Psychological Resources Act · Utah HB 452, mental health chatbots
Related checklist items: checklist says its ai · checklist names sources · checklist consent and fallback
Safeguard 6 · Change control and independent sign-off
Clinical re-review after model, vendor or configuration changes, with safety thresholds set by people outside the product team.
Harm followed silent updates, tools released before regulatory review, and teams allowed to self-certify their own error rates. 8 incidents point to it.
Learned from what was tried (5):
Australia's TGA moved to compliance action in August 2026 against scribes that perform medical-device functions, such as suggesting diagnoses or treatment, without registration.
LessonRegulators are acting on scribes that make clinical suggestions, but plain transcription scribes stay unregulated. Consent and a non-AI path therefore depend on each clinic's own policy.
Copan announced FDA 510(k) clearance (K251511) for PhenoMATRIX on February 17, 2026, bringing the software under FDA authorisation. The cleared US use leaves final image assessment and result decisions to trained laboratory personnel.
LessonAuthorisation came after the fact, and patient samples in US labs were read by unauthorised AI before the recall. Checking regulatory status before go-live, rather than after a recall, would have closed that gap.
NEDA took the Tessa chatbot offline after it gave weight-loss advice to people seeking eating-disorder support.
LessonPulling the tool stopped the harm but exposed the gap: the human helpline it replaced was already gone. The case suggests keeping a human service running until an AI tool has proven itself.
Stanford Medicine apologised and dropped the formula. The new allocation ranked staff by clinical exposure: high-acuity settings (ED, ICU, COVID units) came first, then lower-risk hospital areas, then ambulatory and remote staff. Faculty who had received early doses waited until frontline residents and fellows were vaccinated. By 21 December more than 600 residents and fellows had been vaccinated, and on 22 December Stanford set up a public distribution dashboard.
LessonThe case suggests that a simple rules-based formula can go wrong in ways nobody sees until it runs on real staff lists. Testing it against the groups it is meant to protect before using it, and publishing results openly (as the dashboard later did), might have caught the problem sooner.
Under a class settlement approved in 2016, Idaho's Department of Health and Welfare agreed to build a new budget system (a 'supports budget model') with input from the ACLU, participants and families. It also agreed to let people see their assessment documents, fund representatives for budget appeals, allow increases for documented health or safety needs, and keep each participant's highest past budget in place until the new model is ready.
LessonThe case suggests that a court order to rebuild an automated tool can take years longer than planned. The interim measures, such as disclosure, funded appeals and frozen budgets, did more for the people affected than the promised new model has so far.
Related checklist items: checklist regulatory status · checklist change review
Example tools built on these patterns
Signs worth looking for in a health chatbot, and questions a clinic might ask before an AI tool reaches patients. Printable.
Example multi-turn scenarios, each reproducing a real failure, with behaviour to look for. Download as JSON.
Questions drawn from the incidents for before go-live, signals they point to afterwards, and related standards and laws.
Six principles drawn from the incidents: disclosure, a way out, persistent vulnerability, hand-off triggers, a person owns the decision, protected reporting.
Not medical or legal advice. This site documents how AI systems have failed; it is not a source of care or advice. For a health problem, seek medical attention or talk to a health professional; in an emergency, call 911 or your local emergency number. For legal questions, consult a lawyer.
If you are in crisis: in the US, call or text 988 (Suicide & Crisis Lifeline). Elsewhere, see findahelpline.com.
To report harm from an AI tool, contact the relevant authority: see where to report a problem. AI Med Risk does not take complaints or act on individual cases.