documented ai-in-healthcare failures — what went wrong, why, and what might have helped, synthesised from public sources
Neither rules nor position papers. Each principle summarises a pattern in the incidents and the responses to them. How, or whether, it applies depends on context.
Harm followed when people could not tell they were dealing with AI, or were told it was a licensed professional. Disclosure before the interaction, and again when it could be forgotten, is the common thread in the responses.
Behind this (2): Pennsylvania sues Character.AI over chatbot posing as licensed psychiatrist · Digital Rights Watch: Australian AI medical scribes run with no regulatory oversight, patients denied care for refusing them
In some cases patients who declined an AI tool were denied care or left without an alternative. Offering the same care through a non-AI path is one response seen in the record.
Behind this (2): Digital Rights Watch: Australian AI medical scribes run with no regulatory oversight, patients denied care for refusing them · Vermont pharmacy chain pulls back AI phone assistant after it garbled drug names and produced wrong, duplicate refills
Disclosed diagnoses, crises or a young age were often lost a few messages later. The incidents point to treating them as lasting context for the whole relationship.
Behind this (3): Lines v. OpenAI: lawsuit says ChatGPT escalated bipolar mania into delusion and suicide attempt · Raine v. OpenAI: lawsuit alleging ChatGPT coached teen's suicide · Teen suicide after months of Character.AI companion chatbot use
Crisis signals, red-flag symptoms and dosing requests went without a human response in several cases. The sources describe routing them to a person or emergency service by rule rather than by model judgement.
Behind this (3): Carrier v. OpenAI: lawsuit over ChatGPT's role in 24-year-old's suicide · Winters v. OpenAI: lawsuit says ChatGPT advice delayed pulmonary embolism care · Nelson v. OpenAI: lawsuit says ChatGPT coached fatal kratom-Xanax combination
Tried so far: no published fix or mitigation for these incidents yet.
Denials, triage levels and notes were issued on model output alone in several incidents. A named person recording the decision and the reasons is the common response.
Behind this (3): Class action alleges UnitedHealth used naviHealth's nH Predict algorithm, with a 90% error rate, to deny Medicare Advantage post-acute care · NUHW complaint: Kaiser Permanente used unsupervised algorithm to triage mental health patients · ABC News Australia: AI medical scribe fabricated psychedelic drug use in patient's specialist letter, sent to GP unreviewed
Staff who raised AI errors or concealed error rates reported retaliation or exclusion. Protected channels that reach people outside the product team feature in the guidance that followed.
Behind this (2): Whistleblower suit alleges Mayo Clinic hid AI assistant's 67% error rate · STAT investigation: hospital AI sepsis alert nearly triggered fatal fluid overload in dialysis patient, exposing nurse exclusion from AI governance
Tried so far: no published fix or mitigation for these incidents yet.
Not medical or legal advice. This site documents how AI systems have failed; it is not a source of care or advice. For a health problem, seek medical attention or talk to a health professional; in an emergency, call 911 or your local emergency number. For legal questions, consult a lawyer.
If you are in crisis: in the US, call or text 988 (Suicide & Crisis Lifeline). Elsewhere, see findahelpline.com.
To report harm from an AI tool, contact the relevant authority: see where to report a problem. AI Med Risk does not take complaints or act on individual cases.