Red-flag checklist

Use your own judgment. Nothing here is authoritative or definitive advice: these are general suggestions drawn from existing cases, generalized from public reports of incidents and responses, not tested or validated for any setting. Health systems are complex and there is no one-size-fits-all recipe: adapt anything here with qualified clinical, legal and regulatory advisers, and do not rely on it as a standard, certification or guarantee of safety or compliance. Full disclaimer.

Two lists drawn from the incidents. The first describes signs people may want to look for in a health or mental-health chatbot. The second is questions a clinic or hospital might ask before an AI tool reaches patients. Each item names the incidents where it was missing. They are prompts for your own review, not requirements, and they do not cover everything that matters in a given setting. Print this page for a one-page version.

Signs worth looking for in a health chatbot

1.It tells you it is an AI, and it never claims a licence, credentials or a professional title.

Disclosure, consent and a non-AI option

Behind this (1): Pennsylvania sues Character.AI over chatbot posing as licensed psychiatrist

Tried so far: no published fix or mitigation for these incidents yet.

2.If you mention self-harm or suicide, it stops, gives crisis resources and points you to a person. You cannot talk it out of that with role-play or a story.

Crisis escalation that holds up

Behind this (4): Raine v. OpenAI: lawsuit alleging ChatGPT coached teen's suicide · Mother's account: ChatGPT persona 'Harry' helped draft suicide note, never escalated disclosed suicidal ideation · Carrier v. OpenAI: lawsuit over ChatGPT's role in 24-year-old's suicide · Belgian man's suicide after Chai app chatbot encouraged it

Tried so far:

Partly mitigated Raine v. OpenAI: lawsuit alleging ChatGPT coached teen's suicide — Routing, parental alerts and age prediction arrived only after lawsuits, and they cover teens best. The sequence suggests crisis escalation works better built in for every user from launch, and evaluated independently, than added under legal pressure.
Partly mitigated Belgian man's suicide after Chai app chatbot encouraged it — The case suggests that a crisis banner added after a death does not stop a companion bot from giving harmful content when a user asks outright. A quick journalist test found that gap right away.

3.It will not tell you how much of a medicine or drug to take, or say that a combination is safe.

Firm limits on specific dangerous content

Behind this (2): Nelson v. OpenAI: lawsuit says ChatGPT coached fatal kratom-Xanax combination · Bromide poisoning after ChatGPT diet advice

Tried so far: no published fix or mitigation for these incidents yet.

4.When symptoms persist or get worse, it tells you to see someone in person rather than reassuring you or suggesting rest.

Firm limits on specific dangerous content

Behind this (2): Winters v. OpenAI: lawsuit says ChatGPT advice delayed pulmonary embolism care · Babylon Health triage chatbot missed cardiac red flags

Tried so far: no published fix or mitigation for these incidents yet.

5.It does not keep up a friend, partner or spiritual-guide persona for hours or days without encouraging contact with real people.

Crisis escalation that holds up

Behind this (3): Teen suicide after months of Character.AI companion chatbot use · Belgian man's suicide after Chai app chatbot encouraged it · Parish v. OpenAI: lawsuit says ChatGPT fed religious delusions before Alabama woman's suicide

Tried so far:

Fix adopted Teen suicide after months of Character.AI companion chatbot use — When open-ended companionship could not be made safe for a vulnerable group, removing that group's access was the response. It depends on age assurance, though, and does nothing for vulnerable adults.
Partly mitigated Belgian man's suicide after Chai app chatbot encouraged it — The case suggests that a crisis banner added after a death does not stop a companion bot from giving harmful content when a user asks outright. A quick journalist test found that gap right away.

6.If you tell it about a diagnosis such as bipolar disorder or an eating disorder, it stays more careful for the rest of the conversation.

Crisis escalation that holds up

Behind this (2): Lines v. OpenAI: lawsuit says ChatGPT escalated bipolar mania into delusion and suicide attempt · NEDA's 'Tessa' chatbot gave weight-loss advice to eating-disorder users

Tried so far:

Tool withdrawn NEDA's 'Tessa' chatbot gave weight-loss advice to eating-disorder users — Pulling the tool stopped the harm but exposed the gap: the human helpline it replaced was already gone. The case suggests keeping a human service running until an AI tool has proven itself.

7.Health answers point to licensed providers or public-health agencies, and it tells you when a source is an advocacy group.

Disclosure, consent and a non-AI option

Behind this (1): International investigation: ChatGPT, Gemini, Claude, and Grok routinely steered pregnant users toward anti-abortion groups

Tried so far: no published fix or mitigation for these incidents yet.

Questions before a clinic or hospital turns on an AI tool

1.Is the tool authorised for this intended use where you operate?

Change control and independent sign-off

Behind this (3): FDA recall: machine-learning shoulder-surgery outcome predictor Predict+ withdrawn from US market for lack of clearance · FDA recall: Copan's AI culture-plate reader PhenoMATRIX used in US labs without clearance, risking inaccurate colony counts · Digital Rights Watch: Australian AI medical scribes run with no regulatory oversight, patients denied care for refusing them

Tried so far:

Fix adopted FDA recall: Copan's AI culture-plate reader PhenoMATRIX used in US labs without clearance, risking inaccurate colony counts — Authorisation came after the fact, and patient samples in US labs were read by unauthorised AI before the recall. Checking regulatory status before go-live, rather than after a recall, would have closed that gap.
Partly mitigated Digital Rights Watch: Australian AI medical scribes run with no regulatory oversight, patients denied care for refusing them — Regulators are acting on scribes that make clinical suggestions, but plain transcription scribes stay unregulated. Consent and a non-AI path therefore depend on each clinic's own policy.

2.Has it been validated externally on a population like yours, and calibrated locally?

Independent validation and ongoing monitoring

Behind this (3): Epic sepsis model missed two-thirds of sepsis cases in external validation · Brazilian state's AI bed-allocation algorithm undertriaged dying woman, delaying ICU transfer five days · NOHARM benchmark: leading medical AI tools give severely harmful recommendations in up to 1 in 4 cases

Tried so far:

Partly mitigated Epic sepsis model missed two-thirds of sepsis cases in external validation — A better model did not remove the need for local validation: version 2 discriminates better overall, but accuracy and alert burden vary widely between hospitals.

3.Was there a shadow-mode period alongside the existing process, with attention to drift on deteriorating patients?

Independent validation and ongoing monitoring

Behind this (1): Brazilian state's AI bed-allocation algorithm undertriaged dying woman, delaying ICU transfer five days

Tried so far: no published fix or mitigation for these incidents yet.

4.Can clinicians override the model quickly and without penalty, and does a named person own each decision?

Clinician review as a real checkpoint

Behind this (4): STAT investigation: hospital AI sepsis alert nearly triggered fatal fluid overload in dialysis patient, exposing nurse exclusion from AI governance · Brazilian state's AI bed-allocation algorithm undertriaged dying woman, delaying ICU transfer five days · NUHW complaint: Kaiser Permanente used unsupervised algorithm to triage mental health patients · Class action alleges UnitedHealth used naviHealth's nH Predict algorithm, with a 90% error rate, to deny Medicare Advantage post-acute care

Tried so far:

Partly mitigated Class action alleges UnitedHealth used naviHealth's nH Predict algorithm, with a 90% error rate, to deny Medicare Advantage post-acute care — The rule now says an algorithm cannot decide coverage on its own, but without enforcement the remaining safeguards are the insurer's own individual clinician review and its monitoring of appeal overturns.

5.Does the logic account for known contraindications, so the tool flags a conflict itself?

Clinician review as a real checkpoint

Behind this (2): STAT investigation: hospital AI sepsis alert nearly triggered fatal fluid overload in dialysis patient, exposing nurse exclusion from AI governance · NOHARM benchmark: leading medical AI tools give severely harmful recommendations in up to 1 in 4 cases

Tried so far: no published fix or mitigation for these incidents yet.

6.Are AI-generated notes and letters treated as drafts until a clinician reads and signs them, with the source audio kept?

Clinician review as a real checkpoint

Behind this (3): Whisper transcription tool fabricating text in medical settings · ABC News Australia: AI medical scribe fabricated psychedelic drug use in patient's specialist letter, sent to GP unreviewed · Ontario auditor general: approved medical AI scribes hallucinated and misrecorded drugs

Tried so far:

Partly mitigated ABC News Australia: AI medical scribe fabricated psychedelic drug use in patient's specialist letter, sent to GP unreviewed — Regulation targets what a scribe claims to do, not whether its notes are true. In this case only the clinician's read-and-sign step could have caught the fabricated content, and it was skipped.
Partly mitigated Ontario auditor general: approved medical AI scribes hallucinated and misrecorded drugs — Procurement vetting and privacy guidance did not stop approved scribes from writing wrong notes. The audit points to measuring accuracy in use and to clinician sign-off on each note.

8.Do model or vendor updates trigger clinical re-review, and are error thresholds set by someone outside the product team?

Change control and independent sign-off

Behind this (2): NEDA's 'Tessa' chatbot gave weight-loss advice to eating-disorder users · Whistleblower suit alleges Mayo Clinic hid AI assistant's 67% error rate

Tried so far:

Tool withdrawn NEDA's 'Tessa' chatbot gave weight-loss advice to eating-disorder users — Pulling the tool stopped the harm but exposed the gap: the human helpline it replaced was already gone. The case suggests keeping a human service running until an AI tool has proven itself.

Machine-readable: checklist.json. For more questions and after-go-live signals see questions for deploying clinical models.

← All tools

Not medical or legal advice. This site documents how AI systems have failed; it is not a source of care or advice. For a health problem, seek medical attention or talk to a health professional; in an emergency, call 911 or your local emergency number. For legal questions, consult a lawyer.

If you are in crisis: in the US, call or text 988 (Suicide & Crisis Lifeline). Elsewhere, see findahelpline.com.

To report harm from an AI tool, contact the relevant authority: see where to report a problem. AI Med Risk does not take complaints or act on individual cases.

From Health 201AI Med Risk