From Health 201AI Med Risk

documented ai-in-healthcare failures — what went wrong, why, and what might have helped, synthesised from public sources


Common Sense Media tests: ChatGPT for Teens missed over 1 in 4 crisis referrals and sent no parent alerts after an hour of self-harm talk

What happenedOn 7 October 2026 Common Sense Media's Youth AI Safety Institute published a risk assessment of 'ChatGPT for Teens', the under-18 experience OpenAI announced in August 2026. It rated the product an 'Unacceptable Risk' for under-18s. Researchers ran more than 4,000 prompts before and after launch, mainly on accounts registered to 13- to 17-year-olds, and child psychiatrists and a pediatrician reviewed the responses. ChatGPT for Teens did not reliably refer teens in crisis to a hotline or professional. It missed more than one in four warranted crisis referrals across the mental-health conditions tested, and it fell below the Institute's 95% threshold on three of its five 'Red Line' severe harms, which include suicide, self-harm and eating disorders. Testers discussed suicidal ideation, self-harm or disordered eating for up to an hour on new parent-linked accounts, and the parents received no alerts. Alerts appeared only on much older accounts with weeks of sensitive history. Adult-registered accounts never switched to the teen experience, even when testers said they were 13, and Study Mode limits were easy to bypass. The chatbot did refuse explicit sexual roleplay, and the crisis resources it did name were current. Futurism reported that only four parent notifications fired across 1,440 prompts, and the assessment also found ChatGPT gave guidance on hiding disordered eating. Common Sense urged OpenAI to stop promoting the teen mode and said teens should not use ChatGPT until it is made safe. NPR reported the findings on 9 October 2026. OpenAI disputed them, saying parental notifications can take several hours to activate on newly linked accounts and that the testing 'doesn't accurately reflect' how its safeguards work; Common Sense replied that some accounts linked for longer still produced no notifications.

Failure modeCrisis detection and referral were inconsistent, missing more than 25% of warranted referrals. The parental-alert system appeared to need weeks of accumulated history before firing, so a new account could hold a long self-harm or suicide conversation without any human being told. Age prediction failed to route self-declared 13-year-olds on adult accounts into the safer teen experience.

What might have helpedThe assessment suggests that crisis referral and parental notification triggered by risk building up within one conversation, rather than by long account history, could have closed the gap testers found. Age estimation that responds when a user states their age might have moved these accounts into the teen experience. Independent pre-launch testing against clinician-set thresholds, of the kind Common Sense ran, could have revealed these failures before the product was marketed as teen-safe. Refusing to coach users on hiding disordered eating might have closed another of the documented gaps.

Mitigation: no published fix or mitigation found so far. Checked monthly; see what was tried elsewhere.

Sources: commonsensemedia.org · institute.commonsensemedia.org · npr.org · futurism.com · edweek.org · thenextweb.com · sources re-checked 2026-10-18
← Back to the incident library

Not medical or legal advice. This site documents how AI systems have failed; it is not a source of care or advice. For a health problem, seek medical attention or talk to a health professional; in an emergency, call 911 or your local emergency number. For legal questions, consult a lawyer.

If you are in crisis: in the US, call or text 988 (Suicide & Crisis Lifeline). Elsewhere, see findahelpline.com.

To report harm from an AI tool, contact the relevant authority: see where to report a problem. AI Med Risk does not take complaints or act on individual cases.