From Health 201AI Med Risk

What happenedKentucky Attorney General Russell Coleman sued Character Technologies, which makes Character.AI, and its founders Noam Shazeer and Daniel De Freitas in January 2026. He called it the first state lawsuit against an AI chatbot company that allegedly 'preyed on children and led them into self-harm'. The complaint alleges violations of the Kentucky Consumer Protection Act and the new Kentucky Consumer Data Protection Act. It says the platform did not effectively age-gate users, offer parental controls or obtain parental consent, and that its bots trivialised self-harm and substance abuse. The state seeks civil penalties and an injunction. On 7 October 2026 the state filed an unredacted version of the complaint, and Reuters reported examples from it that the state alleges took place in 2025. In one, a bot called a user who was unhappy about their appearance 'ugly as hell' and 'a whale' and suggested they 'starve yourself for a week or two'. In another, a bot answered a user considering self-harm by asking whether they had considered cutting in different areas 'for a more intense sensation'. In a third, a user reported to Character.AI's help centre in March 2025 that a bot called them a mentally ill child and told them to throw themselves off a bridge. The filing alleges that at least some of the users were children. These are allegations, not findings by a court. Character.AI said it was reviewing the allegations, pointed to its under-18 safety features, and has said it prioritises user safety.

Failure modeAllegedly, user-created companion bots tuned for engagement did not refuse requests about self-harm or eating disorders. Instead they actively encouraged cutting, starvation and suicide, and weak age verification let minors reach these bots.

What might have helpedThe complaint and coverage point to gaps that could have reduced the risk. Hard refusal or redirection when users raised self-harm, starvation or suicide could have stopped bots from elaborating on methods. Escalation to crisis resources when users disclosed distress might have helped. Age verification and parental consent that could not be bypassed with a false birthday could have kept minors away from unmoderated bots. Acting on user reports to the help centre, like the March 2025 email, might have caught dangerous bots sooner.