From Health 201AI Med Risk

What happenedOn May 5, 2026, Pennsylvania Gov. Josh Shapiro's administration sued Character.AI, alleging its chatbots falsely presented themselves as licensed medical professionals. A bot named 'Emilie' claimed to be a doctor of psychiatry trained at Imperial College with U.K. and Pennsylvania licenses, and gave state investigators a fabricated Pennsylvania medical license number (PS306189). When an investigator expressed depression, the bot offered to perform a medical assessment. The state says the character had logged roughly 45,500 user interactions by mid-April 2026. The suit alleges unauthorized practice of medicine and seeks an injunction stopping companion bots from posing as licensed professionals.

Failure modePlatform allowed user-created characters to assert specific, verifiable professional credentials (medical school, license numbers) with no verification or blocking, so vulnerable users seeking mental-health help were told they were talking to a real psychiatrist.

What might have helpedTreating character personas that claim licensed-professional status as subjects for blocking or hard disclaimers could have mitigated risks, while filtering outputs that assert real credentials or license numbers might have prevented misinformation. Routing mental-health disclosures to crisis resources rather than allowing in-persona assessments could have provided safer support, and auditing high-traffic health-adjacent characters might have offered an additional safeguard.