NEDA's 'Tessa' chatbot gave weight-loss advice to eating-disorder users
What happenedThe US National Eating Disorders Association replaced its human helpline with the 'Tessa' chatbot, which was found giving calorie-deficit and weight-loss advice to people seeking eating-disorder support. NEDA took it offline.
Failure modeGeneric wellness content surfaced to a population for whom it is contraindicated; vendor update added generative behavior beyond the validated rule-based script; no clinical re-validation.
What might have helpedTreating contraindication checks as population-specific could have helped identify relevant risks more precisely. Implementing change control where any model or vendor update triggers a clinical re-review might have ensured ongoing safety validation. Maintaining a human fallback for high-risk populations could have provided an additional layer of oversight when automated systems were insufficient.
Tool withdrawn Since thenNEDA took the Tessa chatbot offline after it gave weight-loss advice to people seeking eating-disorder support.
Still openThe tool was withdrawn rather than fixed; the human helpline it replaced had already been closed.
LessonPulling the tool stopped the harm but exposed the gap: the human helpline it replaced was already gone. The case suggests keeping a human service running until an AI tool has proven itself.
Sources: npr.org