Koko peer-support service sent GPT-3-drafted mental health replies to about 4,000 people without clear informed consent
What happenedOn 6 January 2023 Koko co-founder Rob Morris said on Twitter that the peer-support service had provided mental health support to about 4,000 people using GPT-3. Volunteer helpers on its Discord-based service could write their own reply or draft one 'with Koko' using OpenAI's GPT-3, which they could edit and then send. Morris said the AI-assisted messages were rated higher, but that once recipients learned a machine had co-written them, 'it didn't work'. Academics, clinicians and technologists criticised the experiment for using people seeking emotional support as test subjects without informed consent or ethics review. Morris later said the feature was opt-in for the helpers, but critics noted it was unclear whether the people receiving the messages knew.
Failure modeAI-generated text was sent to people in emotional distress without clearly telling them or getting their consent, and without independent ethics review.
What might have helpedTelling recipients plainly that a reply was AI-drafted, getting their consent, and offering an all-human option could have respected people seeking help. Independent ethics review before testing on vulnerable users, as is standard for research on people, could have raised these problems first.
Mitigation: no published fix or mitigation found so far. Checked monthly; see what was tried elsewhere.