From Health 201AI Med Risk

What happenedIn April 2020 Google Health researchers presented a study (CHI 2020) on the first real-world deployment of their deep-learning system for detecting diabetic retinopathy, in 11 clinics in Thailand. In the lab the system had been more than 90% accurate and could return results in under 10 minutes. In the clinics, as reported by MIT Technology Review and TechCrunch, it often rejected eye photos taken in poorly lit rooms, and slow internet connections held up results. Nurses and patients were frustrated, and some patients whose images were rejected were told to see a specialist at another site, which meant extra travel. The researchers concluded that local conditions and clinic workflows had to be accounted for when deploying such tools.

Failure modeA model validated on high-quality lab images was deployed without accounting for real clinic conditions (lighting, bandwidth, workflow), so many images were rejected and screening was disrupted.

What might have helpedThe researchers' own field study suggests that testing locally in the target clinics before scale-up, checking image quality and connectivity, and designing a fallback when an image is rejected could have reduced disruption. This kind of validation outside the lab is what the study itself came to provide.