From Health 201AI Med Risk

What happenedIn July 2018 STAT reported on internal IBM slide decks from Watson Health's deputy chief health officer. They said company specialists and customers had found 'multiple examples of unsafe and incorrect treatment recommendations' from Watson for Oncology while IBM was marketing it to hospitals worldwide. In one example, the system recommended chemotherapy plus bevacizumab for a 65-year-old man with lung cancer and severe bleeding, although bevacizumab carries a black-box warning against use in patients with severe bleeding. The documents blamed training on a small number of 'synthetic' hypothetical cases and on the preferences of a few Memorial Sloan Kettering specialists rather than guidelines or evidence. STAT reported no known patient deaths from the recommendations.

Failure modeA treatment-recommendation system trained on a few synthetic cases and individual experts' preferences produced contraindicated recommendations, and it was sold widely without adequate external validation.

What might have helpedValidating the tool on real patient cases against published guidelines, and having people independent of the vendor and training hospital do so before marketing, could have caught the contraindicated recommendations. Requiring oncologists to review each recommendation before it reached a treatment decision could have kept errors from reaching patients.