NOHARM benchmark: leading medical AI tools give severely harmful recommendations in up to 1 in 4 cases
What happenedResearchers from Stanford, Harvard and the ARISE Network posted the NOHARM benchmark on arXiv on 1 December 2025 and released updated versions through July 2026. Fortune covered the July 2026 results on 29 July 2026. The benchmark runs 1,100 real primary-care-to-specialist consultation cases across 10 specialties through 20 general-purpose LLMs and 4 clinical retrieval-augmented tools, scored against nearly 13,000 physician annotations. According to Fortune, the July 2026 run included OpenAI's GPT-5.6 Sol, Anthropic's Claude Fable 5, OpenEvidence and Doximity Ask. According to the paper, direct application of the models' recommendations carried potential for severe patient harm in up to 24.6% of cases, and errors of omission made up more than 80% of severe errors. Fortune reported that 76.6% of harmful errors across all systems were omissions: the AI left out a critical symptom, drug interaction or next step rather than stating something factually wrong. The study is a preprint and has not been peer-reviewed.
Failure modeClinical AI tools disproportionately fail by silently omitting safety-critical information rather than asserting clearly incorrect facts, making the errors far harder to catch on review since there is nothing wrong to flag — only something missing that the reviewing clinician has to already know to look for.
What might have helpedTreating clinical AI output requirements as structured, checklist-style formats with mandatory sections for red flags, contraindications, and drug interactions could have made omissions visible as blanks rather than invisible. Independent benchmarking against real-world case sets before clinical deployment might have provided a more robust validation of performance. Physician review workflows that specifically probe for what the AI didn't mention, rather than just verifying the correctness of what it did say, could have served as an additional safeguard against oversight.
Mitigation: no published fix or mitigation found so far. Checked monthly; see what was tried elsewhere.
Updates: 2026-10-13: arXiv shows NOHARM was first posted on 1 December 2025, with v4 on 13 July 2026, so 'published July 2026' and the reported date are wrong; the figures also differ by version (v1 and the current abstract say up to 24.6% severe harm and over 80% of severe errors being omissions, while Fortune reports 76.6% of harmful errors as omissions).