From Health 201AI Med Risk

What happenedCMS's Wasteful and Inappropriate Service Reduction (WISeR) model launched in January 2026 and is set to run until 2031. In six states (New Jersey, Ohio, Oklahoma, Texas, Arizona and Washington), traditional Medicare providers must get prior approval for 13 services, including skin substitutes, epidural injections and kyphoplasty. Private vendors use AI systems to evaluate those requests. More than 1,000 pages of records that the Electronic Frontier Foundation obtained through a FOIA lawsuit against CMS were published on 8 September 2026. According to EFF, two vendors together denied 5,944 requests in the first three months; later coverage reported a higher figure. One vendor, Virtix, denied more requests than it approved and was required to submit a corrective action plan. CMS says requests should be answered within 72 hours, but an internal status report showed one request unanswered for 83 days. Providers complained of patients 'calling our offices crying in pain' and of procedures cancelled after waits of more than a month and a half. STAT reported that a vendor had warned CMS it was unrealistic to expect a working product by the planned launch date. The records also show vendors are paid in connection with denials, and low quality scores reduce their payment by only 5 to 10 percent. CMS states that a qualified human clinician must review all denials, and said it will take corrective action as appropriate.

Failure modeAn AI-driven coverage review was launched quickly at scale with weak vendor readiness and payment incentives tied to denials. The result was high denial rates and processing delays far beyond the stated 72-hour standard, which delayed time-sensitive procedures, while the AI's role and methods stayed largely undisclosed.

What might have helpedEFF and other commentators point out that the clinician-review requirement may mean little if AI recommendations shape human decisions; tracking how often reviewers overturn AI recommendations could show whether that review is a real check. Independent, published monitoring of denial rates, overturn-on-appeal rates and turnaround times from launch could have exposed the delays and one vendor's outlier denial rate sooner. Separating vendor pay from denials, and confirming that a system works before go-live (a vendor reportedly warned it would not), could also have reduced the risk.

What happenedA STAT News investigation published August 11, 2026 (reported by Katie Palmer, with follow-up coverage from Nurse.org and Tech Times) described an incident at St. Rose Dominican Hospital in Henderson, Nevada, where an elderly patient arrived with dangerously low blood pressure and the hospital's AI system flagged her for sepsis, triggering a protocol calling for immediate IV fluids. A bedside nurse with 14 years of experience noticed a dialysis catheter below her collarbone during his exam — flooding her with fluids risked pulmonary edema. When he raised the concern, the charge nurse initially insisted on following the AI alert; the nurse refused, and a physician intervened with dopamine instead. The broader investigation, examining 106 large U.S. health systems, found at least 26 have a named AI governance body but only six of the documented structures showed a named nurse leader publicly shaping how clinical AI is governed, despite nurses being the largest segment of the healthcare workforce and the ones most often catching what algorithms miss at the bedside.

Failure modeAn AI sepsis-alert protocol recommended a standard fluid bolus without accounting for a patient-specific contraindication (a dialysis catheter indicating renal failure, where fluid overload risks pulmonary edema), and the surrounding institutional process treated the algorithm's output as a directive to be followed rather than an input for clinical judgment — with frontline nurses, who most reliably catch this kind of contraindication, systematically excluded from the governance bodies that design and oversee these tools.

What might have helpedIntegrating known contraindications, such as renal failure or heart failure, into sepsis and fluid-bolus alert logic could have allowed the system to flag conflicts automatically, reducing reliance on clinicians to identify them retrospectively. Establishing explicit and unpenalized override authority for bedside clinicians within AI-driven protocols might have prevented a default to algorithmic compliance. Including named frontline nurse representation on clinical AI governance and procurement committees, alongside physician and IT leadership, could have provided a broader perspective in decision-making processes.

What happenedThe National Union of Healthcare Workers filed a complaint with California's Department of Managed Health Care and the U.S. Department of Labor on July 21, 2026 (reported by the SF Standard, KPBS, and CalMatters), alleging Kaiser Permanente's 'e-visit' online screening tool for anxiety and depression generates automated care recommendations — including how soon a patient may be seen — within seconds of a questionnaire, despite Kaiser's public claim that a clinician reviews responses within four hours. California law (Health and Safety Code §1367.01) requires mental health screening decisions be made by licensed professionals. Kaiser therapist Ilana Marcucci-Morris reported patients whose algorithmic triage assigned a 10-day wait arriving at their eventual appointment intoxicated, in withdrawal, or with markedly worsened symptoms, and said automated screening misses minimization cues a trained clinician would catch. Kaiser has previously paid $200 million (2023) and $31.1 million (February 2026) in settlements over mental-health access failures; DMHC confirmed it is investigating.

Failure modeAn automated triage algorithm made wait-time and care-pathway determinations for mental-health patients without real-time licensed-clinician review, missing the tone, hedging, and symptom-minimization cues clinicians rely on to catch patients who understate their risk — resulting in unsafe wait times for patients who deteriorated in the interim.

What might have helpedRouting mental-health triage decisions through licensed-clinician review before a wait time or care pathway is finalized, rather than treating it as a nominal downstream audit, could have ensured more rigorous oversight. Treating questionnaire-based self-report as a floor for urgency, rather than a ceiling, might have prevented the underestimation of patient needs. Additionally, granting independent regulator access to the underlying algorithm, instead of relying on vendor or payer self-attestation, would have provided greater transparency into the decision-making process.

What happenedTraci Tamiko Eto, Mayo Clinic's former research director and AI compliance lead, filed a civil suit (reported July 2026 by Minnesota Public Radio and others) alleging she was pushed out after raising ten whistleblower complaints about Mayo's AI program. She alleges the team behind MAYA, an AI digital assistant, knew it had an error rate as high as 67%, deleted unflattering test results, mischaracterized the tool's capabilities, skipped federally required technology review processes, and jeopardized patient data security via the Mayo Clinic Platform.

Failure modeInternal governance failure: known high error rate concealed rather than reported, validation evidence deleted, and the designated compliance function sidelined — allowing an underperforming clinical AI tool to advance despite failed testing.

What might have helpedTreating validation as an independent, non-team-controlled process with immutable test records could have provided a more objective check on system performance. Protected internal reporting channels for AI safety staff might have facilitated the early identification of concerns without fear of reprisal. Deployment gates tied to documented error-rate thresholds that product teams cannot self-certify could have ensured that release decisions were based on verified metrics rather than internal assessments.

What happenedRebeca Cardoso Tenente Molina, a 32-year-old psychologist, was hospitalized June 2, 2026 in São João Nepomuceno, Brazil for suspected gallstones. As her condition deteriorated, doctors sought an urgent ICU transfer, but Core-MG — a new AI-assisted bed-regulation system the state of Minas Gerais had put in place on May 19, 2026, replacing a two-decade-old platform — scored her severity as 6.8, which her family and treating doctors say should have been rated a 10. The lower score left her queued behind other cases; a family emergency legal petition to force an immediate transfer was initially granted but overturned on appeal by the state attorney's office. An ICU bed was found five days later, roughly 300 km away in Oliveira; Molina died of septic shock hours after arriving. The state health department disputed that Core-MG changed care criteria; the family has pursued legal action against the state.

Failure modeAn algorithmic severity score, not a treating clinician, effectively gated a time-critical ICU transfer, and doctors reported losing the authority to override the system's assessment even as lab results showed clear deterioration — collapsing a life-or-death triage decision into a single unchallenged number.

What might have helpedRetaining a fast, unambiguous override to bypass an algorithmic severity score when bedside findings or trending labs contradict it could have supported time-critical transfers. Running any new triage or bed-allocation system in shadow mode alongside the prior process before cutover, with real-time monitoring for score drift on deteriorating patients, might have provided a safer transition. An independent audit of severity-scoring logic before statewide rollout represents one safeguard that could have been implemented.

What happenedResearchers from Stanford, Harvard and the ARISE Network posted the NOHARM benchmark on arXiv on 1 December 2025 and released updated versions through July 2026. Fortune covered the July 2026 results on 29 July 2026. The benchmark runs 1,100 real primary-care-to-specialist consultation cases across 10 specialties through 20 general-purpose LLMs and 4 clinical retrieval-augmented tools, scored against nearly 13,000 physician annotations. According to Fortune, the July 2026 run included OpenAI's GPT-5.6 Sol, Anthropic's Claude Fable 5, OpenEvidence and Doximity Ask. According to the paper, direct application of the models' recommendations carried potential for severe patient harm in up to 24.6% of cases, and errors of omission made up more than 80% of severe errors. Fortune reported that 76.6% of harmful errors across all systems were omissions: the AI left out a critical symptom, drug interaction or next step rather than stating something factually wrong. The study is a preprint and has not been peer-reviewed.

Failure modeClinical AI tools disproportionately fail by silently omitting safety-critical information rather than asserting clearly incorrect facts, making the errors far harder to catch on review since there is nothing wrong to flag — only something missing that the reviewing clinician has to already know to look for.

What might have helpedTreating clinical AI output requirements as structured, checklist-style formats with mandatory sections for red flags, contraindications, and drug interactions could have made omissions visible as blanks rather than invisible. Independent benchmarking against real-world case sets before clinical deployment might have provided a more robust validation of performance. Physician review workflows that specifically probe for what the AI didn't mention, rather than just verifying the correctness of what it did say, could have served as an additional safeguard against oversight.

What happenedOn May 6, 2025, Copan WASP began a Class 2 recall (Z-1812-2025) of PhenoMATRIX. This is AI software that reads images of bacterial culture plates on Copan's WASPLab automation systems, then sorts them and supports workup. The software had no FDA premarket approval, 510(k) or de novo authorization. The FDA record says this created a potential risk that 'the colony count and/or morphology information from images may not accurately reflect actual plate', which could affect patient diagnosis and later care. Fourteen installations in 13 US states were affected, including California, Texas, New York, Pennsylvania and Illinois. Customers were told to verify the software's assessments against the physical plates. No patient harm was reported. Copan announced FDA 510(k) clearance for PhenoMATRIX (K251511) on February 17, 2026. Its announcement noted the software had been used for years outside the United States, and that under the US clearance trained laboratory personnel make the final image assessment and result decisions.

Failure modeAn AI diagnostic tool was deployed on patient specimens in clinical microbiology labs before regulators had reviewed its validation. Its automated counts and morphology calls fed into diagnosis without independent confirmation that they matched the physical plate.

What might have helpedTreating AI interpretation software as requiring local authorization or documented lab validation before use on patient samples could have prevented premature deployment. Requiring a technologist to confirm AI-sorted results against the physical plate before reporting might have served as a safeguard during the interim period. Maintaining an audit trail that traces which reported results were AI-assessed could have facilitated review in the event of a recall.

What happenedOn August 30, 2024, Blue Ortho emailed active US users that it was withdrawing Predict+ from the US market until it had regulatory approval. Predict+ is a machine-learning clinical decision support tool, marketed with Exactech's shoulder arthroplasty planning software. It predicts individual patient outcomes after shoulder replacement surgery. The FDA classified the action as a Class 2 recall (Z-0209-2025) 'for lack of pre-market clearance/approval', covering 136 software units distributed nationwide. No patient harm or specific prediction error was reported. In April 2025 Exactech released a new version of Predict+ in Europe and Australia and stated that it 'has not received FDA marketing authorization'.

Failure modeMachine-learning outcome predictions used to inform surgical planning were distributed to US surgeons before any regulatory review of the model's validation. Clinicians had no independent assurance that the predictions were accurate for their patients.

What might have helpedTreating the regulatory status of predictive models embedded in surgical planning software as a prerequisite for patient use could have prevented unauthorized deployment. Disabling machine learning prediction features in jurisdictions where they lack authorization might have mitigated compliance risks. Requiring documented external validation on comparable populations before surgeons incorporate these predictions into consent or planning discussions could have ensured greater reliability.

What happenedIn December 2023 a proposed class action was filed in the US District Court for the Western District of Kentucky alleging that Humana used the nH Predict model, which compares a patient with about six million others to predict how long they will need care, to cut off payment for post-acute rehabilitation for Medicare Advantage members prematurely. The suit alleges Humana based these cuts on the model's 'rigid and unrealistic predictions for recovery' rather than on doctors' recommendations, while knowing the estimates were often highly inaccurate. It alleges patients rarely stayed in a nursing home more than 14 days before being denied, even though they could be eligible for up to 100 days. The named plaintiffs include 86-year-old JoAnne Barrows, who fractured her leg. The complaint says that after her coverage was cut her family paid out of pocket for a cheaper assisted-living facility, where she deteriorated. On 15 August 2025 Judge Rebecca Grady Jennings partly denied Humana's motion to dismiss: claims for breach of contract, breach of the implied covenant of good faith and fair dealing, unjust enrichment and common-law fraud were allowed to proceed, while state statutory unfair-practices and insurance bad-faith claims were dismissed. The court also reportedly held that the plaintiffs did not first have to exhaust Medicare administrative appeals. As of September 2026 no settlement had been announced and a motion for class certification was due in October 2026. The court has not ruled on the merits, and these remain allegations.

Failure modeA population-level length-of-stay prediction was allegedly applied as a hard cutoff for individual coverage, overriding treating clinicians' judgement.

What might have helpedTreating a model's length-of-stay estimate as one input and leaving the decision to clinicians who know the patient could have avoided rigid cutoffs. Independent checks of how often predicted stays matched patients' actual needs, and tracking how many denials were overturned on appeal, could have exposed the inaccuracy the suit alleges.

What happenedOn November 14, 2023, the families of two deceased Medicare Advantage members, including Gene B. Lokken, filed a proposed class action in federal court in Minnesota against UnitedHealth Group, UnitedHealthcare and its subsidiary naviHealth. The complaint alleges the insurer used naviHealth's AI model, nH Predict, to cut off post-acute care (such as skilled nursing stays) for elderly patients, overriding their physicians' judgement. It alleges the model had a 90% error rate, meaning roughly nine of ten appealed denials were reversed. It also alleges the companies kept using it because only about 0.2% of policyholders ever appeal. Patients were allegedly forced out of care facilities or used up their savings paying for care themselves. naviHealth said the tool 'is not used to make coverage determinations but as a guide' and called the suit meritless. On February 13, 2025, U.S. District Judge John Tunheim dismissed five of the seven claims. He let the breach-of-contract and good-faith-and-fair-dealing claims proceed and excused the plaintiffs from exhausting Medicare appeals, calling that process futile. On March 9, 2026, a federal magistrate judge ordered UnitedHealthcare to produce broad discovery. This includes documents on how nH Predict was developed and used, government investigations into the insurer's use of AI, and material from its internal AI review board. The judge also refused its request to stage discovery. The case remained in discovery later in 2026.

Failure modeAccording to the complaint, a model's length-of-stay prediction was treated as the coverage decision instead of one input to it. This let it override treating physicians. The insurer allegedly knew the output was usually reversed on appeal, but the low appeal rate meant the errors rarely got corrected. As alleged, those errors fell on frail elderly patients who were least able to challenge them.

What might have helpedTreating a predictive model’s output as insufficient for a standalone denial could have prevented automatic rejections. Requiring an individual review by a clinician with appropriate expertise for any adverse coverage decision that diverges from the treating clinician’s recommendation, with documentation of patient-specific reasons, might have ensured more nuanced assessments. Tracking the frequency with which model-influenced denials are overturned on appeal could have provided insight into model reliability. Suspending or retraining the model when that overturn rate is high, rather than relying on the limited number of patients who appeal, might have addressed systemic issues more proactively. Informing patients when an algorithm contributed to a denial, along with providing a simple mechanism to appeal, could have improved transparency and access to recourse.

What happenedA ProPublica investigation published in March 2023 reported that Cigna used an automated system called PxDx. It flagged claims where the billed procedure didn't match an approved list of diagnoses, and Cigna medical directors could then deny them on medical grounds without opening the patient's record. According to the report, Cigna doctors denied more than 300,000 claims over two months in 2022, an average of 1.2 seconds per claim. A former Cigna doctor was quoted: 'We literally click and submit... It takes all of 10 seconds to do 50 at a time.' Patients were left with unexpected bills. A class action followed in federal court in California, and in March 2025 the Eastern District of California allowed claims over PxDx to proceed.

Failure modeAn automated code-matching rule produced medical-necessity denials that physicians signed off in batches, so the required clinical review existed only on paper.

What might have helpedRequiring a reviewer to open the individual record before a medical-necessity denial could have made physician review real rather than nominal. Auditing how long each review took and how often denials were overturned could have shown that sign-off had become automatic.

What happenedJAMA Internal Medicine external validation of Epic's widely deployed sepsis prediction model found AUC 0.63 (vs claimed 0.76–0.83), missing 67% of sepsis cases while generating high alert volumes across hundreds of hospitals.

Failure modeProprietary model deployed at scale without independent external validation; training-label leakage (clinician suspicion baked into features); alert fatigue.

What might have helpedTreating independent external validation as a prerequisite before deployment could have provided an additional layer of scrutiny. Transparent performance reporting might have offered clearer insights into system behavior, while local recalibration could have better aligned the technology with specific environmental conditions. Monitoring real-world sensitivity, rather than focusing solely on the number of alerts fired, might have revealed nuances in performance that standard metrics overlook.

What happenedIn December 2020 Stanford Medicine used a rules-based formula to rank staff for its first 5,000 COVID-19 vaccine doses. The formula weighed factors such as age, job role and department. Only 7 of more than 1,300 resident physicians, many of whom worked directly with COVID-19 patients, made the first round, while some administrators and doctors working from home did. Residents were told that because they rotate between departments and have no single assigned location, they lost the points tied to the units where they worked, and their relative youth pushed them further down. On 18 December at least 100 residents protested. Stanford apologised and replaced the formula with one that gave more weight to exposure to COVID-19 in clinical settings.

Failure modeThe allocation formula relied on administrative data (assigned location, age) that didn't reflect actual clinical exposure, so the people at highest risk were ranked low.

What might have helpedChecking the ranked output against common sense, for example asking whether the staff who treat COVID-19 patients were near the top, could have caught the flaw before it was announced. Involving the affected groups, such as residents, in reviewing the formula could also have helped.

What happenedIn April 2020 Google Health researchers presented a study (CHI 2020) on the first real-world deployment of their deep-learning system for detecting diabetic retinopathy, in 11 clinics in Thailand. In the lab the system had been more than 90% accurate and could return results in under 10 minutes. In the clinics, as reported by MIT Technology Review and TechCrunch, it often rejected eye photos taken in poorly lit rooms, and slow internet connections held up results. Nurses and patients were frustrated, and some patients whose images were rejected were told to see a specialist at another site, which meant extra travel. The researchers concluded that local conditions and clinic workflows had to be accounted for when deploying such tools.

Failure modeA model validated on high-quality lab images was deployed without accounting for real clinic conditions (lighting, bandwidth, workflow), so many images were rejected and screening was disrupted.

What might have helpedThe researchers' own field study suggests that testing locally in the target clinics before scale-up, checking image quality and connectivity, and designing a fallback when an image is rejected could have reduced disruption. This kind of validation outside the lab is what the study itself came to provide.

What happenedIn October 2019 Ziad Obermeyer and colleagues published a study in Science of a commercial algorithm, reported to be Optum's, that US health systems used to identify patients for high-risk care-management programmes. The algorithm predicted future healthcare costs as a stand-in for health need. Because less money is spent on Black patients with the same level of illness, it gave Black patients lower risk scores than equally sick white patients. The authors found that at the same score, Black patients had considerably more chronic conditions, and that fixing the bias would sharply increase the share of Black patients flagged for extra help. Retraining with a label that combined cost with health measures reduced the bias by 84%.

Failure modeLabel-choice bias: the model predicted healthcare spending instead of illness, building unequal access to care into who was flagged for extra help.

What might have helpedThe authors suggest that checking whether the prediction target actually measures health need, and auditing calibration across racial groups with direct health measures, could have revealed the gap. Their retraining with health-based labels showed the bias could be largely removed.

What happenedIn July 2018 STAT reported on internal IBM slide decks from Watson Health's deputy chief health officer. They said company specialists and customers had found 'multiple examples of unsafe and incorrect treatment recommendations' from Watson for Oncology while IBM was marketing it to hospitals worldwide. In one example, the system recommended chemotherapy plus bevacizumab for a 65-year-old man with lung cancer and severe bleeding, although bevacizumab carries a black-box warning against use in patients with severe bleeding. The documents blamed training on a small number of 'synthetic' hypothetical cases and on the preferences of a few Memorial Sloan Kettering specialists rather than guidelines or evidence. STAT reported no known patient deaths from the recommendations.

Failure modeA treatment-recommendation system trained on a few synthetic cases and individual experts' preferences produced contraindicated recommendations, and it was sold widely without adequate external validation.

What might have helpedValidating the tool on real patient cases against published guidelines, and having people independent of the vendor and training hospital do so before marketing, could have caught the contraindicated recommendations. Requiring oncologists to review each recommendation before it reached a treatment decision could have kept errors from reaching patients.

What happenedIn 2016 the Arkansas Department of Human Services began using a computerised assessment algorithm, instead of nurses' judgement, to set how many hours of in-home care people in its ARChoices Medicaid waiver programme received. Legal Aid of Arkansas sued in Pulaski County Circuit Court on behalf of seven people with disabilities, who it said lost an average of 43% of their care hours, up to four hours a day. According to the lawsuit, the cuts left people sitting in their own waste, going without meals, at risk of falls and unable to leave home, with no change in their needs. In Arkansas Department of Human Services v. Ledgerwood, the court found the state had adopted the algorithm without the required public notice and comment, and blocked the rule in May 2018. The state later stopped using the algorithm.

Failure modeAn opaque algorithm replaced clinical assessment in setting care hours. It cut services sharply without any change in patients' needs and without transparency or meaningful appeal.

What might have helpedPublishing the assessment method for public comment, as the court found the law required, and testing its outputs against nurses' assessments before rollout could have exposed the drastic cuts. Keeping a clinician able to override the allocation could have protected people whose needs hadn't changed.

What happenedThe ACLU of Idaho brought a class action, K.W. v. Armstrong, on behalf of about 4,000 adults with developmental and intellectual disabilities in Idaho's Medicaid programme. Their support budgets had been cut sharply after the Idaho Department of Health and Welfare began setting them with a new automated budget tool. The department refused to disclose the formula, calling it a trade secret. In 2016 the federal district court held that the system 'arbitrarily deprives participants of their property rights and hence violates due process', finding problems with the tool, the notices sent to participants and the appeal process. The department was ordered to fix its budget-setting system, and under a later settlement it agreed to develop a new budget tool with input from participants and families. Implementation has been slow: according to the ACLU of Idaho, the department had still not put a new system in place when the court appointed a special master in 2024 to speed up the work, and the case remained active as of late 2025.

Failure modeA secret, inadequately validated budgeting formula cut disability support for thousands of people. They could neither see nor effectively challenge how their budgets were calculated.

What might have helpedThe court's findings suggest that disclosing how budgets are calculated, testing the tool to confirm it reflects people's assessed needs, and offering a real appeal with a human review could have prevented arbitrary cuts.