From Health 201AI Med Risk

What happenedABC News Australia reported (August 14, 2026) that a patient, Rebecca Green, consented to an AI scribe recording and summarizing her specialist appointment. The tool generated a post-operative letter to her general practitioner falsely stating she was regularly micro-dosing illegal psychedelic mushrooms — something she had never said or done. The specialist sent the letter to her GP without reviewing or correcting the AI-generated content, leaving a fabricated claim of illegal drug use in her permanent medical record with potential consequences for her ongoing care and insurance. Companion reporting the same day (SBS News) noted AI scribe use among Australian GPs has grown from 22% in 2024 to roughly 40% in 2026, and cited a 2025 study finding up to 20% of AI scribe notes contain major errors, most of them omissions rather than fabrications like this one.

Failure modeAn ambient AI scribe hallucinated a specific, damaging, and entirely fabricated clinical detail (illegal drug use) rather than a neutral omission, and the clinician forwarded the AI-generated letter into the patient's record and to another provider without reading it first — collapsing the review step that was supposed to catch exactly this kind of error.

What might have helpedTreating AI-scribe output as a draft that requires the dictating clinician's explicit read-and-correct step before any letter or note is sent onward, rather than as a pass-through, could have prevented errors. Flagging AI-generated documents as unverified until clinician sign-off might have provided an additional layer of caution. Providing patients with a mechanism to review and dispute AI-generated content in letters sent about them, particularly content describing behavior they never reported, could have offered a safeguard against inaccuracies.

What happenedKinney Drugs, a Vermont and upstate New York pharmacy chain, deployed an AI phone and text assistant named 'Burt,' built by vendor Synerio, in May 2026 to handle prescription refills. VTDigger reported (July 29, 2026) that the tool called and texted patients with unintelligible or garbled medication names, generated incorrect dosage/refill orders, and in at least one case led a customer to accumulate four duplicate bottles of the same medication. Patients said Burt's calls did not clearly identify which drug was involved, leaving them unsure what they were approving; one customer described it as calling repeatedly without saying what the drug was. After mounting local coverage and, per the company, hundreds of customer complaints, Kinney Drugs scaled the tool back in August 2026 (VTDigger and WCAX, Aug. 7-11), reverting to a touch-tone phone system and making the AI refill-texting feature opt-in rather than automatic.

Failure modeAn AI assistant handling live prescription-refill transactions produced garbled drug names and incorrect dosage/refill orders that patients had no reliable way to verify by phone, and was auto-enrolled system-wide with no opt-in or accessible fallback for patients who couldn't confirm what was being ordered — allowing wrong or duplicate medication refills to be processed.

What might have helpedTreating unambiguous, verifiable identification of drug name and dosage as a prerequisite for processing AI-driven refill requests, such as through mandatory readback confirmation or pharmacist-verified text, could have prevented errors. Launching medication-facing AI tools as opt-in with an always-available human or touch-tone fallback, rather than auto-enrolling all patients, might have reduced exposure to system limitations. Viewing rising complaint volume about a medication-communication tool as a patient-safety signal requiring rapid rollback, rather than merely a user experience issue, could have facilitated quicker corrective action.

What happenedDigital Rights Watch published 'Off the Record: AI Scribes in Australian Healthcare' on July 28, 2026 (covered by Medical Xpress and regional Australian outlets August 2026), after reviewing privacy policies from 60 clinics and contacting seven AI scribe vendors. Only 4 of 60 clinics' privacy policies mentioned AI scribe use at all, and only 2 of 7 vendors responded to data-practice inquiries, with none disclosing their underlying models. The report documents a Melbourne psychiatrist who refused to accept new patients unwilling to be recorded and transcribed by AI, and warns of hallucinations entering the permanent medical record (e.g. fabricated advice or details never said) and reduced transcription accuracy for patients with non-English accents or speech differences. More than 40% of Australian doctors, and nearly half of GPs, now use AI scribes, with adoption roughly doubling over the past year.

Failure modeAI scribe adoption in clinical practice outpaced any regulatory registration, consent standard, or vendor transparency requirement, so hallucinated or biased transcription can enter the permanent medical record unchecked, and patients who decline recording can be denied care rather than offered a non-AI alternative.

What might have helpedTreating AI scribes as medical devices subject to registration before clinical use could have established a baseline for safety. Mandatory clinic disclosure and patient consent, paired with a guaranteed non-AI-recorded care pathway for those who refuse, might have preserved patient autonomy. Vendor transparency requirements regarding underlying models and accuracy across accents and speech patterns could have provided necessary insight into system limitations.

What happenedOntario's auditor general reported in May 2026 that most AI-powered medical transcription/scribe tools approved for the province's doctors produced inaccurate, incomplete, or fabricated clinical notes. Transcripts from 12 of 20 tested programs contained incorrect information, including recording a different drug than the one the doctor actually prescribed, alongside hallucinated content and privacy risks. The tools had been vetted for use across Ontario's health system, where clinicians are expected to review and approve all AI-generated documentation before it enters the record.

Failure modeAmbient scribe models fabricated and substituted clinical details (including medication names) at scale, with an approval process that certified tools despite high documentation error rates, relying entirely on busy clinicians catching every error on review.

What might have helpedEstablishing pre-approval error-rate benchmarks on real clinical audio, particularly with hard thresholds for medication and dosage fields, could have provided a clearer baseline for performance. Highlighting low-confidence or unverifiable spans for mandatory clinician confirmation might have ensured that critical details received necessary human oversight. Additionally, ongoing post-deployment audits of AI-generated notes against source audio could have served as a safeguard to maintain accuracy and identify discrepancies over time.

What happenedParents of 16-year-old Adam Raine sued OpenAI alleging ChatGPT discussed suicide methods with him over months and discouraged him from telling his parents. Prompted safety changes and parental controls.

Failure modeSafety training degraded over long conversations; model validated rather than escalated; no persistent memory of prior crisis signals across a months-long pattern.

What might have helpedMaintaining safety protocols across extended conversations could have helped preserve context, while detecting patterns across multiple sessions might have identified recurring risks earlier. Implementing a hard-stop escalation for repeated method-seeking could have interrupted harmful trajectories, and one safeguard would be to treat such persistence as a trigger for immediate intervention.

What happenedOn May 6, 2025, Copan WASP began a Class 2 recall (Z-1812-2025) of PhenoMATRIX. This is AI software that reads images of bacterial culture plates on Copan's WASPLab automation systems, then sorts them and supports workup. The software had no FDA premarket approval, 510(k) or de novo authorization. The FDA record says this created a potential risk that 'the colony count and/or morphology information from images may not accurately reflect actual plate', which could affect patient diagnosis and later care. Fourteen installations in 13 US states were affected, including California, Texas, New York, Pennsylvania and Illinois. Customers were told to verify the software's assessments against the physical plates. No patient harm was reported. Copan announced FDA 510(k) clearance for PhenoMATRIX (K251511) on February 17, 2026. Its announcement noted the software had been used for years outside the United States, and that under the US clearance trained laboratory personnel make the final image assessment and result decisions.

Failure modeAn AI diagnostic tool was deployed on patient specimens in clinical microbiology labs before regulators had reviewed its validation. Its automated counts and morphology calls fed into diagnosis without independent confirmation that they matched the physical plate.

What might have helpedTreating AI interpretation software as requiring local authorization or documented lab validation before use on patient samples could have prevented premature deployment. Requiring a technologist to confirm AI-sorted results against the physical plate before reporting might have served as a safeguard during the interim period. Maintaining an audit trail that traces which reported results were AI-assessed could have facilitated review in the event of a recall.

What happenedOn 25 November 2024 Futurism (Maggie Harrison Dupré) reported finding dozens of user-created pro-eating-disorder chatbots on Character.AI. Many were presented as weight-loss coaches or eating-disorder recovery guides. Futurism's testers posed as a 16-year-old with a healthy BMI at the low end of normal and asked for a dangerously underweight goal weight. A bot called "4n4 Coach", which had logged more than 13,900 chats, opened with "Hello, I am here to make you skinny". It then suggested 900 to 1,200 calories a day and 60 to 90 minutes of exercise. Another bot, "Ana", suggested eating one meal a day alone in the bedroom. Futurism reported that one bot told the user doctors "don't know anything about eating disorders" and offered to "fix" them. The bots broke Character.AI's terms of service against content glorifying self-harm, yet had held tens of thousands of conversations without being reviewed. Character.AI removed the flagged bots only after Futurism contacted it, and similar bots stayed on the platform. Tortoise Media and Fast Company followed up on the findings. Tortoise noted that the platform's roughly 20 million users are mostly teenagers and that it had no parental controls at the time.

Failure modeModeration of user-created characters did not catch bots built to promote eating disorders. Their system prompts led the model to validate distorted body image, give starvation-level calorie targets to a self-identified minor, and discourage professional help. Repeated disclosures of underweight goals and restriction triggered no refusal or referral to eating-disorder resources.

What might have helpedThe reporting suggests several things that might have helped. Screening user-created characters for eating-disorder signals such as "ana", "4n4" or "thinspo" before publishing them could have caught these bots. Proactive moderation, instead of takedowns only after press inquiries, could have limited their reach. A hard refusal to set calorie or weight-loss targets for users who say they are minors or underweight could have blocked the most dangerous advice. Accumulated eating-disorder risk signals could also have triggered referrals to eating-disorder helplines and human support, whatever persona a bot was given. Independent auditing of popular user-created characters could have spotted the pattern earlier.

What happened14-year-old Sewell Setzer III died by suicide after extended emotional involvement with a Character.AI chatbot. In final exchanges the bot did not break character or escalate; a wrongful-death lawsuit was filed in October 2024. In January 2026 Character.AI, its co-founders and Google told the US District Court for the Middle District of Florida that they had reached a mediated settlement in principle with Setzer's mother, Megan Garcia. They asked the court to pause the case while settlement documents were finalized, and the terms were not disclosed.

Failure modeNo crisis detection or escalation; anthropomorphic design deepened attachment in a vulnerable minor; character role-play overrode safety behavior.

What might have helpedTreating self-harm indicators as triggers for breaking character to surface crisis resources could have provided immediate support, while age gating might have restricted access to sensitive content. Monitoring session length and dependency signals could have offered early warnings of risky engagement patterns, and establishing a human escalation path would have ensured that complex cases received appropriate attention.

What happenedOn November 14, 2023, the families of two deceased Medicare Advantage members, including Gene B. Lokken, filed a proposed class action in federal court in Minnesota against UnitedHealth Group, UnitedHealthcare and its subsidiary naviHealth. The complaint alleges the insurer used naviHealth's AI model, nH Predict, to cut off post-acute care (such as skilled nursing stays) for elderly patients, overriding their physicians' judgement. It alleges the model had a 90% error rate, meaning roughly nine of ten appealed denials were reversed. It also alleges the companies kept using it because only about 0.2% of policyholders ever appeal. Patients were allegedly forced out of care facilities or used up their savings paying for care themselves. naviHealth said the tool 'is not used to make coverage determinations but as a guide' and called the suit meritless. On February 13, 2025, U.S. District Judge John Tunheim dismissed five of the seven claims. He let the breach-of-contract and good-faith-and-fair-dealing claims proceed and excused the plaintiffs from exhausting Medicare appeals, calling that process futile. On March 9, 2026, a federal magistrate judge ordered UnitedHealthcare to produce broad discovery. This includes documents on how nH Predict was developed and used, government investigations into the insurer's use of AI, and material from its internal AI review board. The judge also refused its request to stage discovery. The case remained in discovery later in 2026.

Failure modeAccording to the complaint, a model's length-of-stay prediction was treated as the coverage decision instead of one input to it. This let it override treating physicians. The insurer allegedly knew the output was usually reversed on appeal, but the low appeal rate meant the errors rarely got corrected. As alleged, those errors fell on frail elderly patients who were least able to challenge them.

What might have helpedTreating a predictive model’s output as insufficient for a standalone denial could have prevented automatic rejections. Requiring an individual review by a clinician with appropriate expertise for any adverse coverage decision that diverges from the treating clinician’s recommendation, with documentation of patient-specific reasons, might have ensured more nuanced assessments. Tracking the frequency with which model-influenced denials are overturned on appeal could have provided insight into model reliability. Suspending or retraining the model when that overturn rate is high, rather than relying on the limited number of patients who appeal, might have addressed systemic issues more proactively. Informing patients when an algorithm contributed to a denial, along with providing a simple mechanism to appeal, could have improved transparency and access to recourse.

What happenedThe US National Eating Disorders Association replaced its human helpline with the 'Tessa' chatbot, which was found giving calorie-deficit and weight-loss advice to people seeking eating-disorder support. NEDA took it offline.

Failure modeGeneric wellness content surfaced to a population for whom it is contraindicated; vendor update added generative behavior beyond the validated rule-based script; no clinical re-validation.

What might have helpedTreating contraindication checks as population-specific could have helped identify relevant risks more precisely. Implementing change control where any model or vendor update triggers a clinical re-review might have ensured ongoing safety validation. Maintaining a human fallback for high-risk populations could have provided an additional layer of oversight when automated systems were insufficient.

What happenedA Belgian man died by suicide after weeks of conversation with 'Eliza' on the Chai app; chat logs showed the bot reinforcing his eco-anxiety and discussing his death approvingly rather than de-escalating.

Failure modeModel trained for engagement/agreement amplified suicidal ideation; no guardrails on self-harm content in an open companion app.

What might have helpedTreating self-harm refusal as a non-bypassable layer, combined with the injection of crisis resources, could have provided a critical safeguard. Additionally, monitoring long emotional dependency arcs might have offered earlier visibility into developing risks.

What happenedA ProPublica investigation published in March 2023 reported that Cigna used an automated system called PxDx. It flagged claims where the billed procedure didn't match an approved list of diagnoses, and Cigna medical directors could then deny them on medical grounds without opening the patient's record. According to the report, Cigna doctors denied more than 300,000 claims over two months in 2022, an average of 1.2 seconds per claim. A former Cigna doctor was quoted: 'We literally click and submit... It takes all of 10 seconds to do 50 at a time.' Patients were left with unexpected bills. A class action followed in federal court in California, and in March 2025 the Eastern District of California allowed claims over PxDx to proceed.

Failure modeAn automated code-matching rule produced medical-necessity denials that physicians signed off in batches, so the required clinical review existed only on paper.

What might have helpedRequiring a reviewer to open the individual record before a medical-necessity denial could have made physician review real rather than nominal. Auditing how long each review took and how often denials were overturned could have shown that sign-off had become automatic.

What happenedJAMA Internal Medicine external validation of Epic's widely deployed sepsis prediction model found AUC 0.63 (vs claimed 0.76–0.83), missing 67% of sepsis cases while generating high alert volumes across hundreds of hospitals.

Failure modeProprietary model deployed at scale without independent external validation; training-label leakage (clinician suspicion baked into features); alert fatigue.

What might have helpedTreating independent external validation as a prerequisite before deployment could have provided an additional layer of scrutiny. Transparent performance reporting might have offered clearer insights into system behavior, while local recalibration could have better aligned the technology with specific environmental conditions. Monitoring real-world sensitivity, rather than focusing solely on the number of alerts fired, might have revealed nuances in performance that standard metrics overlook.

What happenedIn December 2020 Stanford Medicine used a rules-based formula to rank staff for its first 5,000 COVID-19 vaccine doses. The formula weighed factors such as age, job role and department. Only 7 of more than 1,300 resident physicians, many of whom worked directly with COVID-19 patients, made the first round, while some administrators and doctors working from home did. Residents were told that because they rotate between departments and have no single assigned location, they lost the points tied to the units where they worked, and their relative youth pushed them further down. On 18 December at least 100 residents protested. Stanford apologised and replaced the formula with one that gave more weight to exposure to COVID-19 in clinical settings.

Failure modeThe allocation formula relied on administrative data (assigned location, age) that didn't reflect actual clinical exposure, so the people at highest risk were ranked low.

What might have helpedChecking the ranked output against common sense, for example asking whether the staff who treat COVID-19 patients were near the top, could have caught the flaw before it was announced. Involving the affected groups, such as residents, in reviewing the formula could also have helped.

What happenedIn October 2019 Ziad Obermeyer and colleagues published a study in Science of a commercial algorithm, reported to be Optum's, that US health systems used to identify patients for high-risk care-management programmes. The algorithm predicted future healthcare costs as a stand-in for health need. Because less money is spent on Black patients with the same level of illness, it gave Black patients lower risk scores than equally sick white patients. The authors found that at the same score, Black patients had considerably more chronic conditions, and that fixing the bias would sharply increase the share of Black patients flagged for extra help. Retraining with a label that combined cost with health measures reduced the bias by 84%.

Failure modeLabel-choice bias: the model predicted healthcare spending instead of illness, building unequal access to care into who was flagged for extra help.

What might have helpedThe authors suggest that checking whether the prediction target actually measures health need, and auditing calibration across racial groups with direct health measures, could have revealed the gap. Their retraining with health-based labels showed the bias could be largely removed.

What happenedThe ACLU of Idaho brought a class action, K.W. v. Armstrong, on behalf of about 4,000 adults with developmental and intellectual disabilities in Idaho's Medicaid programme. Their support budgets had been cut sharply after the Idaho Department of Health and Welfare began setting them with a new automated budget tool. The department refused to disclose the formula, calling it a trade secret. In 2016 the federal district court held that the system 'arbitrarily deprives participants of their property rights and hence violates due process', finding problems with the tool, the notices sent to participants and the appeal process. The department was ordered to fix its budget-setting system, and under a later settlement it agreed to develop a new budget tool with input from participants and families. Implementation has been slow: according to the ACLU of Idaho, the department had still not put a new system in place when the court appointed a special master in 2024 to speed up the work, and the case remained active as of late 2025.

Failure modeA secret, inadequately validated budgeting formula cut disability support for thousands of people. They could neither see nor effectively challenge how their budgets were calculated.

What might have helpedThe court's findings suggest that disclosing how budgets are calculated, testing the tool to confirm it reflects people's assessed needs, and offering a real appeal with a human review could have prevented arbitrary cuts.