Questions for deploying clinical models

Use your own judgment. Nothing here is authoritative or definitive advice: these are general suggestions drawn from existing cases, generalized from public reports of incidents and responses, not tested or validated for any setting. Health systems are complex and there is no one-size-fits-all recipe: adapt anything here with qualified clinical, legal and regulatory advisers, and do not rely on it as a standard, certification or guarantee of safety or compliance. Full disclaimer.

For teams looking at prediction, triage, allocation, coverage and documentation tools. Questions drawn from the incidents, and signals the incidents point to after go-live, each linked to the incidents behind it and to standards and laws that cover the same ground. They are prompts, not a certification or a complete list; what fits depends on your setting.

Questions before go-live

1.Is regulatory status confirmed for this intended use and jurisdiction?

US FDA Australia TGA

Behind this (2): FDA recall: machine-learning shoulder-surgery outcome predictor Predict+ withdrawn from US market for lack of clearance · FDA recall: Copan's AI culture-plate reader PhenoMATRIX used in US labs without clearance, risking inaccurate colony counts

Tried so far:

Fix adopted FDA recall: Copan's AI culture-plate reader PhenoMATRIX used in US labs without clearance, risking inaccurate colony counts — Authorisation came after the fact, and patient samples in US labs were read by unauthorised AI before the recall. Checking regulatory status before go-live, rather than after a recall, would have closed that gap.

2.Is there external validation on a comparable population, with results you can read rather than a vendor summary?

The Joint Commission and Coalition for Health AI US NIST US ONC

Behind this (2): Epic sepsis model missed two-thirds of sepsis cases in external validation · NOHARM benchmark: leading medical AI tools give severely harmful recommendations in up to 1 in 4 cases

Tried so far:

Partly mitigated Epic sepsis model missed two-thirds of sepsis cases in external validation — A better model did not remove the need for local validation: version 2 discriminates better overall, but accuracy and alert burden vary widely between hospitals.

3.Was validation run by people outside the product team, with test records that cannot be altered?

The Joint Commission and Coalition for Health AI US NIST

Behind this (1): Whistleblower suit alleges Mayo Clinic hid AI assistant's 67% error rate

Tried so far: no published fix or mitigation for these incidents yet.

4.Was there a shadow-mode period alongside the existing process before cutover?

US NIST

Behind this (1): Brazilian state's AI bed-allocation algorithm undertriaged dying woman, delaying ICU transfer five days

Tried so far: no published fix or mitigation for these incidents yet.

5.Are contraindications and population exclusions encoded in the logic?

US FDA

Behind this (2): STAT investigation: hospital AI sepsis alert nearly triggered fatal fluid overload in dialysis patient, exposing nurse exclusion from AI governance · NEDA's 'Tessa' chatbot gave weight-loss advice to eating-disorder users

Tried so far:

Tool withdrawn NEDA's 'Tessa' chatbot gave weight-loss advice to eating-disorder users — Pulling the tool stopped the harm but exposed the gap: the human helpline it replaced was already gone. The case suggests keeping a human service running until an AI tool has proven itself.

6.Is there a named clinician review step and an explicit, unpenalised override?

The Joint Commission and Coalition for Health AI World Health Organization

Behind this (3): Class action alleges UnitedHealth used naviHealth's nH Predict algorithm, with a 90% error rate, to deny Medicare Advantage post-acute care · NUHW complaint: Kaiser Permanente used unsupervised algorithm to triage mental health patients · Brazilian state's AI bed-allocation algorithm undertriaged dying woman, delaying ICU transfer five days

Tried so far:

Partly mitigated Class action alleges UnitedHealth used naviHealth's nH Predict algorithm, with a 90% error rate, to deny Medicare Advantage post-acute care — The rule now says an algorithm cannot decide coverage on its own, but without enforcement the remaining safeguards are the insurer's own individual clinician review and its monitoring of appeal overturns.

7.Are frontline staff, including nurses, part of the group that approves the tool?

The Joint Commission and Coalition for Health AI

Behind this (1): STAT investigation: hospital AI sepsis alert nearly triggered fatal fluid overload in dialysis patient, exposing nurse exclusion from AI governance

Tried so far: no published fix or mitigation for these incidents yet.

8.Are patient disclosure, consent where required, and a non-AI pathway in place?

The Joint Commission and Coalition for Health AI Information and Privacy Commissioner of Ontario World Health Organization

Behind this (2): Digital Rights Watch: Australian AI medical scribes run with no regulatory oversight, patients denied care for refusing them · Vermont pharmacy chain pulls back AI phone assistant after it garbled drug names and produced wrong, duplicate refills

Tried so far:

Partly mitigated Digital Rights Watch: Australian AI medical scribes run with no regulatory oversight, patients denied care for refusing them — Regulators are acting on scribes that make clinical suggestions, but plain transcription scribes stay unregulated. Consent and a non-AI path therefore depend on each clinic's own policy.
Tool withdrawn Vermont pharmacy chain pulls back AI phone assistant after it garbled drug names and produced wrong, duplicate refills — In hindsight, opt-in enrolment and a non-AI fallback could have been the launch design rather than the rollback. Rising complaint volume was an early warning that was handled as a service issue rather than a safety signal.

9.Is there a change-control plan saying which updates trigger re-validation and who signs off?

US FDA US FDA

Behind this (2): NEDA's 'Tessa' chatbot gave weight-loss advice to eating-disorder users · Whistleblower suit alleges Mayo Clinic hid AI assistant's 67% error rate

Tried so far:

Tool withdrawn NEDA's 'Tessa' chatbot gave weight-loss advice to eating-disorder users — Pulling the tool stopped the harm but exposed the gap: the human helpline it replaced was already gone. The case suggests keeping a human service running until an AI tool has proven itself.

After go-live: signals the incidents point to

Signal	Rather than only	From
Real-world sensitivity against confirmed outcomes	the number of alerts fired	Epic sepsis model missed two-thirds of sepsis cases in external validation
Appeal-overturn rate for model-influenced denials	the share of patients who appeal	Class action alleges UnitedHealth used naviHealth's nH Predict algorithm, with a 90% error rate, to deny Medicare Advantage post-acute care
Error rate in medication and dose fields, checked against source audio	clinician satisfaction with note quality	Ontario auditor general: approved medical AI scribes hallucinated and misrecorded drugs · Whisper transcription tool fabricating text in medical settings
Override frequency and the outcomes of overridden cases	compliance with the algorithm	Brazilian state's AI bed-allocation algorithm undertriaged dying woman, delaying ICU transfer five days · STAT investigation: hospital AI sepsis alert nearly triggered fatal fluid overload in dialysis patient, exposing nurse exclusion from AI governance
Complaint volume on patient-facing tools, treated as a safety signal	call-deflection or automation rate	Vermont pharmacy chain pulls back AI phone assistant after it garbled drug names and produced wrong, duplicate refills
Safety events reported through a protected, blinded channel	an absence of reports	Whistleblower suit alleges Mayo Clinic hid AI assistant's 67% error rate

Signal

Rather than only

From

Real-world sensitivity against confirmed outcomes

the number of alerts fired

Epic sepsis model missed two-thirds of sepsis cases in external validation

Appeal-overturn rate for model-influenced denials

the share of patients who appeal

Class action alleges UnitedHealth used naviHealth's nH Predict algorithm, with a 90% error rate, to deny Medicare Advantage post-acute care

Error rate in medication and dose fields, checked against source audio

clinician satisfaction with note quality

Ontario auditor general: approved medical AI scribes hallucinated and misrecorded drugs · Whisper transcription tool fabricating text in medical settings

Override frequency and the outcomes of overridden cases

compliance with the algorithm

Brazilian state's AI bed-allocation algorithm undertriaged dying woman, delaying ICU transfer five days · STAT investigation: hospital AI sepsis alert nearly triggered fatal fluid overload in dialysis patient, exposing nurse exclusion from AI governance

Complaint volume on patient-facing tools, treated as a safety signal

call-deflection or automation rate

Vermont pharmacy chain pulls back AI phone assistant after it garbled drug names and produced wrong, duplicate refills

Safety events reported through a protected, blinded channel

an absence of reports

Whistleblower suit alleges Mayo Clinic hid AI assistant's 67% error rate

Standards, guidance and laws

Source	What it covers	Safeguards
Responsible Use of AI in Healthcare The Joint Commission and Coalition for Health AI · 2025-09 · guidance	Seven elements for health systems: governance, local validation, risk-based monitoring, blinded AI safety-event reporting, patient disclosure and consent.	Independent validation and ongoing monitoring Clinician review as a real checkpoint Disclosure, consent and a non-AI option Change control and independent sign-off
AI-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations (draft) US FDA · 2025-01 · guidance · on Clingantry	Lifecycle expectations for AI devices: validation, transparency, performance monitoring after authorisation.	Independent validation and ongoing monitoring Change control and independent sign-off
Predetermined Change Control Plans for AI-enabled devices US FDA · 2024-12 · guidance	Manufacturers pre-specify which model changes are allowed and how each is validated, instead of shipping silent updates.	Change control and independent sign-off
HTI-1 decision support intervention transparency US ONC (ASTP) · 2024-01 · rule · on Clingantry	Certified EHRs must expose source attributes for predictive decision support: intended use, validation, and known risks.	Independent validation and ongoing monitoring Disclosure, consent and a non-AI option
AI Risk Management Framework US NIST · 2023-01 · framework	Govern, map, measure and manage: the general-purpose risk process most health AI governance programmes build on.	Independent validation and ongoing monitoring Change control and independent sign-off
Ethics and governance of AI for health: guidance on large multi-modal models World Health Organization · 2024-01 · guidance	Obligations for developers, providers and deployers of generative AI in health, including human oversight and transparency.	Clinician review as a real checkpoint Disclosure, consent and a non-AI option
AI Scribes: Key Considerations for the Health Sector Information and Privacy Commissioner of Ontario · 2026-01 · guidance	Vendor assessment, contractual safeguards, ongoing monitoring and governance for AI scribes, with a companion checklist.	Independent validation and ongoing monitoring Disclosure, consent and a non-AI option Clinician review as a real checkpoint
Digital scribes Australia TGA · 2025-08 · guidance	Transcription-only scribes are not medical devices; scribes that interpret, diagnose or recommend are, and must be registered. Compliance action began in August 2026.	Change control and independent sign-off
California SB 243, companion chatbots State of California · 2026-01 · law	In force January 1, 2026. Companion-chatbot operators must keep a protocol that prevents suicide and self-harm content and refers at-risk users to crisis services, disclose AI to minors, and prompt breaks.	Crisis escalation that holds up Disclosure, consent and a non-AI option
Illinois HB 1806, Wellness and Oversight for Psychological Resources Act State of Illinois · 2025-08 · law	Bans AI from providing therapy or making therapeutic decisions; AI may only support licensed professionals with administrative tasks.	Clinician review as a real checkpoint Disclosure, consent and a non-AI option
Utah HB 452, mental health chatbots State of Utah · 2025-05 · law	Mental-health chatbots must disclose they are AI and protect user health data; a safe harbour rewards pre-deployment testing, crisis escalation, clinical oversight and monitoring.	Crisis escalation that holds up Disclosure, consent and a non-AI option Independent validation and ongoing monitoring

Source

What it covers

Safeguards

Responsible Use of AI in Healthcare
The Joint Commission and Coalition for Health AI · 2025-09 · guidance

Seven elements for health systems: governance, local validation, risk-based monitoring, blinded AI safety-event reporting, patient disclosure and consent.

Independent validation and ongoing monitoring Clinician review as a real checkpoint Disclosure, consent and a non-AI option Change control and independent sign-off

AI-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations (draft)
US FDA · 2025-01 · guidance · on Clingantry

Lifecycle expectations for AI devices: validation, transparency, performance monitoring after authorisation.

Independent validation and ongoing monitoring Change control and independent sign-off

Predetermined Change Control Plans for AI-enabled devices
US FDA · 2024-12 · guidance

Manufacturers pre-specify which model changes are allowed and how each is validated, instead of shipping silent updates.

Change control and independent sign-off

HTI-1 decision support intervention transparency
US ONC (ASTP) · 2024-01 · rule · on Clingantry

Certified EHRs must expose source attributes for predictive decision support: intended use, validation, and known risks.

Independent validation and ongoing monitoring Disclosure, consent and a non-AI option

AI Risk Management Framework
US NIST · 2023-01 · framework

Govern, map, measure and manage: the general-purpose risk process most health AI governance programmes build on.

Independent validation and ongoing monitoring Change control and independent sign-off

Ethics and governance of AI for health: guidance on large multi-modal models
World Health Organization · 2024-01 · guidance

Obligations for developers, providers and deployers of generative AI in health, including human oversight and transparency.

Clinician review as a real checkpoint Disclosure, consent and a non-AI option

AI Scribes: Key Considerations for the Health Sector
Information and Privacy Commissioner of Ontario · 2026-01 · guidance

Vendor assessment, contractual safeguards, ongoing monitoring and governance for AI scribes, with a companion checklist.

Independent validation and ongoing monitoring Disclosure, consent and a non-AI option Clinician review as a real checkpoint

Digital scribes
Australia TGA · 2025-08 · guidance

Transcription-only scribes are not medical devices; scribes that interpret, diagnose or recommend are, and must be registered. Compliance action began in August 2026.

Change control and independent sign-off

California SB 243, companion chatbots
State of California · 2026-01 · law

In force January 1, 2026. Companion-chatbot operators must keep a protocol that prevents suicide and self-harm content and refers at-risk users to crisis services, disclose AI to minors, and prompt breaks.

Crisis escalation that holds up Disclosure, consent and a non-AI option

Illinois HB 1806, Wellness and Oversight for Psychological Resources Act
State of Illinois · 2025-08 · law

Bans AI from providing therapy or making therapeutic decisions; AI may only support licensed professionals with administrative tasks.

Clinician review as a real checkpoint Disclosure, consent and a non-AI option

Utah HB 452, mental health chatbots
State of Utah · 2025-05 · law

Mental-health chatbots must disclose they are AI and protect user health data; a safe harbour rewards pre-deployment testing, crisis escalation, clinical oversight and monitoring.

Crisis escalation that holds up Disclosure, consent and a non-AI option Independent validation and ongoing monitoring

← All tools

Not medical or legal advice. This site documents how AI systems have failed; it is not a source of care or advice. For a health problem, seek medical attention or talk to a health professional; in an emergency, call 911 or your local emergency number. For legal questions, consult a lawyer.

If you are in crisis: in the US, call or text 988 (Suicide & Crisis Lifeline). Elsewhere, see findahelpline.com.

To report harm from an AI tool, contact the relevant authority: see where to report a problem. AI Med Risk does not take complaints or act on individual cases.

From Health 201AI Med Risk