When production AI misbehaves, the first job is to stop harmful or uncontrolled actions while preserving the evidence needed to explain what happened.
Declare the incident, contain the affected action path using an operational control, preserve the complete decision trace, choose rollback, disable, or degraded service based on the failure boundary, and add the failure to the evaluation set before release resumes.
Do not begin with prompt editing. That changes evidence while the incident is active and may leave the same unsafe tool, route, tenant, or data source available. Containment must work through a kill switch, feature flag, permission boundary, model route, traffic policy, or human-review gate that responders can operate independently of a redeploy.
Open an incident when an output, action, cost, latency, or control breach crosses a defined threshold. Assign one decision owner and one current incident record.
Stop the narrowest unsafe action path first. Disable the tool, tenant, retrieval source, model route, or automated write while preserving safe read-only or human-reviewed service where possible.
Roll back when a known-good version restores the whole boundary. Disable when no safe automated path remains. Degrade when a bounded alternative, such as retrieval-only results or mandatory review, remains useful.
Record input, retrieved context, prompt and policy versions, model and parameters, tool requests and results, control decisions, output, timestamps, tenant, and correlation identifiers under the normal access and retention rules.
State what must be true before traffic returns: the failed case passes, adjacent cases do not regress, containment controls are reset deliberately, and monitoring can detect recurrence.
Add a redacted reproduction to the versioned evaluation set with the expected control response and a release-blocking threshold. The same failure must not depend on institutional memory.
Engineering reasoning: the distinction between rollback, disable, and degrade is operational, not semantic. It is determined by which boundary can be restored safely and how quickly responders can prove it.
Google's SRE workbook recommends a clear command line, defined roles, a working record of debugging and mitigation, and early incident declaration. It describes incident management practices at large service organisations; it does not determine the right containment boundary for the reader's AI system. Read the Google SRE chapter (retrieved 2026-08-16).
NIST's Manage resources collect approaches for prioritising, responding to, and monitoring AI risks. The playbook is a risk-management resource, not evidence that any specific control or recovery plan has been exercised successfully. Read the NIST resource (retrieved 2026-08-16).
For agent-based workflows, these documents expand on the decision above. They describe our method, not client results.
Identify the failed authority boundary and the containment evidence needed before an agent can restart.
Distinguish emergency suspension from restart approval and record who holds each decision right.
Bring the release candidate, its current controls, and the decision you need to make. The first conversation identifies the evidence that exists and the gaps that still need owners.
We review your enquiry and reply by email. No commitment to a paid project; scope and terms are agreed before work begins.
Prefer email? dev@alpha-ai.dev