NIST Secure Software Development Framework
NIST SP 800-218 groups secure development practices around preparation, protection, production, and vulnerability response. Read NIST SP 800-218.
AI production readiness checklist
Decide whether the AI system is ready to carry real work.
This checklist covers the engineering evidence needed for a controlled production decision. It is not an organisational maturity score and it does not assume that every candidate should ship.
Checklist
Eight gates for production readiness
-
Purpose and accountable owner
The workflow, intended result, current baseline, affected people, decision authority, and accountable business and technical owners are named.
-
Acceptance criteria and evaluation
Representative work, known exceptions, prohibited outcomes, escalation cases, quality thresholds, and release-blocking failures are versioned and reviewable.
-
Data and access boundary
Approved sources, retention, privacy, permissions, secrets, tenant boundaries, and data-location requirements are defined for the real environment.
-
System and dependency behaviour
Model, retrieval, tool, API, queue, storage, and network failures have explicit timeout, retry, idempotency, fallback, and partial-write behaviour.
-
Human authority
People know which decisions remain theirs, what evidence accompanies a review, how to override or stop the system, and when escalation is mandatory.
-
Observability and incident response
Quality, safety, latency, cost, exceptions, drift, and dependency health have signals, thresholds, owners, runbooks, and a tested response path.
-
Release, rollback, and recovery
The release can be limited, compared, disabled, or rolled back. State can be reconciled after partial execution, and recovery has been exercised.
-
Economics, support, and transfer
The complete operating cost, support model, change process, documentation, evaluation assets, runbooks, and long-term owner are accepted.
How to use it
Require evidence, an owner, and a disposition for every gate.
For each item, link the artifact that supports the answer, name who approved it, and record one of four states: ready, ready with a named control, blocked, or not applicable with a reason. An unchecked box is not a defect until the team understands why it matters to this workflow.
Engineering reasoning: a checklist reduces omission risk, but it can create false confidence when completion is treated as evidence. The useful artifact is the decision record behind each item.
Public sources
Sources that inform the checklist
NIST AI RMF Playbook: Measure
NIST's Measure resources support context-specific assessment and documented evidence rather than a universal score. Read the NIST resource.
Google SRE: Monitoring Distributed Systems
The SRE chapter explains observable system behaviour and alerts tied to action, which informs the monitoring and response gates. Read the SRE chapter.
OWASP Top 10 for LLM Applications
OWASP documents application-layer risks that should be considered when defining data, output, tool, and permission controls. Read the OWASP project.
Code audit
What does a code audit of an AI system cover?
A code audit of an AI system reads 4 artifacts that a conventional code audit does not: the prompt and model configuration, the context and tool boundary, the evaluation assets, and the run record from real usage.
Static review of the application code still applies, and it still finds the defects it always found. It is not sufficient here, because the behaviour that decides whether the system can be trusted in production is mostly not expressed in the code. A prompt edit, a model version change, a retrieval index rebuild, or a widened tool permission can change the output without changing a line that a reviewer would read.
Prompt and model configuration
Whether prompts, model versions, and generation parameters are versioned with the code, and where an untracked change can alter behaviour without a release.
Context and tool boundary
What the system is allowed to read, which tools it may call, whose permissions it acts under, and what it does when a dependency fails or returns nothing.
Evaluation assets
Whether cases, expected behaviour, rubric, runner, and thresholds exist, and whether someone other than the original author can re-run them and get the same answer.
Run record
Traces, exceptions, escalations, overrides, and cost from real usage, which show the failure classes that reading the repository cannot reveal.
On an inherited AI codebase the finding is rarely that the code is bad. It is that the system has no versioned evaluation set, so no change can be shown to be an improvement; that acceptance criteria were agreed in conversation and never written down, so readiness is a matter of opinion; and that the person who held the working model of the system has left. None of those are visible in a repository read, and all of them block a production decision.
Engineering reasoning: treat the audit output as a disposition per gate above, with the evidence attached, rather than a severity-ranked defect list. A defect list tells you what to fix; the gates tell you whether to deploy, revise, or stop.
Published method
Agent frameworks for this decision
For agent-based workflows, these documents expand on the decision above. They describe our method, not client results.
AI Agent Control Checklists
Attach a control record, accountable owner and retained evidence to each applicable readiness gate.
AI and Agent Governance Framework
Name who can approve the release, accept a residual risk and authorize a later material change.
Next decision
Review one system against the checklist
Bring the release candidate, its current controls, and the decision you need to make. The first conversation identifies the evidence that exists and the gaps that still need owners.
We review your enquiry and reply by email. No commitment to a paid project; scope and terms are agreed before work begins.
Prefer email? dev@alpha-ai.dev