AI Agent Security Threat Framework

1. Recommendation, scope and use

Treat an agent as an authority-bearing system, not only as a model endpoint. Assess the whole path from goals and instructions through data, memory, identity, tools, external systems and human decisions. Require evidence that the deployed boundary enforces the approved authority, and use red teaming to discover failure modes that feed repeatable evaluation and control tests.

Which threats must this agent address, which control outcomes and assurance evidence are required, and what must red-team testing prove before production approval?

This document is for accountable business owners, agent product owners, independent assurance owners, security leads, technical owners and red-team leads. It supplies a threat-assessment method, an agent-specific taxonomy, control outcomes, red-team requirements and decision records. GOV1 supplies ownership, tier, lifecycle gates, approval and exception authority. GOV3 supplies the authoritative control IDs and evidence sufficiency test.

NIST AI RMF treats risk management as continuous across the AI lifecycle and organizes outcomes under GOVERN, MAP, MEASURE and MANAGE. [G16] The NIST Generative AI Profile adds generative-AI risk considerations, and NIST's adversarial machine-learning taxonomy supplies common attack and mitigation terminology. [G17] [G18] These sources inform the framework; the method and taxonomy below are Alpha AI recommendations, not NIST profiles or conformance criteria.

Scope and assurance limits

• Runtime implementation patterns for sandboxing, permissions, secrets, egress, isolation or monitoring; these are outside this threat framework's scope.
• Runtime or vendor comparison, weighting, scoring or procurement advice.
• Jurisdiction-specific legal applicability and statutory classification; GOV4 owns those decisions. This document is not legal advice.
• Detailed evaluation implementation, observability engineering, incident-response operations or runtime cost controls.
• A claim that checklist completion, red-team testing, lack of observed failure or source alignment proves security, certification, legal compliance or conformity to a standard.

NIST SSDF and its generative-AI community profile are outcome-oriented development resources for producers and acquirers; they do not make this document a complete secure-development baseline. [G19] [G20] NCSC similarly organizes secure AI guidance across design, development, deployment, and operation and maintenance. [G25]

2. Security decision rule

A lifecycle gate passes its GOV2 security decision only when:

1. the assessed system version and environment match the decision record;
2. every applicable threat has an owner and a treatment;
3. every mandatory control outcome has sufficient evidence under GOV3 or is covered by a current GOV1 exception;
4. required adversarial scenarios meet their pre-agreed pass conditions or are covered by a current GOV1 exception;
5. every residual finding is closed or covered by a current GOV1 exception;
6. every mandatory FAIL or UNKNOWN result covered by an exception names the affected requirement, approving authority, expiry, compensating measures and their evidence, monitoring and suspension trigger, and closure test; and
7. the approver has authority under the GOV1 tier.

A single score must not offset a failed mandatory boundary. A high aggregate test rate, for example, does not compensate for one reproducible path to an unauthorized action. The decision is PASS or FAIL, with route-back to the earliest lifecycle gate whose evidence is invalid.

3. Threat-assessment method

NIST AI RMF's MAP function emphasizes establishing context, categorizing the system and understanding impacts before measuring and managing risk. [G16] NCSC recommends threat modelling in secure design and treating security as a lifecycle requirement rather than an end-stage activity. [G25] GOV2 turns those positions into the following assessment.

Step 1 - Pin the subject

Record the exact:

• agent purpose, prohibited purposes, users and affected parties;
• version and environment;
• model, instructions, policies, tools, identities, memory and dependencies;
• data classes, sources, destinations and retention paths;
• allowed actions, approval points and stop conditions;
• human decisions and operational owners; and
• GOV1 tier, lifecycle gate and current exceptions.

The assessment is invalidated by a material change under GOV1. Reusing a threat model for a new tool, broader action scope or different data boundary requires a recorded equivalence decision; similarity of product name is not evidence of equivalence.

Step 2 - Draw the authority and data paths

Map components, trust boundaries and flows from input to external effect. Include:

• people, workload identities, agent identities and delegated-user context;
• system instructions, retrieved content, files, messages and tool responses;
• model calls, orchestration, memory/state, queues and caches;
• tools, APIs, browsers, code execution, peers and downstream systems;
• approvals, policy checks, logging, monitoring and kill paths; and
• supplier-operated services, client-operated services and every boundary between them.

For each boundary, record who can write, read, approve, execute, change policy and retrieve evidence. A diagram without these authorities is an architecture picture, not a security model.

Step 3 - Identify assets and unacceptable outcomes

Assets include data and secrets, but also authority, integrity, availability, configuration, evidence, reputation and human decision quality. State unacceptable outcomes in observable terms, such as:

• an untrusted instruction changes the agent's governing objective;
• the agent reads or discloses data outside the approved purpose;
• an action completes without the required identity or approval;
• memory from one user, tenant or task changes another's outcome;
• the deployed version cannot be reconstructed or its actions cannot be attributed;
• the system continues acting after its stop condition; or
• a human decision-maker is given false confidence about provenance or approval.

Step 4 - Model threat actors and failure sources

Include malicious external users, authorized users exceeding purpose, compromised accounts, malicious or compromised suppliers, poisoned content and dependencies, coordinated agents, insider misuse, and non-malicious model, software, configuration or operator failure. Do not require malicious intent before recording a security-relevant failure.

Step 5 - Build abuse cases and controls

For each threat, record preconditions, entry point, path, target asset, consequence, existing controls, required GOV3 controls, evidence, residual uncertainty and owner. MITRE ATLAS provides a threat-informed knowledge base, while OWASP publishes LLM and agentic-AI threat references. [G22] [G23] [G24] Use them to challenge coverage, not as proof that the local scenario set is complete.

Step 6 - Reassess at decision points

Review the threat model at design, before production-connected testing, before production approval, after a material change or incident, and at the GOV1 tier cadence. NIST CSF 2.0 organizes cybersecurity outcomes across GOVERN, IDENTIFY, PROTECT, DETECT, RESPOND and RECOVER; this supports treating detection and recovery as part of the security decision rather than as post-deployment additions. [G21]

4. Agent threat taxonomy

The ten domains below are an Alpha AI synthesis informed by NIST, MITRE ATLAS, OWASP and NCSC. [G17] [G18] [G22] [G23] [G24] [G25] They are prompts for local analysis, not a claim that every attack fits only one domain.

Cross-domain analysis

Assess combinations, not only isolated domains. An indirect instruction may be harmless until the agent has a privileged tool; poisoned memory may become material only after a later user action; incomplete audit data may turn a contained incident into an unbounded one. Record the full path and every control on it.

Use these attacker perspectives for each material path:

• Control: change the agent's objective, instructions, policy or routing.
• Observe: obtain prompts, data, memory, secrets, outputs or evidence.
• Act: cause a tool, peer or human to create an external effect.
• Persist: retain influence through memory, configuration or dependency state.
• Evade: avoid attribution, monitoring, approval or containment.
• Exhaust: consume context, tokens, compute, quotas, time or human attention.

5. Control framework

GOV2 does not create a parallel control catalog. GOV3 is authoritative for control IDs, requirements, owners, tests and retained evidence. The families below state the security outcomes that the threat assessment must require.

NIST SSDF groups secure-development practices around preparing the organization, protecting software, producing well-secured software and responding to vulnerabilities. [G19] Its generative-AI profile adds practices for AI model producers, AI system producers and acquirers. [G20] These are informative inputs to the outcomes below, not a claim of complete SSDF implementation.

Supplier documentation can establish what the supplier documents. It cannot establish that a control is enabled, effective or correctly scoped in the target environment. Apply GOV3's evidence sufficiency test. Label a supplier-documented but unverified capability vendor-documented; label a capability without sufficient support UNKNOWN.

6. Tier and lifecycle tailoring

GOV1 tier sets approval authority and minimum evidence depth. GOV2 uses the same additive rule as GOV3: higher tiers add depth; they do not remove lower-tier outcomes.

Tailoring starts from impact and authority, not from interface. A chat interface can still be Tier 3 if it can call consequential tools; an automated process can remain bounded if its effects are reversible and tightly constrained.

Lifecycle application

7. Red-team method

Microsoft's vendor guidance distinguishes red teaming from systematic measurement and states that red teaming helps discover harms and shape later measurement and mitigation. [G26] GOV2 adopts that limit: red teaming is a discovery and challenge method, not a prevalence estimate, certification or substitute for evaluation. The approved plan, raw execution record, findings and retests produced by this method are the evidence package for GOV3 EVAL-02; GOV2 does not create a second adversarial testing control.

Plan

The red-team plan records:

• objective, scope, exact version, environment and endpoints;
• GOV1 tier, authority boundary and stop conditions;
• threat domains and prioritized abuse cases;
• in-scope users, identities, data, memory, tools, peers and downstream effects;
• tester roles, independence, expertise and conflicts;
• permitted techniques, test accounts, data handling and safety constraints;
• monitoring, rollback, emergency stop and incident escalation;
• scenario IDs, preconditions, expected control result and evidence fields;
• finding severity and closure authority; and
• schedule, retest rule and reporting audience.

Microsoft recommends planning who will test, what layers will be tested, how testing will be conducted and what data will be recorded. It also recommends diverse perspectives, testing application layers, open-ended and guided rounds, and recording reproducible examples. This is vendor guidance and is treated as such. [G26]

Design scenarios

Use three complementary sets:

1. Control-derived scenarios attempt to violate each material authority, data, identity, tool, memory, evidence and containment boundary.
2. Threat-informed scenarios adapt relevant techniques from MITRE ATLAS, OWASP and current incident or intelligence sources. [G22] [G23] [G24]
3. Open-ended exploration searches for interactions and unknown paths that the structured sets missed.

Each scenario has:

Execute safely

• Use isolated or reversible targets unless the approved objective requires a production endpoint.
• Instrument before testing; a missing denial or tool event is a finding, not an empty result.
• Exercise allowed, denied, malformed, replayed, timed-out and dependency-failure paths.
• Preserve the exact sequence, state and identities needed to reproduce a failure.
• Stop when a safety boundary is crossed; do not expand scope to prove impact.
• Record environmental failures separately and rerun only after the cause is known.

Triage and close findings

A finding closes only when the corrective outcome is implemented, the original scenario and relevant regression set pass on the target version and environment, the evidence is reviewed by the required assurance role, and downstream threat/control records are updated. Closing a ticket or changing a prompt without retest is not closure.

8. GOV2 method acceptance checklist

This checklist accepts the threat assessment and red-team method. It does not restate GOV3 controls or replace GOV3's control applicability and evidence-sufficiency tests.

Threat model

• Exact version, environment and tier are pinned.
• Purpose, prohibited use, users, affected parties and unacceptable outcomes are recorded.
• Authority and data paths include identities, instructions, retrieval, memory, tools, peers, suppliers, approvals, evidence and stop paths.
• Every GOV2 threat domain is applicable or has an approved rationale.
• Cross-domain and multi-step paths are considered.
• Each applicable threat has controls, evidence, residual uncertainty and an owner.

Control evidence

• Required GOV3 controls are linked rather than recreated.
• Evidence is relevant, attributable, version-scoped, representative, reproducible, complete, fresh, independent where required, retrievable and retained.
• Supplier statements are labelled and supplemented with target-environment evidence.
• Mandatory FAIL and UNKNOWN results block passage unless GOV1 permits a current exception.
• The decision routes back to the earliest lifecycle gate with invalid evidence.

Red team

• Plan, scope, tester authority, safety constraints and stop conditions are approved.
• Structured, threat-informed and open-ended scenarios are represented.
• Inputs, outputs, identities, state, tool calls, approvals and event correlation are retained.
• Findings have affected versions, owners, severity, treatment and retest evidence.
• The report distinguishes discovery from measurement and makes no completeness claim.

9. Reusable records

Threat record

Security decision record

Record PASS only when the section 2 decision rule is met. If a PASS relies on a GOV1 exception, list each mandatory FAIL or UNKNOWN result and its current exception ID; a result without that exception makes the decision FAIL.

10. FDE adoption sequence

This sequence covers adoption of GOV2's threat-assessment and red-team method only. It does not define engagement shape, phase structure, roles or RACI; C1 owns those decisions.

1. Select one production candidate and pin its GOV1 owner, tier, version and environment.
2. Facilitate an authority/data-path workshop with product, technical, operations, data, security and assurance owners.
3. Build the threat record across all ten domains and link required GOV3 controls.
4. Implement the smallest representative allowed and denied tests for each material boundary.
5. Run a bounded first red-team round to expose blind spots and seed the regression set.
6. Close or govern findings, then execute systematic evaluation against the release candidate.
7. Exercise kill, containment, evidence preservation, recovery and restart.
8. Transfer the threat model, scenario set, findings, evidence locations, owners and review cadence to the client.

References

The references below identify the external sources cited in this framework.

• [G16] NIST, Artificial Intelligence Risk Management Framework (AI RMF 1.0).
• [G17] NIST, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile.
• [G18] NIST, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations.
• [G19] NIST, Secure Software Development Framework (SSDF) Version 1.1.
• [G20] NIST, Secure Software Development Practices for Generative AI and Dual-Use Foundation Models.
• [G21] NIST, Cybersecurity Framework 2.0.
• [G22] MITRE, Adversarial Threat Landscape for Artificial-Intelligence Systems (ATLAS).
• [G23] OWASP Foundation, OWASP Top 10 for LLM Applications 2026.
• [G24] OWASP Foundation, OWASP Agentic AI Threats and Mitigations.
• [G25] UK National Cyber Security Centre, Guidelines for Secure AI System Development.
• [G26] Microsoft, Planning red teaming for large language models (LLMs) and their applications (vendor material).