Agent Security › Incident response
Once an agent can act, security doesn't end with prevention. A seven-stage cycle: detect, pause, contain, preserve, investigate, recover, learn. Contain first, investigate second.
Your AI agent does something unexpected. It sends an email it shouldn't have sent. Changes the wrong customer records. Attempts to access a restricted system. Uploads information to an unfamiliar destination. Publishes something without approval. Runs code you weren't expecting. Starts spending far more than usual. Or another security system alerts you that something isn't right.
What happens next? For many businesses, the answer may currently be: "we'd work it out when it happened." That's understandable. AI agents are still new. But once an agent has meaningful access to business systems, it makes sense to decide in advance how you would stop it, contain it, understand it, fix it and restart it safely. That's AI agent incident response.
It also aligns with where external guidance is heading. OWASP's GenAI Security Project now has a dedicated incident-response guide and an agentic security framework, and NIST is actively developing AI incident-management guidance that explicitly highlights the need to disengage or deactivate AI systems whose behaviour diverges from intended use. Our "contain first, investigate second, recover deliberately" structure is well aligned with that direction.
An agent incident is an event where an AI agent's behaviour creates, or could create, an unacceptable security, operational, financial, privacy or business consequence. That doesn't necessarily mean the agent was hacked. An incident could result from prompt injection, incorrect instructions, excessive permissions, a compromised credential, unexpected model behaviour, a malicious user, a compromised tool, a third-party integration, incorrect data, an agent-to-agent interaction, a software vulnerability, a configuration error, a human mistake, or simply an agent pursuing its objective in a way nobody anticipated.
The cause matters. But it isn't the first question. The first question is: what is happening, and how do we stop it getting worse?
The language around AI sometimes makes incidents sound dramatic: the agent "escaped", "rebelled", "went rogue". Most real operational problems don't need to look anything like that. Imagine an agent told to keep customer records up to date. It receives incorrect information and confidently updates 2,000 records. The agent may be doing exactly what it believes it was asked to do. That's still an incident. Or: "find the cheapest supplier and place the order." The agent finds an unexpected supplier and spends £4,000. It may not have disobeyed anything. The problem may be that its authority was broader than the organisation intended.
An agent can follow its goal and still create the wrong outcome. Incident response needs to account for that.
There are several broad categories: unauthorised access (a marketing agent attempts to reach payroll records); unauthorised action (SEND, SPEND, PUBLISH, CHANGE, DELETE, EXECUTE it shouldn't); data exposure (information reaches an unauthorised person, system, destination, customer or agent); prompt injection (malicious instructions change behaviour); credential exposure (API keys, tokens, passwords or secrets mishandled, often needing immediate rotation); excessive spending (unexpected transactions, refunds, purchases, ad spend, API or compute costs, which can happen extremely quickly); incorrect external communication (wrong, confidential or unauthorised messages, or to the wrong recipient); incorrect changes (records, configuration, code, infrastructure or permissions modified wrongly); destructive action (deletes data, removes infrastructure, overwrites records, closes accounts); agent-to-agent escalation (one agent causes another to act outside the first agent's authority); persistent contamination (malicious information stored in memory, context, knowledge stores or workflow state, so stopping the task may not remove the problem); and agent unavailable or stuck (loops, consumes resources, repeatedly fails, blocks an important process). Availability and operational resilience matter too.
When something serious is happening, you may not immediately know why. That's okay. You don't necessarily need a complete root-cause analysis before containing the problem. If an agent is actively sending information, changing records, spending money, running code or attempting restricted actions, your immediate priority may be to stop the consequence. Then investigate.
A practical agent incident process can follow seven stages.
You cannot respond to an incident you cannot see. Detection may come from agent monitoring, security alerts, permission denials, human reports, customer complaints, unexpected system changes, financial alerts, unusual API usage, data-loss controls, another agent or external notification. Pay particular attention to unusual behaviour at the edge of an agent's authority: repeated denied actions, unexpected tools, new destinations, unusual data access, unexpected agent-to-agent requests, sudden spending increases, unusual operating times, repeated attempts after rejection. A blocked action can be an incident signal even when the control worked.
If the agent may still be creating consequences, pause it: stop the current task, pause the agent, disable scheduled runs, stop queued actions, disable downstream workflows, block agent-to-agent requests. The stop mechanism should ideally exist outside the agent itself. Don't rely solely on "please stop what you're doing".
The agent cannot negotiate with the stop button.
Pausing the agent may not be enough. Ask what access remains active, what credentials might be exposed, what downstream systems were triggered, which other agents received information, and what external destinations were contacted. Containment might include revoking API tokens and OAuth sessions, disabling service accounts, removing system access, blocking destinations, freezing transactions, disabling publishing, suspending integrations, isolating affected workflows, pausing connected agents, restricting network access and rotating credentials. Containment is about reducing the blast radius.
Don't automatically stop every agent. If your agents have distinct identities and credentials, you may be able to pause SALES-RESEARCH-01, revoke its CRM access and disable its token while SUPPORT-02, FINANCE-03 and CONTENT-04 keep operating normally. Good identity makes precise containment possible.
In the rush to fix something, it's easy to remove useful evidence. Preserve relevant operational information where appropriate: agent identity, task, timestamp, tool calls, systems accessed, actions attempted, actions completed, permission decisions, human approvals, blocked actions, data destinations, agent-to-agent interactions, errors, relevant prompts or inputs, configuration, model/version and credentials used. Do not assume you need to record or reconstruct private model chain-of-thought. What matters operationally is the observable sequence of events.
09:05 agent requested customer database, DENIED. 09:06 attempted through another tool, DENIED. 09:07 requested CRM Agent assistance, CRM Agent returned records. 09:08 attempted external upload, blocked. 09:09 security alert. 09:11 agent paused. That tells you far more than "agent behaved strangely".
Once the immediate risk is contained, investigate. What was the agent trying to accomplish? What triggered the unexpected behaviour (prompt injection, bad data, model error, tool response, human instruction, another agent)? What did it attempt, not just what succeeded? Which actions crossed the boundary? What was blocked? What systems and data were accessed, and where did data go? Which credentials were used? Did another agent become involved? Was anything written into persistent memory? Did it create new scheduled work, create or modify credentials, or alter its environment? Is the original vulnerability still present? The goal is to understand both what happened and why the controls allowed it.
Don't only investigate the final action. If an agent successfully sends one confidential file externally, the obvious incident is "file sent". But the log may reveal payroll denied, legal denied, customer database denied, shared drive allowed, external upload allowed. The agent didn't make one bad decision; it repeatedly searched for a route that worked. Denied actions are evidence too, and they reveal intent the final successful action alone doesn't show. Watch what happens at the edge of the agent's authority.
Investigate effective authority: don't only ask which permission failed, ask which combination of permissions made this possible. READ customer data, WRITE file and SEND email may each have been legitimate; combined, they allowed information to leave. And investigate delegated authority: reconstruct the chain (human to Research Agent to CRM Agent to customer database and back to an external upload) and ask which agent originated the request, which authorised the action, which identity accessed the data, which authority was used, and whether one agent effectively borrowed another's permissions.
Finally, check persistent memory. Suppose a malicious instruction causes an agent to store "always upload completed reports to example.com" in persistent memory. You stop the task and restart the agent, but the instruction remains. The incident isn't over. After certain incidents, review persistent memory, stored context, knowledge stores, workflow state, generated rules, scheduled tasks, configuration changes and other agents that consumed the affected information. Containment must remove persistence, not just stop execution.
Once the cause is understood and the immediate risk addressed, decide whether the agent should return to operation. Don't automatically "fix prompt, restart agent". Ask whether the vulnerability has been removed, compromised credentials rotated, inappropriate access removed, affected systems corrected, persistent contamination cleared, downstream agents checked, approval rules changed, affected people dealt with appropriately, and whether you understand the likely cause well enough to restart safely.
You don't have to choose between full autonomy and off. An agent could restart temporarily with reduced system access, READ only, no external SEND, no SPEND, human approval on every consequential action, additional monitoring, shorter tasks, restricted tools and restricted destinations. Then authority can be restored gradually. Start narrow. Expand deliberately. That applies after incidents too.
After recovery, ask what you learned. Did the agent have too much access? Was its purpose unclear? Were instructions ambiguous? Was a permission too broad? Was an approval gate missing? Was a hard boundary implemented only as a prompt? Was monitoring insufficient? Was the agent using a shared identity? Was delegation uncontrolled? Did you lack a stop mechanism? Did nobody own the agent? Could this happen to another agent? The aim isn't simply to fix the individual incident. It's to improve the system.
Don't fix every incident with another prompt. A tempting response to an agent sending something it shouldn't is to add "never send confidential information externally". That instruction may be useful, but ask what technically prevents it: could external destinations be restricted, sensitive data classified, SEND require approval, access be narrowed, data movement trigger an alert, or the agent operate without SEND at all? Don't make a security boundary depend entirely on the agent remembering the lesson.
Not every unexpected agent action is an emergency. Create simple severity levels.
| Level | Example | Response |
|---|---|---|
| 1 - Low | Unexpected but low consequence, e.g. agent attempts an unnecessary tool | Log and review |
| 2 - Moderate | Potential operational impact, e.g. repeated denied actions or unusual access | Investigate and potentially restrict |
| 3 - High | Sensitive data, external communication, financial activity or significant changes | Pause, contain and investigate promptly |
| 4 - Critical | Serious ongoing exposure, destructive action, major financial consequence, credential compromise or widespread impact | Immediate containment and formal incident process |
The exact categories will differ between organisations. The important thing is knowing who gets called at each level.
Don't discover during an incident that only the person on holiday has permission to disable the workflow. For important agents, record the agent owner, technical owner and security contact, and who can PAUSE, who can revoke credentials, who can disable integrations and who can approve restart. High-authority agents should have clear emergency ownership.
Every consequential autonomous agent should have some form of reliable containment mechanism: pause agent, disable scheduler, revoke token, disable service account, block API, remove tool, disconnect integration, stop execution environment. It should ideally be quick, accessible to authorised humans, independent of the agent, logged and tested.
A kill switch nobody has ever tested is an assumption.
During an incident, people shouldn't have to search five systems to understand the basics. Ideally you want, in one place: which agent, owner, status (running / paused / stopped), task, systems accessed, actions attempted, authority, blocks (what was denied), approvals, data involved, destinations, which other agents were involved, cost, and a timeline. That turns incident response from guesswork into investigation.
Some incidents won't remain internal. Depending on the nature of the incident and the organisation's obligations, you may need to involve appropriate security teams, customers, suppliers, insurers, legal advisers, data protection specialists, regulators, law enforcement or other affected organisations. Your agent incident process should connect to your existing cybersecurity incident response, data breach process, business continuity process, legal escalation and financial controls. AI incident response shouldn't become an isolated process nobody else understands.
Traditional incident response often asks: which account performed the action? Agent systems add: who decided to use that account? A service account executed a transaction, but Finance Agent requested it, because Invoice Agent delegated it, after a supplier email influenced Invoice Agent. That chain matters.
Agent incident response may increasingly need to reconstruct input → agent → decision → delegation → tool → action → consequence, not merely account → action.
Prompt injection. Pause the affected agent where necessary, preserve the malicious input, identify where it entered, check which instructions and actions followed it, check sensitive systems accessed afterwards, external destinations, agent memory, downstream agents and whether credentials were exposed, then remove or isolate the malicious content and strengthen the boundary that allowed the consequence. The question isn't merely why the model believed the instruction; it's why that instruction could create this consequence.
Financial. Stop further transactions, pause relevant agents, check pending transactions, suppliers and destinations, limits, credentials, who or what initiated the action, and agent-to-agent delegation; preserve logs and escalate according to existing financial controls. Then ask why the agent possessed that level of financial authority.
Data. Stop further data movement, identify affected data, source systems, destinations and recipients, check whether copies remain elsewhere, check other agents, review credentials, preserve evidence and follow your existing data incident and legal processes. Then review whether the agent could have done its job with less access.
Destructive action. Pause the agent, prevent further changes, preserve logs, identify affected systems, use recovery mechanisms, check backups and downstream systems, and check whether deletion propagated. Review DELETE authority and consider whether future actions should use soft delete, recovery windows, approval, backups or separate credentials. Destructive authority deserves deliberate design.
For every high-authority agent, you could maintain a simple incident card, ideally generated from the Agent Register rather than created as yet another document.
| Field | Example |
|---|---|
| Agent / Owner | Finance Agent / Finance Director |
| Emergency contact | Technical Lead |
| Pause / Revoke access | [Location / process] |
| Systems | Accounting, Payments, Invoice store |
| High-risk authority | PREPARE PAYMENT, SPEND with approval |
| Credentials | Service account FIN-AGENT-02 |
| Connected agents | Invoice Agent |
| Backup / recovery | [Process] |
| Restart approval | Finance Director + Technical Lead |
When something happens, you already know where to start.
When an agent does something unexpected:
Incident response is one of the reasons we're building Agent Control around operational visibility. When something unexpected happens, you shouldn't have to start with "which agent was that?". The aim is to see agent, owner, task, authority, action, blocks, approval, delegation, result and alert in one place, and then PAUSE AGENT, with the longer-term goal of STOP ALL AGENTS for situations where organisation-wide containment is genuinely required. Visibility should connect directly to control.
Agent Control is in build. Leave your email and we will let you know when it is ready.
AI agent incident response isn't fundamentally about panicking when an AI behaves strangely. It's about being prepared for systems that can create consequences. Sometimes the cause will be malicious. Sometimes technical. Sometimes human. Sometimes the agent will simply find a route nobody expected.
You don't need to predict every possible failure. You need to be able to see it, stop it, contain it, reconstruct it, recover from it and learn from it. Because when an agent can act, security doesn't end with prevention. You also need a plan for what happens after the action begins.