When the Playbook Breaks: AI Incident Response for Systems That Don't Behave Like Anything Else

Three years after the explosion of GenAI in the enterprise, most organizations now have an inventory of their AI systems, an acceptable use policy, and — at best — a process for approving AI use cases. Far fewer, however, have answered a seemingly simple question about AI Incident Response: what exactly do we do on the day one of these tools is compromised?

Most security organizations have mature playbooks for ransomware, business email compromise, and cloud account takeover. Yet very few have a documented response for an AI-specific scenario: discovering that a prompt injection attack has been successfully exploited against a production AI assistant for the past three weeks.

As large language models move from pilots into customer-facing applications and autonomous agents, security incidents involving AI systems are no longer hypothetical. Yet traditional incident response frameworks assume properties that AI systems do not have: deterministic behavior, patchable vulnerabilities, and evidence that lives in familiar places like system logs and disk images. When those assumptions break, so does the playbook.

 

AI Incident Response in 2026: Why the Tipping Point Has Arrived

Incident response for AI systems is not a new topic, but in 2026 two independent forces are converging, turning a theoretical risk into an operational priority for CISOs and boards.

 

Autonomous Agents Have Moved Beyond the Pilot Phase

Deloitte's 2026 State of AI in the Enterprise documents how quickly agentic AI is moving from experimentation to production. By 2027, 74% of organizations expect to use AI agents at least "moderately", yet only 21% claim to have a mature governance model for agentic AI in place[1]. The attack surface is no longer the single LLM, but chains of agents invoking one another, often with inherited permissions and limited observability.

 

The European Regulatory Framework Enters Its Operational Phase

In 2026, AI incidents move from a theoretical risk to a regulatory and operational reality. The AI Act (Regulation EU 2024/1689) introduces, under Article 73[2], a dedicated serious incident reporting obligation for providers of high-risk AI systems. In parallel, organizations subject to NIS2 and — in the financial sector — DORA may already face notification obligations when AI-related incidents impact essential services, critical processes, or ICT systems.

In short, 2026 is the year in which two trajectories intersect: the operational maturity of agentic AI on one side, and the entry into force of a binding regulatory framework on the other. For security functions, the cost of not having an incident response playbook extended to AI systems is no longer merely theoretical or reputational — it is regulatory, and it is measured against timelines and standards that assume the capability is already in place.

 

What Counts as an AI Security Incident?

The first challenge is definitional. An AI security incident is any event in which the confidentiality, integrity, or availability of an AI system — or of the data and tools it can reach — is compromised through AI-specific attack vectors that are not identifiable with traditional security systems. In practice, this includes several recurring scenarios, such as:

• a prompt injection;
• data poisoning;
• model poisoning and supply chain attack;
• tool poisoning and misuse;
• rogue agent deployment.

Notice what these have in common: none of them necessarily involves malware, a compromised credential, or a network intrusion. Many would never trip a traditional SOC detection. An attacker manipulating a customer-facing chatbot into revealing another user's data is, from the network's point of view, just another API call with a 200 response code. The incident materializes at the application and semantic layer, not the infrastructure layer.

To address this asymmetry, organizations should:

1. Extend their incident classification taxonomy to explicitly include AI-related events;
2. Define severity criteria that consider not only the actions actually performed by the model, but also its potential access to data and resources, as well as a broader range of impact dimensions[3];
3. Establish ownership in advance: SOC, application team, AI platform team, or a formalized cross-functional collaboration.

 

Detecting AI Incidents: There Is No Response Without Logging

Managing AI incidents begins well before the incident itself: it starts with logging. At a minimum, organizations should retain:

• the prompts submitted to the model;
• the outputs generated;
• tool and function calls;
• queries issued to the retrieval system;
• the identity context of the user (or system) behind each interaction.

Without this record, there is no way to establish what happened, when it started, or how far it spread.

Logging conversations creates real tension with privacy obligations: prompts routinely contain personal and confidential data. The answer is not to give up logging, but to treat interaction logs as sensitive-data archives, with strict access controls, time-bounded retention and redaction or masking mechanisms where feasible. A log that cannot be legally retained is as useless during an investigation as a log that was never collected.

However, traditional event-level logging is insufficient for agentic and multi-agent workflows. Capturing isolated events without correlation risks obscuring the full causal chain of an attack: a compromised session may span multiple agents, retrieval operations and tool invocations before its intent becomes visible. Organizations must evolve their detection and observability capabilities to map end-to-end interactions through distributed tracing frameworks—such as OpenTelemetry—that correlate discrete actions (spans) into complete end-to-end workflows (traces) using unique identifiers. This end-to-end reconstruction is what allows security teams to see attack patterns that remain invisible when logs are examined one event at a time.

Detection capabilities must also evolve. Signature-based approaches struggle against attacks expressed in natural language, which can be phrased in an essentially unlimited number of ways.

An effective detection program layers three complementary approaches:

• pattern-based rules for known prompt-injection payloads and jailbreak strings (useful as a first line of defense, but insufficient on their own — any determined attacker can rephrase around them);
• semantic intent classifiers that analyse the content of prompts and outputs to detect malicious intent regardless of phrasing — for example, requests attempting to override instructions, extract system prompts, exfiltrate data or invoke unauthorized tools;
• behavioral anomaly detection that identifies deviations from established baselines in how users, sessions and agents behave — anomalous tool call sequences, unusual retrieval query volumes, data-enumeration output patterns, or sudden spikes in guardrail refusals followed by successfully completed requests (a classic jailbreak-in-progress signal).

The detection surface of GenAI systems is inherently bidirectional: protection cannot be limited to inbound prompts[4] but must also apply controls and policies to outbound outputs. Consequently, effective detection mechanisms should inspect both inputs and outputs, and correlate detection signals across the entire AI interaction chain — including retrieval operations, tool and function calls, and identity context — rather than focusing solely on the conversational interface. This is where the distributed tracing approach introduced above becomes the connective tissue: it is what makes end-to-end correlation possible in practice.

 

Containing AI Incidents: When There Is No Simple Kill Switch

Traditional incident response containment isolates a host or disables an account. For an AI system embedded in business workflows, the equivalent decisions are harder. Fully shutting down a customer-facing assistant may be the safest choice but also costly.

For this reason, the IR team should define — and regularly test — a graduated containment strategy tailored to each AI system in production. The ability to revoke an AI agent's access to a specific connector in minutes, without redeploying the whole application, is the difference between surgical containment and a service outage.

The example below illustrates what such a containment playbook can look like: a set of concrete actions mapped to specific MITRE ATLAS scenarios, with clear ownership assigned through a RACI structure and explicit distinction between containment (limiting the consequences already produced) and eradication (structural remediation, deferred to a separate phase). This is a reference example, not a template to be adopted as-is: every organization should build its own playbook based on its AI systems, its threat model and its operational context. What matters is that the playbook exists, is documented, is testable and is exercised before the incident makes it necessary.

The procedure should be exercised before it is actually needed. A playbook that only works on paper is not containment.

 

Forensics on Non-Deterministic AI Systems

Investigating AI incidents introduces a fundamentally new challenge: the system under investigation does not reliably reproduce its own behavior. The same malicious prompt may succeed in one session and fail in the next. Forensics therefore rely primarily on replay — reconstructing events from interaction logs — rather than on re-execution.

An emerging paradox of AI forensics deserves explicit consideration: when the response team attempts to use frontier models to examine malicious payloads, attack prompts or compromised interaction logs, the safety guardrails of those same models may refuse the analysis, treating hostile content as input not to be processed. The defender, in other words, risks being obstructed by the very safety mechanisms designed to prevent misuse[5]. Organizations should therefore ensure governed access to alternative models (for example open-weight ones in an isolated environment) usable during investigations and define in advance the selection criteria and usage constraints for each.

Blast radius assessment also shifts perspective. The key question is not only "what did the attacker do", but "what was reachable from the compromised session":

• which documents were retrievable through the RAG system;
• which tools the agent could invoke;
• which downstream systems trusted the outputs produced by the model.

Mapping model permissions and data pathways in advance dramatically accelerates this analysis. Frameworks such as MITRE ATLAS and the OWASP Top 10 for LLM Applications provide useful vocabularies for classifying observed techniques and communicating findings.

 

Recovery and Remediation for AI Security Incidents

For most software vulnerabilities, recovery ends with a patch. For an AI system, remediation may require:

• hardening the system prompt;
• implementing and reviewing guardrail policies;
• enforcing least privilege access;
• additional model fine-tuning or adversarial training;
• upgrading to a new model version.

Each of these changes can alter system behavior in seemingly unrelated areas. Recovery must therefore always include regression testing[6]: re-running a standing suite of adversarial and functional test cases to confirm that the fix closes the exploited path without degrading legitimate behavior or opening new ones.

Finally, close the loop. Every AI incident is an input to the system's threat model, its monitoring rules, and its next round of adversarial testing. Organizations that treat these events as isolated anomalies will keep responding to the same AI incident challenges under different names.

 

Building an AI Incident Response Capability: A Practical Roadmap

There is no need to rebuild the AI incident response program from scratch. It should be extended deliberately, in controlled phases.

 

0–30 days: Foundations

• Extend the incident taxonomy with AI-specific categories (e.g. prompt injection, data and model poisoning, tool poisoning and misuse, rogue agent deployment).
• Build an inventory of AI systems in production, with criticality classification (business impact, data processed, effective permissions).
• Audit the minimum logging (prompts, outputs, tool calls, retrieval queries, identity context) for every in-scope system.
• Perform a preliminary mapping of each AI system to the applicable notification regimes (AI Act, GDPR, NIS2, DORA).

 

31–60 days: Operational Capability

• Define and regularly exercise a containment playbook for every AI system in production.
• Integrate AI-specific behavioral detections into the SOC (tool call anomalies, RAG spikes, enumeration patterns).
• Clarify the RACI between SOC, AI platform team, application owner and privacy/legal office. An AI incident naturally has at least four owners: without clarity, no one responds.
• Align the threat model to MITRE ATLAS and the OWASP Top 10 for LLM Applications.

 

61–90 days: Maturity and Assurance

• Run at least one tabletop exercise on a realistic scenario (recommended: indirect prompt injection with RAG-based exfiltration).
• Implement an AI regression suite integrated into the release process; no deployment without a passing suite.
• Formally update the IR playbook and secure governance approval (Risk Committee, Audit Committee).
• Define and baseline metrics for board reporting: number of AI systems in scope, logging coverage, mean detection and containment time by scenario, and outcomes of the most recent tabletop exercises.

 

Conclusion: The Time to Prepare Is Now

The difference between an AI incident that is rapidly contained and a crisis with operational, reputational, and regulatory consequences is not determined at the moment of the attack, but much earlier: by the organization's ability to prepare, rehearse, and validate its response capabilities.

By 2026, it has become clear that this moment can no longer be postponed: the first public AI incidents have already occurred, regulatory expectations are becoming more defined, and autonomous AI agents are becoming increasingly embedded in business processes. The question is no longer whether an AI incident will occur, but when — and whether, in that moment, the organization will be prepared to respond or forced to improvise in front of the board and regulatory authorities.

As Benjamin Franklin famously observed: "By failing to prepare, you are preparing to fail." In the context of AI security, this statement takes on a practical meaning: organizations that invest today in preparation, playbook exercises, and continuous validation of their response capabilities will be best positioned to manage their first real AI incident.