AI Security Operations, Incident Response & Recovery, 25 min read

Detecting runtime threats, misuse and compromise

In this lesson you will:

  • Detect threats across inputs, prompts, retrieval chains, outputs and resource consumption
  • Detect misuse of AI access and agency
  • Detect secret exposure and tool-execution anomalies
  • Monitor integrity of AI assets across the lifecycle
  • Use AI-driven security operations such as UEBA, automated triage and threat hunting

Prevention will sometimes fail. Detection engineering for AI looks for the signatures and behaviours of attacks across the whole request path — and increasingly uses AI to help.

1. Detection across the request path

Detection points along the AI request path.
Figure 8.5 Detection points along the request path, with example detections at each.
Point Example detections
Inputs and prompts Injection and jailbreak classifier hits; encoded payloads; repeated rephrasing of refused requests; unusual languages or scripts for the user base
Retrieval chains Retrieval of documents outside a user's usual domain; spikes in sensitive-document retrieval; newly added documents retrieved unusually often
Outputs PII or secrets in responses; external URLs or encoded data; output length anomalies; responses revealing system prompts
Resource consumption Token spikes per identity; long-running sessions; repeated maximum-length requests; agent iteration counts near limits

2. AI access and agency misuse

  • Unauthorised identities — calls to model endpoints from unregistered service principals or unknown API keys (shadow AI or stolen keys).
  • Over-privileged agents in action — agents using scopes they rarely use; first-time use of destructive tools.
  • Delegation anomalies — agents acting without a corresponding user request; unusual delegation chains in multi-agent systems.
  • Approval bypass attempts — repeated attempts at actions just below approval thresholds (e.g. many ₹1,999 refunds when approval starts at ₹2,000).

3. Secret exposure and tool-execution anomalies

  • Secrets in prompts, outputs or logs — API keys, passwords and tokens detected by secret scanners on AI traffic.
  • Tool-execution anomalies — new sequences (read files → call external webhook), calls at unusual times, parameters outside normal ranges, high error rates (probing), sandboxed code attempting network access.

4. AI asset and lifecycle integrity

  • Model integrity — loaded artefact hashes match the registry; unexpected model versions in production.
  • Knowledge-base integrity — bulk additions or edits; documents from unusual authors; content containing instruction-like text.
  • Pipeline integrity — unscheduled training runs; promotions outside change windows; disabled gates.
  • Configuration integrity — drift from approved prompts, tools and guardrail settings.

5. AI-driven security operations

AI also strengthens the SOC itself:

AI-driven SOC capabilities: UEBA, automated triage, threat hunting and response.
Figure 8.6 AI for the SOC. Useful at scale — but the SOC's own AI is also an attack surface.
  • UEBA (user and entity behaviour analytics) — baselines for users and entities, including agents and service identities; flags deviations such as an agent suddenly reading HR data.
  • Automated alert triage — models summarise alerts, enrich them with context and suggest priority, reducing analyst fatigue.
  • Threat hunting — natural-language querying of logs; hypothesis generation from threat intelligence; clustering of similar events.
  • Response assistance — drafting incident summaries, suggesting playbook steps.

Cautions: AI triage can be wrong or manipulated (attackers can craft log content that injects instructions into an AI analyst); keep humans accountable for decisions; apply the same guardrails, least privilege and logging to SOC agents as to any other agent.

Worked example — detection rules for a customer-service agent

Rule Logic Response
Threshold probing ≥ 5 refunds between 90% and 100% of the approval threshold by one agent session in 1 hour Suspend session; human review
New destructive tool use First use of close_account by an agent identity in 30 days Alert system owner
Exfil pattern Output contains ≥ 3 external URLs with query strings longer than 100 characters Block output; alert SOC
Secret in prompt Secret scanner match in user prompt Mask in logs; notify user; open ticket to rotate if internal
Unregistered caller Model API call from a principal not in the AI inventory Block; alert as possible shadow AI or stolen key
Poisoning signal Newly added knowledge-base document retrieved in > 20% of answers within 24 hours Quarantine document; content review

🔐 Security lens

Your SOC's AI assistant reads attacker-controlled content all day — log lines, email bodies, URLs. Threat-model it like any other agent exposed to untrusted input.

Key takeaways

  • Detect across inputs, retrieval, outputs and resource consumption.
  • Watch for unauthorised identities, unusual scope use, delegation anomalies and threshold probing.
  • Scan AI traffic for secrets; flag unusual tool sequences, parameters and sandbox network attempts.
  • Monitor integrity of models, knowledge bases, pipelines and configuration.
  • UEBA, automated triage, threat hunting and response assistance scale the SOC — but SOC AI is itself an injectable agent.

Self-check

  1. What pattern might indicate an attacker probing an approval threshold?
  2. How could an attacker target a SOC's AI triage assistant?
  3. Give one signal of knowledge-base poisoning.