5.2 Monitoring, logging and AI security telemetry
In this lesson you will:
- Say what to log at each layer of an AI system so that an incident can be reconstructed.
- Correlate AI telemetry with the rest of your security data, and automate safe first responses.
- Protect AI logs so that they stand up as evidence.
- Log prompts and outputs without creating a new privacy problem.
- Watch AI behavior live, with guards that act automatically and can be undone.
Opening case: "200 OK"
A Meridian relationship manager reports that WealthDesk showed her a portfolio summary for a client who is not hers. The security team opens the logs.
The application log shows: "POST /assist 200 OK, 2.3s, user=rm-0412". The model provider's dashboard shows token counts per day. That is all.
The team cannot answer: What did she ask? Which documents did retrieval return, and did the permission check run? What did the model output? Did it happen to anyone else? Was it an attack or a bug? Re-asking the same question today returns the right answer (section 1.3 explains why that proves nothing).
Two weeks later, after better logging is added, the security team wants to search prompts for signs of injection, but the privacy officer objects that full prompts contain clients' financial details and should not be readable by the whole SOC.
By the end of this section you will be able to specify the telemetry that answers every one of the team's questions, and design it so the privacy officer can agree.
What to collect
Traditional logs show that a request reached a server. AI investigations need to know what the model saw, what it decided, and what it did. That requires telemetry from every layer, joined by a common trace ID so one user request can be followed end to end [1][3]. Open standards such as the OpenTelemetry semantic conventions for generative AI help make this consistent across tools [15]. Figure 5.3 shows the sources and where they flow.
| Layer | What to log |
|---|---|
| Gateway | Caller identity, application, timestamps, the prompt (or a hash and masked copy), token counts, verdicts of input and output filters |
| Agent / harness | The plan, each step, which tool was chosen, approvals requested and given, loop counts, on-behalf-of identities |
| Retrieval | The query, the IDs and sources of chunks returned, and the result of the permission check. Without this, oversharing and indirect injection are almost impossible to investigate |
| Model | Exact model and version, key settings (temperature, system prompt version), the output, latency and errors |
| Tools | Which tool, with what parameters, what it returned, which credential was used |
| Platform and identity | Registry, pipeline and configuration changes; token issuance and OAuth grants |
Worked example: one trace, end to end
With trace-level telemetry, the WealthDesk question becomes a single lookup:
| Time | Layer | Event (trace 7f3a…) |
|---|---|---|
| 10:14:02.110 | Gateway | rm-0412 · prompt (masked): "Summarize the portfolio for [CLIENT-TOKEN-88]" · input filter: pass |
| 10:14:02.140 | Retrieval | query → 6 chunks returned · permission check: SKIPPED (cache hit) · sources: client-88 portfolio docs |
| 10:14:03.900 | Model | model v-2026-09 (pinned) · system prompt v14 · output: portfolio summary |
| 10:14:04.020 | Gateway | output filter: pass · 1,840 tokens |
The cause is visible immediately: a retrieval cache returned chunks cached for another user and skipped the permission check. It is a bug, not an attack. Searching the retrieval logs for permission check: SKIPPED then answers "did it happen to anyone else?" (it did: 37 times in 9 days).
Correlation, automation and threat intelligence
Send AI telemetry to the SIEM and correlate it with identity, endpoint, network and cloud logs. Useful correlations:
- an AI account that suddenly queries at volume from a new location (possible stolen key);
- an agent tool call to an external address shortly after the agent retrieved a document from an untrusted source (possible indirect injection);
- a model provider key used from an IP range outside Meridian (section 4.7).
SOAR (security orchestration, automation and response) can execute safe, pre-approved responses automatically: block a key that breaches extraction thresholds, tighten an input filter, quarantine a document that triggered injection detections, suspend an agent, or open an incident with the full trace attached. Keep automated actions reversible, and leave consequential decisions to people.
Use and share AI threat intelligence: new jailbreak and injection techniques, malicious models and packages, indicators from AI supply-chain incidents, and adversary techniques mapped to MITRE ATLAS [10]. Sources include vendor advisories, national CERTs (CERT-In's LiteLLM advisory in section 3.7 is an example), industry sharing groups, and government–industry AI incident collaboration arrangements [19].
Protecting logs as evidence
AI logs may become evidence in an investigation, a regulatory inquiry or a court case, so protect their integrity and completeness [13][14]:
- send them to a separate, access-controlled store that the AI system's operators cannot alter;
- use write-once (immutable) storage or append-only logging;
- hash log files or batches so tampering is detectable;
- synchronize clocks across components, or the trace cannot be put in order;
- monitor for gaps in logging (a silent logger is an incident);
- document chain of custody when logs are exported for an investigation.
Privacy-preserving telemetry
Prompts and outputs are full of personal and confidential data, so logging them in full creates a new store of sensitive data, exactly the privacy officer's concern. Balance detection value against privacy with tiered access, as Figure 5.4 shows.
| Tier | Contents | Who and how long |
|---|---|---|
| 1 · Metadata | Identities, sizes, filter scores, tool names, timings | Broad SOC access; longer retention |
| 2 · Masked content | Prompts and outputs with personal data and secrets tokenized consistently | Analysts during investigations |
| 3 · Full content | Raw prompts and outputs | Break-glass only: approved, logged, short retention |
Other techniques: store hashes of full prompts for matching known attack strings; keep full content only for flagged or sampled events; and set retention to what investigations and law require, then delete.
Consistent tokenization is the key design choice. If "Client 88" always becomes [CLIENT-TOKEN-88], analysts can see that the same client appears in 37 suspicious traces without ever seeing who the client is. Injection hunting works on Tier 2 data, because instruction-like phrases ("ignore previous instructions", "send this to") survive masking.
Security lens. Ask of any AI logging design: "Could we reconstruct exactly what the model saw and did for a given request?" and "Who can read a customer's raw prompt, and is that access itself logged?" A good design answers yes to the first and "very few, with approval, and yes" to the second.
Real-time behavioral monitoring
Beyond logs, watch AI behavior live:
- refusal and filter-trigger rates;
- output distribution and performance drift (section 1.3);
- agent actions per task and tool error rates;
- resource use: tokens, latency and cost per user and per agent.
Pair monitors with automatic, reversible guards:
| Monitor | Guard |
|---|---|
| An agent exceeds its normal action rate | Pause the agent pending review |
| Harmful-output rate spikes after a model update | Roll back automatically to the previous version |
| Spend exceeds a budget | Throttle and alert |
| Permission-check skips appear in retrieval logs | Disable the cache path; alert the owner |
The opening case, resolved
- The telemetry that answers every question: trace-level logging across gateway, retrieval (including the permission-check result), model (with version and prompt version) and tools, in the SIEM, with synchronized clocks.
- What it showed: a retrieval cache that skipped the permission check, 37 times in 9 days. A real-time guard on "permission check skipped" would have caught the first occurrence.
- Satisfying the privacy officer: tiered access with consistent tokenization. The SOC hunts on metadata and masked content; full prompts are break-glass only, approved, logged and retained briefly.
- Evidence quality: an immutable, separately controlled log store with hashing and chain of custody, so the findings stand up if a client complains to a regulator.
Check your understanding
Try each before reading the reasoning.
1. An investigation into a suspected indirect prompt injection needs to find which document carried the payload. Which log is essential?
Reasoning: retrieval logs recording the IDs and sources of chunks returned for each request, joined by trace ID to the model output and tool calls. Without them, the payload document cannot be identified.
2. A SOAR playbook automatically deletes any user account that triggers three injection alerts. What is wrong?
Reasoning: automated actions should be reversible and proportionate. Suspend or throttle the account and open an incident; leave deletion (a consequential, irreversible action) to a person.
3. A team proposes logging every prompt in full, forever, "for security". How should you respond?
Reasoning: that creates a large store of sensitive data. Use tiered telemetry (metadata broadly, masked content for analysts, full content break-glass only), consistent tokenization, hashing for matching, and a defined retention period.
Key takeaways
- Log what the model saw, decided and did, at every layer, joined by a trace ID. Retrieval logs must include the permission-check result.
- Correlate AI telemetry in the SIEM with identity, endpoint, network and cloud data. Use SOAR for safe, reversible first responses, and AI threat intelligence mapped to ATLAS.
- Protect logs as evidence: separate store, immutability, hashing, clock sync, gap monitoring and chain of custody.
- Make telemetry privacy-preserving: tiered access, consistent tokenization, hashing, sampling and retention limits.
- Monitor AI behavior live and pair monitors with automatic, reversible guards.