Secure AI Architecture, Guardrails & Agentic IAM, 30 min read

Guardrails and controls for AI behaviour

In this lesson you will:

  • Design input and output handling, including prompt sanitisation, content filtering and context boundary validation
  • Build architectural guardrails for autonomous behaviour with harnesses and orchestration
  • Plan functional, performance, load and stress testing for AI systems
  • Validate exception handling and fail-safe behaviour

Guardrails are the controls around a model that constrain what goes in, what comes out and what actions follow. They exist because the model itself cannot be fully trusted to follow instructions.

1. The guardrail stack

Layered guardrails around a model: input, context, model, output and action layers.
Figure 6.8 Guardrails in layers. No single layer stops everything; together they make attacks expensive and failures contained.

Input handling

  • Prompt sanitisation — normalise encodings; strip hidden characters; enforce length limits; reject unsupported file types.
  • Injection and jailbreak detection — classifiers and heuristics that flag known attack patterns. Useful, but bypassable — never the only control.
  • Topic and policy filters — block requests outside the system's purpose.
  • PII and secret detection — redact before sending to a model.

Context handling — context boundary validation

The context window mixes trusted and untrusted text (Module 1). Context boundary validation keeps them apart as far as possible:

  • Mark untrusted content — wrap retrieved documents and tool results in clear delimiters and label them as data, not instructions.
  • Spotlighting / data marking — transform untrusted text (for example, by encoding or interleaving markers) so the model can more easily distinguish it.
  • Instruction hierarchy — models trained to prioritise system and developer instructions over content in data.
  • Minimise context — include only what the task needs; fewer documents means a smaller injection surface.
  • Validate retrieved content — check sources, freshness and classification before it enters the context.

These measures reduce prompt-injection success rates; none eliminates it. That is why the action layer matters most.

Output handling

  • Content filtering — harmful, toxic or policy-violating output.
  • Sensitive-data filtering — PII, secrets and confidential information (DLP).
  • Grounding checks — does the answer match retrieved sources? Are citations real?
  • Schema validation — structured output must match the expected format.
  • Safe rendering — escape before display; never pass raw output to interpreters.

Action handling — architectural guardrails for autonomous behaviour

For agents, the decisive controls sit in the harness (the orchestration code that runs the agent loop), not in the prompt:

Agent harness enforcing policy between the model's proposed actions and real tools.
Figure 6.9 The harness sits between what the model proposes and what actually happens. Policy is enforced in code the model cannot rewrite.
  • Policy engine — every proposed tool call is checked against policy (who, what, which resource, how much) before execution.
  • Allow-listed tools and parameters; deny by default.
  • Approval gates for high-impact or irreversible actions; show the human the exact action and its context.
  • Budgets and limits — iterations, tool calls, spend, time, delegation depth.
  • Plan review — for complex tasks, a plan is checked (by rules or a human) before execution.
  • Dual-model patterns — a separate checker model, without access to untrusted content, reviews proposed actions; or a privileged planner never sees untrusted data while a quarantined model processes it.
  • Kill switch — the ability to halt an agent or all agents immediately.

2. Testing AI systems

Test type Purpose AI-specific notes
Functional Does it do what it should? Use evaluation sets with expected behaviour; test many phrasings, not one; repeat to account for non-determinism
Performance Latency and throughput at expected load Measure with guardrails enabled — they add latency
Load Behaviour at peak volume Include long prompts, large files, concurrent agent tasks
Stress Behaviour beyond limits Push past quotas, exhaust context windows, saturate GPUs; confirm graceful degradation
Adversarial / security Resistance to attack Injection, jailbreak, extraction, abuse cases (Module 7)
Regression Nothing broke after a change Re-run all suites after model, prompt, tool or guardrail changes

Exception handling and fail-safe validation

Deliberately break things and confirm the system fails safe:

  • Model API times out → user gets a clear message or a fallback, not a half-completed action.
  • Guardrail service is down → requests are blocked or routed to humans, not passed unfiltered.
  • Tool returns an error mid-task → the agent stops or rolls back; it does not retry indefinitely or improvise a workaround.
  • Output fails schema validation → no downstream action is taken.
  • Model returns low confidence → escalate to a human.

Worked example — guardrails for an email-handling agent

An agent triages a shared support inbox and can draft replies, create tickets and issue refunds up to ₹2,000.

Layer Guardrail
Input Emails normalised; attachments limited to PDF and images; injection classifier flags suspicious emails for human review
Context Email content wrapped and labelled as untrusted customer data; only the customer's own order history retrieved
Output Drafts filtered for PII of other customers and for commitments outside policy
Action Policy engine: refunds only to the order's original payment method, ≤ ₹2,000, max one per order; anything else → human approval queue; no tool can send email to addresses outside the ticket
Limits 10 tool calls per email; 2-minute timeout; daily refund budget with alert at 80%
Fail-safe If the policy service is unreachable, all actions pause and emails go to the human queue

Test result: in red-team testing, 6% of injected emails still convinced the model to propose a refund to a new account — and 0% succeeded, because the policy engine blocked every one.

🔐 Security lens

Measure guardrails in two numbers: how often the model was manipulated (it will be, sometimes) and how often manipulation achieved anything. Architecture drives the second number to zero.

Key takeaways

  • Layer guardrails on input, context, output and action; detection classifiers help but are bypassable.
  • Context boundary validation — marking, spotlighting, instruction hierarchy, minimal context — reduces injection, but doesn't eliminate it.
  • The harness enforces policy, allow-lists, approvals, budgets, plan review and kill switches in code the model cannot change.
  • Test functionally (many phrasings, repeated runs), for performance and load with guardrails on, and stress past limits.
  • Validate that every failure path fails safe.

Self-check

  1. Why is an injection classifier insufficient as the only defence?
  2. What does the harness's policy engine check before a tool call executes?
  3. If the guardrail service fails, what should happen to user requests?