Secure AI Architecture, Guardrails & Agentic IAM, 25 min read

Secure design principles for AI systems

In this lesson you will:

  • Apply Zero Trust, defence in depth, privacy by design and secure defaults to AI
  • Reason about architectural trade-offs between latency, explainability, cost and security
  • Evaluate build versus buy, and hosted versus self-managed, options for AI

The classic security design principles still hold. What changes is where you apply them: an AI system has a component — the model — that can be persuaded by its inputs, and increasingly it can act. Design must assume that component will sometimes be wrong or manipulated.

1. Principles, applied to AI

Four secure design principles with their AI-specific application.
Figure 6.1 Four design principles and what each means in an AI system.

Zero Trust (ZT) — never trust, always verify; assume breach.

  • Treat the model as an untrusted component: its output is checked before it reaches anything that matters.
  • Every agent, tool and service authenticates; every request is authorised against policy, per call.
  • No implicit trust from network location — an agent inside the VPC is not trusted because of where it runs.
  • Continuous evaluation: permissions can be withdrawn mid-session if behaviour looks wrong.

Defence in depth (DiD) — multiple independent layers, so one failure is not a breach.

  • Input filtering and context isolation and output validation and least-privilege tools and monitoring.
  • No single guardrail — especially not the system prompt — is relied on to stop prompt injection.

Privacy by design — minimise data in prompts, retrieval and logs; keep personal data in erasable stores; default to no training on user content (Module 4).

Secure defaults — new agents start with no tools; new connectors with read-only scopes; retrieval filtered by user by default; logging on; external network egress off.

Two further principles matter especially for AI:

  • Least privilege — for models, agents, tools and pipelines (Lesson 6.6).
  • Fail safe — when a guardrail errors or the model is uncertain, default to the safe outcome (refuse, escalate to a human), not the permissive one.

2. Architectural trade-offs

There is rarely one right architecture. Security professionals need to make trade-offs explicit.

Trade-off triangle between latency, explainability and cost with design options plotted.
Figure 6.2 Latency, explainability and cost pull against each other — and every guardrail adds some latency and cost.
Trade-off Example
Latency vs security Each extra filter or second-model check adds response time; streaming output is faster but harder to filter before users see it
Explainability vs accuracy An interpretable model may be less accurate than a deep model; for high-impact decisions, explainability may win
Cost vs capability Large frontier models cost more per call; smaller models are cheaper, faster, and easier to host privately
Autonomy vs control More autonomy means faster outcomes and less human effort — and larger blast radius
Context vs exposure More retrieved context improves answers but widens what injected instructions or leaks can reach

Document the chosen trade-off and the reason, and revisit it as volumes, threats or regulations change.

3. Build vs buy; hosted vs self-managed

Option Advantages Security considerations
SaaS AI feature Fastest; provider handles most operations Least control; depends on vendor due diligence, contract and configuration
Hosted model API Frontier capability; no GPU operations Data leaves your boundary; provider retention and residency; version changes; you build all guardrails
Managed model in your cloud tenancy Data stays in your cloud; provider-managed infrastructure Shared responsibility with cloud provider; still dependent on model provider updates
Self-hosted open model Full control of data, version and configuration You own patching, hardening, scaling, GPU security, model supply chain and safety tuning
Build your own model Maximum control and differentiation Highest cost and expertise; full responsibility for data, training security and evaluation

Decision factors: data sensitivity and residency; required capability; latency; cost at expected volume; in-house skills; regulatory expectations; exit strategy and concentration risk.

Worked example — choosing a deployment for a bank's document assistant

Requirement Implication
Customer KYC documents; data must stay in India Rules out providers without an in-country region and zero-retention terms
Answers must cite sources for auditors RAG with citations; any model can do this
3,000 staff, steady volume Self-hosted cost becomes competitive at this scale
Small ML-ops team Fully self-managed open model is risky to run

Decision: a managed model deployed in the bank's own cloud tenancy in an Indian region, with a self-hosted smaller open model as a tested fallback. Trade-off recorded: slightly lower capability than the newest hosted frontier model, accepted for residency and control.

🔐 Security lens

Write "the model is untrusted" into your architecture principles. It changes design reviews: people stop asking "will the model behave?" and start asking "what happens when it doesn't?"

Key takeaways

  • Zero Trust treats the model, agents and tools as untrusted and verifies every request; defence in depth avoids relying on any single guardrail.
  • Secure defaults: no tools, read-only scopes, user-filtered retrieval, logging on, egress off. Fail safe when uncertain.
  • Make latency, explainability, cost, autonomy and context trade-offs explicit and documented.
  • Build vs buy and hosted vs self-managed decisions shift responsibility, control and risk — choose against data, capability, skills and exit needs.

Self-check

  1. What does "treat the model as an untrusted component" mean in practice?
  2. Give two secure defaults for a newly created agent.
  3. Why might streaming output complicate output filtering?