Secure design principles for AI systems
In this lesson you will:
- Apply Zero Trust, defence in depth, privacy by design and secure defaults to AI
- Reason about architectural trade-offs between latency, explainability, cost and security
- Evaluate build versus buy, and hosted versus self-managed, options for AI
The classic security design principles still hold. What changes is where you apply them: an AI system has a component — the model — that can be persuaded by its inputs, and increasingly it can act. Design must assume that component will sometimes be wrong or manipulated.
1. Principles, applied to AI
Zero Trust (ZT) — never trust, always verify; assume breach.
- Treat the model as an untrusted component: its output is checked before it reaches anything that matters.
- Every agent, tool and service authenticates; every request is authorised against policy, per call.
- No implicit trust from network location — an agent inside the VPC is not trusted because of where it runs.
- Continuous evaluation: permissions can be withdrawn mid-session if behaviour looks wrong.
Defence in depth (DiD) — multiple independent layers, so one failure is not a breach.
- Input filtering and context isolation and output validation and least-privilege tools and monitoring.
- No single guardrail — especially not the system prompt — is relied on to stop prompt injection.
Privacy by design — minimise data in prompts, retrieval and logs; keep personal data in erasable stores; default to no training on user content (Module 4).
Secure defaults — new agents start with no tools; new connectors with read-only scopes; retrieval filtered by user by default; logging on; external network egress off.
Two further principles matter especially for AI:
- Least privilege — for models, agents, tools and pipelines (Lesson 6.6).
- Fail safe — when a guardrail errors or the model is uncertain, default to the safe outcome (refuse, escalate to a human), not the permissive one.
2. Architectural trade-offs
There is rarely one right architecture. Security professionals need to make trade-offs explicit.
| Trade-off | Example |
|---|---|
| Latency vs security | Each extra filter or second-model check adds response time; streaming output is faster but harder to filter before users see it |
| Explainability vs accuracy | An interpretable model may be less accurate than a deep model; for high-impact decisions, explainability may win |
| Cost vs capability | Large frontier models cost more per call; smaller models are cheaper, faster, and easier to host privately |
| Autonomy vs control | More autonomy means faster outcomes and less human effort — and larger blast radius |
| Context vs exposure | More retrieved context improves answers but widens what injected instructions or leaks can reach |
Document the chosen trade-off and the reason, and revisit it as volumes, threats or regulations change.
3. Build vs buy; hosted vs self-managed
| Option | Advantages | Security considerations |
|---|---|---|
| SaaS AI feature | Fastest; provider handles most operations | Least control; depends on vendor due diligence, contract and configuration |
| Hosted model API | Frontier capability; no GPU operations | Data leaves your boundary; provider retention and residency; version changes; you build all guardrails |
| Managed model in your cloud tenancy | Data stays in your cloud; provider-managed infrastructure | Shared responsibility with cloud provider; still dependent on model provider updates |
| Self-hosted open model | Full control of data, version and configuration | You own patching, hardening, scaling, GPU security, model supply chain and safety tuning |
| Build your own model | Maximum control and differentiation | Highest cost and expertise; full responsibility for data, training security and evaluation |
Decision factors: data sensitivity and residency; required capability; latency; cost at expected volume; in-house skills; regulatory expectations; exit strategy and concentration risk.
Worked example — choosing a deployment for a bank's document assistant
| Requirement | Implication |
|---|---|
| Customer KYC documents; data must stay in India | Rules out providers without an in-country region and zero-retention terms |
| Answers must cite sources for auditors | RAG with citations; any model can do this |
| 3,000 staff, steady volume | Self-hosted cost becomes competitive at this scale |
| Small ML-ops team | Fully self-managed open model is risky to run |
Decision: a managed model deployed in the bank's own cloud tenancy in an Indian region, with a self-hosted smaller open model as a tested fallback. Trade-off recorded: slightly lower capability than the newest hosted frontier model, accepted for residency and control.
🔐 Security lens
Write "the model is untrusted" into your architecture principles. It changes design reviews: people stop asking "will the model behave?" and start asking "what happens when it doesn't?"
Key takeaways
- Zero Trust treats the model, agents and tools as untrusted and verifies every request; defence in depth avoids relying on any single guardrail.
- Secure defaults: no tools, read-only scopes, user-filtered retrieval, logging on, egress off. Fail safe when uncertain.
- Make latency, explainability, cost, autonomy and context trade-offs explicit and documented.
- Build vs buy and hosted vs self-managed decisions shift responsibility, control and risk — choose against data, capability, skills and exit needs.
Self-check
- What does "treat the model as an untrusted component" mean in practice?
- Give two secure defaults for a newly created agent.
- Why might streaming output complicate output filtering?