AI Threat Modelling, Red Teaming & Assurance, 30 min read

Planning and running AI red-team engagements

In this lesson you will:

  • Define scope, objectives and test scenarios for AI security assessments
  • Plan adversarial testing of GenAI, RAG and agentic architectures
  • Apply AI red-teaming methodologies and rules of engagement
  • Use AI-assisted tools to scale testing and discover novel attack paths

AI red teaming is structured adversarial testing of an AI system — its model, data, integrations and the humans around it — to find ways it can be made to fail, leak or cause harm. It combines traditional penetration testing with techniques specific to models.

1. Scoping

AI red team scoping canvas: system, objectives, scenarios, constraints.
Figure 7.7 A scoping canvas. Agree it in writing before testing begins.

Agree in writing:

  • System under test — model version, prompts, guardrails, retrieval sources, tools, agents, environments. Pin versions so results are reproducible.
  • Objectives — what would count as a significant finding: data exfiltration, unauthorised actions, safety-policy violations, harmful content, availability impact, bias.
  • Threat actors to emulate — external customer, malicious insider, compromised document source, rogue partner agent.
  • In scope / out of scope — e.g. production vs staging, social engineering of staff, provider infrastructure (usually out of scope without the provider's permission).
  • Rules of engagement — test accounts, data to use, rate limits, stop conditions, how to handle discovered real personal data or harmful outputs, communication channels.
  • Success criteria and reporting — severity scheme, evidence requirements, retest commitments.

2. Test scenarios

Build scenarios from the threat model, the OWASP LLM Top 10 and MITRE ATLAS. For each scenario define the attacker, entry point, goal and how success is measured.

Category Example scenarios
Direct injection & jailbreak Override system instructions; extract the system prompt; produce prohibited content
Indirect injection Poisoned document in the index; malicious email to an inbox agent; hidden text on a web page
Data disclosure Retrieve another user's records; extract training data; enumerate the knowledge base
Excessive agency Make the agent call a tool beyond the user's rights; trigger irreversible actions without approval
Output handling Generate output that executes as script, SQL or shell downstream
Availability Token exhaustion; agent loops; expensive inputs
Bias and safety Disparate responses across demographic variations of the same request
Supply chain Malicious MCP server or adapter in the environment

3. Methodology

A typical AI red-team engagement:

  1. Reconnaissance — understand capabilities, tools, data sources and guardrails from documentation and probing.
  2. Baseline — establish normal behaviour and refusal patterns.
  3. Manual exploration — creative, multi-turn attacks by skilled testers; domain experts for harms specific to the business (finance, health).
  4. Automated testing — large attack libraries and generated variants to measure success rates at scale.
  5. Chaining — combine weaknesses into end-to-end attack paths (injection → tool misuse → exfiltration).
  6. Documentation — capture prompts, context, settings, outputs and success rates as you go.
  7. Debrief and retest — walk developers through findings; retest after fixes.

Because outputs are non-deterministic, report attack success rate (e.g. "succeeded in 14 of 50 attempts") rather than "it worked once".

Red-team engagement loop from recon to retest.
Figure 7.8 The red-team loop. Automated scale and human creativity complement each other.

4. Testing GenAI, RAG and agentic architectures

  • Test the whole system, not the bare model. Guardrails, retrieval and tools change outcomes dramatically.
  • Test through every entry point — chat, file upload, email ingestion, API, retrieved documents, tool results.
  • For RAG — plant test documents (in a controlled copy of the index) and check whether they are retrieved, believed and obeyed; test ACL filtering with users of different entitlements.
  • For agents — use canary resources (fake secrets, honeypot endpoints) to detect exfiltration; test approval gates, budgets and kill switches; attempt delegation abuse in multi-agent systems.
  • For multimodal — images, audio and documents with hidden instructions.

5. AI-assisted security tools

AI now helps test AI:

  • Automated attack generation — models generate and mutate jailbreak and injection attempts; open-source frameworks (for example PyRIT and garak) orchestrate large test campaigns against target systems.
  • Attacker–judge loops — one model attacks, another scores whether the target's output violates policy, enabling thousands of iterations.
  • Fuzzing of tools and parsers behind agents.
  • Coverage mapping — tracking which OWASP/ATLAS categories have been exercised.

Caveats: automated judges make mistakes, so sample and verify; automated tools find known patterns faster than novel ones — keep skilled human testers for creative attack paths; and keep test traffic within rules of engagement and provider terms.

Worked example — scoping a red team for a claims agent

Item Agreed
System Claims agent v2.3, prompt v14, guardrail config v6, staging environment with synthetic policyholders
Objectives Unauthorised payment; disclosure of another policyholder's data; approval-gate bypass; unsafe advice
Emulated actors Dishonest claimant; attacker sending malicious documents; compromised partner garage agent (A2A)
Out of scope Production; the model provider's infrastructure; staff phishing
Rules Test accounts only; 50 attempts per automated scenario; stop and report immediately on any real-data exposure
Evidence Full transcripts, settings, success rates; mapping to OWASP LLM Top 10 and ATLAS
Retest Within 2 weeks of fixes; results feed the release gate

🔐 Security lens

Turn every confirmed red-team finding into an automated regression test. The red team's job is to find new failures; the pipeline's job is to make sure old ones never return.

Key takeaways

  • Scope in writing: system and versions, objectives, emulated actors, boundaries, rules of engagement, reporting and retest.
  • Build scenarios from the threat model, OWASP and ATLAS — across injection, disclosure, agency, output handling, availability, bias and supply chain.
  • Combine manual creativity with automated scale; report success rates, not single successes.
  • Test the full system through every entry point; use canaries and controlled RAG copies.
  • AI-assisted tools scale testing but need human verification and creativity.

Self-check

  1. Why should model, prompt and guardrail versions be pinned during a red-team engagement?
  2. Why report attack success rates rather than "it worked"?
  3. How can canary resources help detect agent data exfiltration?