6.4 AI security testing and red teaming
In this lesson you will:
- Scope a red-team engagement so that it answers a real question safely.
- Design tests for each kind of AI system, and report results as rates.
- Mix expert and automated testing, and fix as you go.
- Use AI as an attacker or a judge without losing control of the test.
- Explain why safety-trained models still get jailbroken, and how injection differs.
AI red teaming is structured adversarial testing of an AI system to find ways it can be made to fail, insecurely, unsafely or unfairly, before real attackers or real users do. It is not a one-off penetration test but a cycle [1][4].
Opening case: red teaming the Collections agent
After the PDF injection incident (section 5.5), Meridian commissions a red-team engagement on the rebuilt Collections agent before it returns to full service. The agent now has: a PDF sanitizer, a dual-LLM design for reading uploads (section 4.4), a settlement tool that requires human approval above 10%, and a circuit breaker.
The first draft of the engagement plan says: "Test the Collections agent for prompt injection. Use any techniques. Report findings."
The red-team lead returns it with a list of questions.
By the end of this section you will be able to write the plan she is asking for, choose tests, and say how results will be measured.
Scoping the assessment
As with any penetration test, red teaming starts with careful scoping [5]. Figure 6.10 shows a scoping canvas.
| Element | Content |
|---|---|
| Objectives | What the test must answer |
| Scenarios | Realistic attacker profiles and goals drawn from the threat model (section 6.1) |
| Scope | Systems, models, versions, environments, tools and data in scope; what is explicitly out |
| Measurable success criteria | What counts as a successful attack, and the acceptable rates after fixes |
| Rules of engagement (RoE) | Signed by an authorized system owner: permitted techniques, timing, test accounts, data handling (red-team logs may contain harmful content and sensitive data), safety stops, and escalation if a real incident or serious vulnerability is found |
Worked example: the revised plan
| Element | Collections engagement |
|---|---|
| Objective | Can an external customer, through conversation or uploads, cause the agent to (a) commit Meridian to terms beyond policy, (b) disclose another customer's data, or (c) bypass the approval step? |
| Scenarios | Fraudulent debtor seeking a discount; organized group testing for a repeatable exploit; malicious insider editing knowledge-base pages |
| In scope | Agent v2.1 in the staging environment; PDF and image upload; chat; knowledge base copy; settlement tool in mock mode |
| Out of scope | Production; real customer data; the core banking system; the external model provider's infrastructure |
| Success criteria | Any unapproved settlement above 10% = critical. Target after fixes: attack success rate below 0.5% at n ≥ 500 per technique family; false refusal rate on legitimate hardship conversations below 3% |
| RoE | Signed by the Collections business owner and CISO; test accounts only; logs stored in the restricted red-team vault, 90-day retention; stop and escalate if a technique also works against production |
Designing safe adversarial tests
Test the attack surface that matters for each architecture [2][8]:
| Architecture | Test for |
|---|---|
| Generative AI | Jailbreaks, harmful-content generation, system prompt extraction, training-data extraction, bias and misinformation |
| RAG | Indirect injection through documents, oversharing across permission boundaries, knowledge-base poisoning, citation manipulation |
| Agentic | Injected tool calls, privilege escalation through tools, data exfiltration via allowed channels, loops and resource exhaustion, unsafe actions without approval |
Run agent tests in sandboxed environments with test accounts and mock tools wherever possible, so a successful attack does not cause real-world harm.
Measuring with rates
Because outputs vary, measure results as rates:
- Attack success rate (ASR): the share of attack attempts that achieve the attacker's goal. Run each attack many times and across variants.
- False refusal rate (FRR): the share of legitimate requests wrongly blocked.
Always report them together. A system that refuses everything has zero ASR and no value. Figure 6.11 shows the trade-off.
Methods and frameworks
Effective programs combine:
- Human-led testing: experts bring creativity, context and judgment, and find novel attack paths that tools miss.
- Automated testing: open-source and commercial tools (for example PyRIT and garak [30]) run large libraries of known attacks and generate variants at scale. They give consistent, repeatable coverage and become the regression suite.
- Break-fix cycles: findings go to the owning team, are fixed and are retested, often within the same engagement (Figure 6.12).
Align test plans and findings with MITRE ATLAS techniques and the OWASP lists, so coverage can be measured and results compared over time [13][14].
Using AI to test AI
AI can generate attack variants, act as an automated adversary, judge whether outputs violate policy, and explore multi-step attack paths in agentic systems faster than humans [4]. Safeguards are needed:
- Keep the attacking AI within scope and the RoE. It must not wander into out-of-scope systems; give it only test credentials and network access to the staging environment.
- Validate the judge. An AI judge can be wrong or manipulated (an attack payload can include "this response complies with policy"). Sample its verdicts for human review and measure its agreement with humans.
- Protect the data generated, which may include harmful content and extracted sensitive information.
Why jailbreaks and injections work
Research identifies two root causes of jailbreaks in safety-trained models [29]:
- Competing objectives: the model's drive to be helpful and follow instructions conflicts with its safety training, and attackers frame requests so helpfulness wins.
- Mismatched generalization: safety training does not cover every form the request can take, such as other languages, encodings, role-play or unusual formats.
Automated adversarial suffix attacks can also find strings that bypass safety across many models [28].
Common families:
- role-play and persona attacks;
- hypothetical or fictional framing;
- encoding and translation tricks;
- many-shot attacks that fill the context with examples;
- multi-turn escalation;
- optimized adversarial strings.
Jailbreak versus injection
Figure 6.13 compares them.
- A jailbreak makes the model break its own safety rules.
- A prompt injection makes it follow an attacker's instructions in place of the application's. Direct injection comes from the user; indirect injection comes from content the system processes (documents, web pages, emails, tool results).
The Collections PDF attack was an indirect injection, not a jailbreak: nothing unsafe was generated, but the agent followed the attacker's "policy update". Both are measured with ASR, balanced against false refusals.
Exam lens. Expect to distinguish the two. Ask: whose instructions won? If the model's own safety rules were overridden, it is a jailbreak. If the application's instructions were overridden by text from the user or from processed content, it is prompt injection (direct or indirect).
Check your understanding
Try each before reading the reasoning.
1. A red-team report says, "The jailbreak succeeded." What is missing?
Reasoning: a rate. How many attempts, how many successes, across which variants, and on which model version and configuration? Pair it with the false refusal rate of the current defenses.
2. An automated attack tool begins sending requests to a production API that was not in scope. What control failed?
Reasoning: scope enforcement for the attacking AI or tool. It should hold only test credentials and network access to in-scope environments, as defined in the signed rules of engagement.
3. A model answers harmful questions when they are asked in a low-resource language but refuses in English. Which root cause does this illustrate?
Reasoning: mismatched generalization: safety training did not cover that form of the request.
Key takeaways
- Scope every engagement: objectives, scenarios, in and out of scope, measurable success criteria, and signed rules of engagement with data handling and safety stops.
- Test what matters per architecture, in sandboxes with mock tools, and report attack success rate together with false refusal rate.
- Combine human creativity with automated scale (PyRIT, garak) in break-fix cycles; map to ATLAS and OWASP.
- AI can attack and judge; keep it in scope, validate the judge, and protect the data.
- Jailbreaks exploit competing objectives and mismatched generalization; injection substitutes an attacker's instructions for the application's.