6.R Review and references
In this lesson you will:
- Recap the main points of Domain 6
- Test yourself on ten exam-style review questions
- Look up the sources cited in this domain's sections
This page closes Domain 6, AI Threats, Testing and Lifecycle Management. Citation numbers in square brackets in the sections, such as [3], refer to the reference list at the end of this page.
Domain summary
- Use STRIDE for systematic component coverage, PASTA for risk-centric analysis and MAESTRO for layered agentic systems; draw on ATLAS, OWASP and the NIST taxonomy as threat libraries.
- Profile threat actors by motivation, capability and access, including insiders and non-human agents.
- Evasion, poisoning, extraction, inversion and membership inference remain the classic families; generative, RAG, agentic and multimodal systems add injection, leakage, excessive agency, memory poisoning and unbounded consumption.
- Judge robustness claims by their threat model and by testing against strong, adaptive attacks.
- Build security gates into every AIDLC stage; separate research, development, training and production environments.
- Put models, data, retrieval, prompts, guardrails and tools under change control; release them as one manifest, roll out by shadow and canary with pre-agreed triggers, and roll back the whole bundle.
- Read certifications and SOC 2 reports for scope and exceptions; check licenses along the pedigree; verify provenance with hashes, signatures and attestations; use AIBOM with VEX.
- Red teaming is a scoped, signed, measured loop; measure attack success rate together with false refusal rate; combine human creativity with automated scale; keep AI-assisted testing in scope.
- Jailbreaks exploit competing objectives and mismatched generalization; injections replace the application's instructions with the attacker's.
- Triage findings on severity, exploitability, reproducibility, impact, reach and detectability; route vulnerabilities, limitations and safety issues to the right owners; package reproductions with success rates.
- Target mitigations at the attack mechanism; tune for both missed attacks and false positives; validate with regression suites of variants behind release gates; monitor residual risk with KRIs and canaries.
- Empirical robustness is practical; certified robustness is provable but narrow. Guardrail models are one layer, never the only one.
Review questions
Answer all ten before opening the answers.
1. A team is designing a multi-agent system in which agents delegate tasks through several layers of frameworks and tools. Which threat modeling method is MOST suited?
- A. STRIDE only
- B. A SWOT analysis
- C. MAESTRO
- D. CVSS scoring
2. A red team reports that its jailbreak succeeded "once". What is the MOST important missing information?
- A. The tester's name
- B. The success rate over many attempts and the exact configuration
- C. The model's training cost
- D. The color of the user interface
3. A system blocks 0% of jailbreak attempts after a fix but also refuses 35% of legitimate customer questions. What is the BEST assessment?
- A. The fix is unacceptable: false refusal rate is too high; tune or redesign
- B. Excellent: attack success rate is zero
- C. Ignore the false refusals
- D. Raise the temperature
4. Which item makes a rollback of an AI system complete?
- A. Restoring the previous model only
- B. Restarting the servers
- C. Clearing the cache only
- D. Restoring every component in the previous release manifest: model, prompts, guardrails, retrieval configuration and tools
5. A supplier provides a SOC 2 Type II report. Which check is MOST important before relying on it for an AI service?
- A. The report's page count
- B. The auditor's logo
- C. Whether the AI service is in scope, the exceptions noted, and the controls the customer must operate
- D. Whether it is longer than last year's
6. A new vulnerability is announced in an ML library. The AIBOM shows it in 12 systems. What helps determine which ones actually need urgent action?
- A. Model cards
- B. VEX statements on exploitability
- C. Rate limits
- D. Differential privacy
7. Which statement BEST distinguishes a jailbreak from a prompt injection?
- A. A jailbreak makes the model break its own safety rules; an injection makes it follow an attacker's instructions in place of the application's
- B. They are identical
- C. Injections only affect images
- D. Jailbreaks only affect agents
8. A model limitation (miscounting table rows) is reported as a critical security vulnerability. What is the BEST action?
- A. Escalate to the CISO as critical
- B. Ignore it
- C. Shut down the system
- D. Reclassify it as a model limitation and route it to the model or product owner, unless it creates a security impact
9. A vendor claims its image classifier is "robust to adversarial attacks". What would MOST increase confidence in the claim?
- A. A marketing brochure
- B. A higher price
- C. An evaluation stating the threat model and showing results against strong, adaptive attacks
- D. The number of parameters
10. Which approach to guardrail models is MOST appropriate?
- A. Use one guardrail model as the only control before agent tools
- B. Evaluate on your own data, test against adversarial inputs, and deploy as one layer of defense in depth
- C. Avoid testing guardrails because they are vendor products
- D. Rely on the system prompt instead
Answers and explanations
| Q | Answer | Explanation |
|---|---|---|
| 1 | C | MAESTRO is designed for layered, multi-agent architectures where threats cross layers. |
| 2 | B | Non-determinism makes single results meaningless; rates and configuration make findings reproducible. |
| 3 | A | Attack success must be balanced against false refusals; a control that blocks normal use fails. |
| 4 | D | Behavior depends on the whole bundle; partial rollback leaves a mismatched system. |
| 5 | C | Scope, exceptions and complementary user controls decide whether the report covers your risk. |
| 6 | B | VEX states whether each product is actually affected, so effort goes where it matters. |
| 7 | A | Jailbreak targets the model's safety; injection hijacks the application's instructions. |
| 8 | D | Honest classification routes work to the right owner and keeps security focus on real vulnerabilities. |
| 9 | C | Credible robustness claims state the threat model and test against adaptive attacks. |
| 10 | B | Guardrail models can be evaded; evaluate them and layer them. |
References
Numbers in square brackets in the text refer to this list. Items [1] to [11] are on ISC2's suggested reference list for the AI Security pilot exam; [10], [11], [31] and [32] are recommended further reading for this domain.
ISC2 suggested references
- Sotiropoulos, J. (2024). Adversarial AI Attacks, Mitigations, and Defense Strategies. Packt Publishing.
- Wilson, S. (2024). The Developer's Playbook for Large Language Model Security. O'Reilly Media.
- Sood, A. (2024). Combating Cyberattacks Targeting the AI Ecosystem. Mercury Learning and Information.
- Yildirim, O. (2026). Agentic AI for Offensive Cybersecurity. Packt Publishing.
- Chebbi, C. (2018). Advanced Infrastructure Penetration Testing. Packt Publishing.
- Ransome, J., and Misra, A. (2018). Core Software Security. Auerbach Publications.
- Crossley, C. (2024). Software Supply Chain Security. O'Reilly Media.
- Malik, V., Huang, K., and Dawson, A. (2025). AI-Native LLM Security. Packt Publishing.
- Ping, D. (2024). The Machine Learning Solutions Architect Handbook, 2nd ed. Packt Publishing.
- Manoj, R. K., et al. (2026). AI Trust, Risk, and Security Management. Wiley-Scrivener.
- Sayles, J. (2024). Principles of AI Governance and Model Risk Management. Apress.
Frameworks, standards and specifications
- Vassilev, A., et al. (2025). Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, NIST AI 100-2e2025. csrc.nist.gov/pubs/ai/100/2/e2025/final
- MITRE Corporation. ATLAS: Adversarial Threat Landscape for Artificial-Intelligence Systems. atlas.mitre.org
- OWASP GenAI Security Project (2024). OWASP Top 10 for LLM Applications 2025. genai.owasp.org/llm-top-10
- OWASP GenAI Security Project. Agentic Security Initiative (agentic AI threats and mitigations). genai.owasp.org
- Huang, K. (2025). Agentic AI Threat Modeling Framework: MAESTRO. Cloud Security Alliance. cloudsecurityalliance.org
- Shostack, A. (2014). Threat Modeling: Designing for Security. Wiley. (STRIDE.)
- UcedaVélez, T., and Morana, M. M. (2015). Risk Centric Threat Modeling: Process for Attack Simulation and Threat Analysis. Wiley. (PASTA.)
- Booth, H., Souppaya, M., Vassilev, A., Ogata, M., Stanley, M., and Scarfone, K. (2024). Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile, NIST SP 800-218A. csrc.nist.gov/pubs/sp/800/218/a/final
- OpenSSF. SLSA: Supply-chain Levels for Software Artifacts. slsa.dev
- Sigstore project. sigstore.dev
- OWASP CycloneDX (ML-BOM) cyclonedx.org and SPDX 3.0 AI and Dataset profiles spdx.dev.
- CISA (2023). Minimum Requirements for Vulnerability Exploitability eXchange (VEX). cisa.gov
- MITRE Corporation. Common Weakness Enumeration (CWE). cwe.mitre.org
Research papers and tools
- Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. (2018). Toward deep learning models resistant to adversarial attacks. ICLR. arXiv:1706.06083
- Cohen, J., Rosenfeld, E., and Kolter, J. Z. (2019). Certified adversarial robustness via randomized smoothing. ICML. arXiv:1902.02918
- Carlini, N., et al. (2019). On evaluating adversarial robustness. arXiv:1902.06705
- Zou, A., Wang, Z., Kolter, J. Z., and Fredrikson, M. (2023). Universal and transferable adversarial attacks on aligned language models. arXiv:2307.15043
- Wei, A., Haghtalab, N., and Steinhardt, J. (2023). Jailbroken: How does LLM safety training fail? NeurIPS. arXiv:2307.02483
- Microsoft. PyRIT: Python Risk Identification Tool for generative AI github.com/Azure/PyRIT; NVIDIA. garak: LLM vulnerability scanner github.com/NVIDIA/garak.
- National Institute of Standards and Technology (2024). AI RMF: Generative Artificial Intelligence Profile, NIST AI 600-1. doi.org/10.6028/NIST.AI.600-1
- Kirchenbauer, J., et al. (2023). A watermark for large language models. ICML. arXiv:2301.10226
- Inan, H., et al. (2023). Llama Guard: LLM-based input-output safeguard for human-AI conversations. arXiv:2312.06674