AI Threats, Testing and Lifecycle Management, 30 min read

6.R Review and references

In this lesson you will:

  • Recap the main points of Domain 6
  • Test yourself on ten exam-style review questions
  • Look up the sources cited in this domain's sections

This page closes Domain 6, AI Threats, Testing and Lifecycle Management. Citation numbers in square brackets in the sections, such as [3], refer to the reference list at the end of this page.

Domain summary

  • Use STRIDE for systematic component coverage, PASTA for risk-centric analysis and MAESTRO for layered agentic systems; draw on ATLAS, OWASP and the NIST taxonomy as threat libraries.
  • Profile threat actors by motivation, capability and access, including insiders and non-human agents.
  • Evasion, poisoning, extraction, inversion and membership inference remain the classic families; generative, RAG, agentic and multimodal systems add injection, leakage, excessive agency, memory poisoning and unbounded consumption.
  • Judge robustness claims by their threat model and by testing against strong, adaptive attacks.
  • Build security gates into every AIDLC stage; separate research, development, training and production environments.
  • Put models, data, retrieval, prompts, guardrails and tools under change control; release them as one manifest, roll out by shadow and canary with pre-agreed triggers, and roll back the whole bundle.
  • Read certifications and SOC 2 reports for scope and exceptions; check licenses along the pedigree; verify provenance with hashes, signatures and attestations; use AIBOM with VEX.
  • Red teaming is a scoped, signed, measured loop; measure attack success rate together with false refusal rate; combine human creativity with automated scale; keep AI-assisted testing in scope.
  • Jailbreaks exploit competing objectives and mismatched generalization; injections replace the application's instructions with the attacker's.
  • Triage findings on severity, exploitability, reproducibility, impact, reach and detectability; route vulnerabilities, limitations and safety issues to the right owners; package reproductions with success rates.
  • Target mitigations at the attack mechanism; tune for both missed attacks and false positives; validate with regression suites of variants behind release gates; monitor residual risk with KRIs and canaries.
  • Empirical robustness is practical; certified robustness is provable but narrow. Guardrail models are one layer, never the only one.

Review questions

Answer all ten before opening the answers.

1. A team is designing a multi-agent system in which agents delegate tasks through several layers of frameworks and tools. Which threat modeling method is MOST suited?

  • A. STRIDE only
  • B. A SWOT analysis
  • C. MAESTRO
  • D. CVSS scoring

2. A red team reports that its jailbreak succeeded "once". What is the MOST important missing information?

  • A. The tester's name
  • B. The success rate over many attempts and the exact configuration
  • C. The model's training cost
  • D. The color of the user interface

3. A system blocks 0% of jailbreak attempts after a fix but also refuses 35% of legitimate customer questions. What is the BEST assessment?

  • A. The fix is unacceptable: false refusal rate is too high; tune or redesign
  • B. Excellent: attack success rate is zero
  • C. Ignore the false refusals
  • D. Raise the temperature

4. Which item makes a rollback of an AI system complete?

  • A. Restoring the previous model only
  • B. Restarting the servers
  • C. Clearing the cache only
  • D. Restoring every component in the previous release manifest: model, prompts, guardrails, retrieval configuration and tools

5. A supplier provides a SOC 2 Type II report. Which check is MOST important before relying on it for an AI service?

  • A. The report's page count
  • B. The auditor's logo
  • C. Whether the AI service is in scope, the exceptions noted, and the controls the customer must operate
  • D. Whether it is longer than last year's

6. A new vulnerability is announced in an ML library. The AIBOM shows it in 12 systems. What helps determine which ones actually need urgent action?

  • A. Model cards
  • B. VEX statements on exploitability
  • C. Rate limits
  • D. Differential privacy

7. Which statement BEST distinguishes a jailbreak from a prompt injection?

  • A. A jailbreak makes the model break its own safety rules; an injection makes it follow an attacker's instructions in place of the application's
  • B. They are identical
  • C. Injections only affect images
  • D. Jailbreaks only affect agents

8. A model limitation (miscounting table rows) is reported as a critical security vulnerability. What is the BEST action?

  • A. Escalate to the CISO as critical
  • B. Ignore it
  • C. Shut down the system
  • D. Reclassify it as a model limitation and route it to the model or product owner, unless it creates a security impact

9. A vendor claims its image classifier is "robust to adversarial attacks". What would MOST increase confidence in the claim?

  • A. A marketing brochure
  • B. A higher price
  • C. An evaluation stating the threat model and showing results against strong, adaptive attacks
  • D. The number of parameters

10. Which approach to guardrail models is MOST appropriate?

  • A. Use one guardrail model as the only control before agent tools
  • B. Evaluate on your own data, test against adversarial inputs, and deploy as one layer of defense in depth
  • C. Avoid testing guardrails because they are vendor products
  • D. Rely on the system prompt instead
Answers and explanations
Q Answer Explanation
1 C MAESTRO is designed for layered, multi-agent architectures where threats cross layers.
2 B Non-determinism makes single results meaningless; rates and configuration make findings reproducible.
3 A Attack success must be balanced against false refusals; a control that blocks normal use fails.
4 D Behavior depends on the whole bundle; partial rollback leaves a mismatched system.
5 C Scope, exceptions and complementary user controls decide whether the report covers your risk.
6 B VEX states whether each product is actually affected, so effort goes where it matters.
7 A Jailbreak targets the model's safety; injection hijacks the application's instructions.
8 D Honest classification routes work to the right owner and keeps security focus on real vulnerabilities.
9 C Credible robustness claims state the threat model and test against adaptive attacks.
10 B Guardrail models can be evaded; evaluate them and layer them.

References

Numbers in square brackets in the text refer to this list. Items [1] to [11] are on ISC2's suggested reference list for the AI Security pilot exam; [10], [11], [31] and [32] are recommended further reading for this domain.

ISC2 suggested references

  1. Sotiropoulos, J. (2024). Adversarial AI Attacks, Mitigations, and Defense Strategies. Packt Publishing.
  2. Wilson, S. (2024). The Developer's Playbook for Large Language Model Security. O'Reilly Media.
  3. Sood, A. (2024). Combating Cyberattacks Targeting the AI Ecosystem. Mercury Learning and Information.
  4. Yildirim, O. (2026). Agentic AI for Offensive Cybersecurity. Packt Publishing.
  5. Chebbi, C. (2018). Advanced Infrastructure Penetration Testing. Packt Publishing.
  6. Ransome, J., and Misra, A. (2018). Core Software Security. Auerbach Publications.
  7. Crossley, C. (2024). Software Supply Chain Security. O'Reilly Media.
  8. Malik, V., Huang, K., and Dawson, A. (2025). AI-Native LLM Security. Packt Publishing.
  9. Ping, D. (2024). The Machine Learning Solutions Architect Handbook, 2nd ed. Packt Publishing.
  10. Manoj, R. K., et al. (2026). AI Trust, Risk, and Security Management. Wiley-Scrivener.
  11. Sayles, J. (2024). Principles of AI Governance and Model Risk Management. Apress.

Frameworks, standards and specifications

  1. Vassilev, A., et al. (2025). Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, NIST AI 100-2e2025. csrc.nist.gov/pubs/ai/100/2/e2025/final
  2. MITRE Corporation. ATLAS: Adversarial Threat Landscape for Artificial-Intelligence Systems. atlas.mitre.org
  3. OWASP GenAI Security Project (2024). OWASP Top 10 for LLM Applications 2025. genai.owasp.org/llm-top-10
  4. OWASP GenAI Security Project. Agentic Security Initiative (agentic AI threats and mitigations). genai.owasp.org
  5. Huang, K. (2025). Agentic AI Threat Modeling Framework: MAESTRO. Cloud Security Alliance. cloudsecurityalliance.org
  6. Shostack, A. (2014). Threat Modeling: Designing for Security. Wiley. (STRIDE.)
  7. UcedaVélez, T., and Morana, M. M. (2015). Risk Centric Threat Modeling: Process for Attack Simulation and Threat Analysis. Wiley. (PASTA.)
  8. Booth, H., Souppaya, M., Vassilev, A., Ogata, M., Stanley, M., and Scarfone, K. (2024). Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile, NIST SP 800-218A. csrc.nist.gov/pubs/sp/800/218/a/final
  9. OpenSSF. SLSA: Supply-chain Levels for Software Artifacts. slsa.dev
  10. Sigstore project. sigstore.dev
  11. OWASP CycloneDX (ML-BOM) cyclonedx.org and SPDX 3.0 AI and Dataset profiles spdx.dev.
  12. CISA (2023). Minimum Requirements for Vulnerability Exploitability eXchange (VEX). cisa.gov
  13. MITRE Corporation. Common Weakness Enumeration (CWE). cwe.mitre.org

Research papers and tools

  1. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. (2018). Toward deep learning models resistant to adversarial attacks. ICLR. arXiv:1706.06083
  2. Cohen, J., Rosenfeld, E., and Kolter, J. Z. (2019). Certified adversarial robustness via randomized smoothing. ICML. arXiv:1902.02918
  3. Carlini, N., et al. (2019). On evaluating adversarial robustness. arXiv:1902.06705
  4. Zou, A., Wang, Z., Kolter, J. Z., and Fredrikson, M. (2023). Universal and transferable adversarial attacks on aligned language models. arXiv:2307.15043
  5. Wei, A., Haghtalab, N., and Steinhardt, J. (2023). Jailbroken: How does LLM safety training fail? NeurIPS. arXiv:2307.02483
  6. Microsoft. PyRIT: Python Risk Identification Tool for generative AI github.com/Azure/PyRIT; NVIDIA. garak: LLM vulnerability scanner github.com/NVIDIA/garak.
  7. National Institute of Standards and Technology (2024). AI RMF: Generative Artificial Intelligence Profile, NIST AI 600-1. doi.org/10.6028/NIST.AI.600-1
  8. Kirchenbauer, J., et al. (2023). A watermark for large language models. ICML. arXiv:2301.10226
  9. Inan, H., et al. (2023). Llama Guard: LLM-based input-output safeguard for human-AI conversations. arXiv:2312.06674