5.R Review and references
In this lesson you will:
- Recap the main points of Domain 5
- Test yourself on ten exam-style review questions
- Look up the sources cited in this domain's sections
This page closes Domain 5, Secure AI Operations and Incident Response. Citation numbers in square brackets in the sections, such as [3], refer to the reference list at the end of this page.
Domain summary
- The incident lifecycle is unchanged; its contents change. Prepare with RACI, release criteria, AI playbooks, tabletops and configuration as code.
- Human override rates need two-sided thresholds: too high suggests degradation or attack, near zero suggests rubber-stamping.
- Log what the model saw, decided and did, at the gateway, agent, retrieval, model and tool layers, joined by a trace ID; protect logs as evidence and mask sensitive content.
- Detect with layered signatures, classifiers and anomaly detection; canary tokens reveal prompt or document leakage. Distinguish access misuse from agency misuse.
- Track conventional and AI-specific vulnerabilities through to verified closure, re-testing many times because of non-determinism; prioritize by exposure, exploitation, data, privileges, controls and consequence.
- Contain with the least disruptive step that stops the harm: tighten, throttle, reduce capability, isolate, roll back, kill switch. Revoke an agent's credentials before stopping it, and hold pending actions.
- Preserve evidence in order of volatility, with hashes, immutable storage and chain of custody. Snapshot before cleaning.
- Eradicate from data, models, derived models, indexes, agent memory, vulnerabilities and access; reintroduce only through regression tests, red-team retest and staged rollout.
- Recover to a verified trusted state; circuit breakers and kill switches act outside the model, fail safe and are tested.
- Plan AI continuity with dependency maps, RTO/RPO, alternatives and degraded modes. Run blameless reviews and track the actions.
Review questions
Answer all ten before opening the answers.
1. An AI system's human reviewers have overridden only 0.1% of decisions this quarter, against a testing baseline of 6%. What is the BEST interpretation?
- A. The model has become nearly perfect
- B. Reviewers may be rubber-stamping or the review step may be bypassed; investigate
- C. The baseline was wrong; lower it
- D. Remove the human review
2. Which log source is MOST essential for investigating an indirect prompt injection in a RAG assistant?
- A. Retrieval logs showing which chunks were returned for each request, joined by trace ID
- B. Firewall logs
- C. HR records
- D. DNS logs only
3. A unique string planted in a system prompt appears in a chatbot's output on a public forum. What does this MOST likely indicate?
- A. A hallucination
- B. Model drift
- C. Normal behavior
- D. The system prompt was extracted
4. A summarization agent suddenly starts calling a file-deletion tool shortly after reading external emails. What type of misuse is this MOST likely?
- A. Access misuse by a stolen key
- B. Resource exhaustion
- C. Agency misuse from injected instructions
- D. Concept drift
5. A jailbreak against a customer-facing agent was fixed with a new filter. It failed in one test attempt. When should the vulnerability be closed?
- A. Immediately
- B. After repeated testing with many attempts and variants shows it no longer succeeds
- C. After a month without reports
- D. Never
6. Attackers are injecting instructions through a single poisoned knowledge-base article. What is the BEST first containment step?
- A. Quarantine the poisoned article (after snapshotting it as evidence) and tighten filters
- B. Shut down the whole AI platform
- C. Retrain the model
- D. Wait for the investigation to finish
7. When stopping a compromised agent that has queued outbound payments, what should happen FIRST?
- A. Delete the agent and its logs
- B. Let queued actions complete
- C. Restart the agent
- D. Revoke the agent's credentials and hold pending actions for review
8. In what order should AI evidence be collected?
- A. Model artifacts first, agent memory last
- B. Alphabetical order
- C. Most volatile first: agent memory and in-flight context, then short-retention logs, configuration, data stores, artifacts
- D. Whatever is easiest
9. Poisoned data was used to fine-tune a model, which was then distilled into two smaller models. What must eradication include?
- A. Only the original fine-tuned model
- B. The data, the fine-tuned model, both derived models, and any indexes or caches built from them
- C. Only the training data
- D. Only the smaller models
10. Where should an AI kill switch operate?
- A. Outside the model, for example by revoking credentials or disabling the endpoint at the gateway
- B. As an instruction in the system prompt
- C. Inside the model's weights
- D. In the user interface only
Answers and explanations
| Q | Answer | Explanation |
|---|---|---|
| 1 | B | A collapse in override rate signals automation bias or a bypassed control, not perfection. |
| 2 | A | Indirect injection arrives in retrieved content; only retrieval logs show which item carried it. |
| 3 | D | A canary string appearing outside the system shows the prompt was extracted. |
| 4 | C | An authorized agent acting outside its task after reading untrusted content suggests manipulation. |
| 5 | B | Non-determinism means one failed attempt proves little; re-test with many attempts and variants. |
| 6 | A | Use the least disruptive step that stops the harm, preserving evidence first. |
| 7 | D | Revoke credentials so it cannot act, and hold pending actions rather than letting them run. |
| 8 | C | Order of volatility: collect what disappears soonest first. |
| 9 | B | Lineage shows all derived artifacts; each must be cleaned or rebuilt. |
| 10 | A | A compromised model cannot be trusted to stop itself; the switch must act outside it. |
References
Numbers in square brackets in the text refer to this list. Items [1] to [8] are on ISC2's suggested reference list for the AI Security pilot exam; [7] and [8] are recommended further reading for this domain.
ISC2 suggested references
- Brotherston, L., Berlin, A., and Reyor, W. F. (2024). Defensive Security Handbook, 2nd ed. O'Reilly Media.
- Sood, A. (2024). Combating Cyberattacks Targeting the AI Ecosystem. Mercury Learning and Information.
- Malik, V., Huang, K., and Dawson, A. (2025). AI-Native LLM Security. Packt Publishing.
- Wilson, S. (2024). The Developer's Playbook for Large Language Model Security. O'Reilly Media.
- Sotiropoulos, J. (2024). Adversarial AI Attacks, Mitigations, and Defense Strategies. Packt Publishing.
- Wendt, D. W. (2025). AI Strategy and Security. Apress.
- Ozkaya, E., and Aslaner, M. (2019). Hands-On Cybersecurity for Finance. Packt Publishing.
- Crossley, C. (2024). Software Supply Chain Security. O'Reilly Media.
Standards, frameworks and guidance
- Nelson, A., Rekhi, S., Souppaya, M., and Scarfone, K. (2025). Incident Response Recommendations and Considerations for Cybersecurity Risk Management: A CSF 2.0 Community Profile, NIST SP 800-61 Rev. 3. csrc.nist.gov/pubs/sp/800/61/r3/final
- MITRE Corporation. ATLAS: Adversarial Threat Landscape for Artificial-Intelligence Systems. atlas.mitre.org
- OWASP GenAI Security Project (2024). OWASP Top 10 for LLM Applications 2025. genai.owasp.org/llm-top-10
- Vassilev, A., et al. (2025). Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, NIST AI 100-2e2025. csrc.nist.gov/pubs/ai/100/2/e2025/final
- Brezinski, D., and Killalea, T. (2002). Guidelines for Evidence Collection and Archiving, RFC 3227. IETF. (Order of volatility.)
- Kent, K., Chevalier, S., Grance, T., and Dang, H. (2006). Guide to Integrating Forensic Techniques into Incident Response, NIST SP 800-86.
- OpenTelemetry. Semantic Conventions for Generative AI systems. opentelemetry.io
- Regulation (EU) 2016/679 (GDPR), Article 33.
- CERT-In (2022). Directions under section 70B(6) of the Information Technology Act, 2000, 28 April 2022.
- Regulation (EU) 2024/1689 (AI Act), Article 73 (reporting of serious incidents); Directive (EU) 2022/2555 (NIS2), Article 23.
- Cybersecurity and Infrastructure Security Agency, Joint Cyber Defense Collaborative (2025). AI Cybersecurity Collaboration Playbook. cisa.gov
- FIRST. Common Vulnerability Scoring System v4.0 and Exploit Prediction Scoring System (EPSS); CISA. Known Exploited Vulnerabilities Catalog. first.org · cisa.gov/known-exploited-vulnerabilities-catalog
- Swanson, M., et al. (2010). Contingency Planning Guide for Federal Information Systems, NIST SP 800-34 Rev. 1.
- ISO 22301:2019. Security and resilience — Business continuity management systems — Requirements.
- Beyer, B., Jones, C., Petoff, J., and Murphy, N. R. (eds.) (2016). Site Reliability Engineering: How Google Runs Production Systems, chapter on postmortem culture. O'Reilly Media.