Secure AI Operations and Incident Response, 30 min read

5.R Review and references

In this lesson you will:

  • Recap the main points of Domain 5
  • Test yourself on ten exam-style review questions
  • Look up the sources cited in this domain's sections

This page closes Domain 5, Secure AI Operations and Incident Response. Citation numbers in square brackets in the sections, such as [3], refer to the reference list at the end of this page.

Domain summary

  • The incident lifecycle is unchanged; its contents change. Prepare with RACI, release criteria, AI playbooks, tabletops and configuration as code.
  • Human override rates need two-sided thresholds: too high suggests degradation or attack, near zero suggests rubber-stamping.
  • Log what the model saw, decided and did, at the gateway, agent, retrieval, model and tool layers, joined by a trace ID; protect logs as evidence and mask sensitive content.
  • Detect with layered signatures, classifiers and anomaly detection; canary tokens reveal prompt or document leakage. Distinguish access misuse from agency misuse.
  • Track conventional and AI-specific vulnerabilities through to verified closure, re-testing many times because of non-determinism; prioritize by exposure, exploitation, data, privileges, controls and consequence.
  • Contain with the least disruptive step that stops the harm: tighten, throttle, reduce capability, isolate, roll back, kill switch. Revoke an agent's credentials before stopping it, and hold pending actions.
  • Preserve evidence in order of volatility, with hashes, immutable storage and chain of custody. Snapshot before cleaning.
  • Eradicate from data, models, derived models, indexes, agent memory, vulnerabilities and access; reintroduce only through regression tests, red-team retest and staged rollout.
  • Recover to a verified trusted state; circuit breakers and kill switches act outside the model, fail safe and are tested.
  • Plan AI continuity with dependency maps, RTO/RPO, alternatives and degraded modes. Run blameless reviews and track the actions.

Review questions

Answer all ten before opening the answers.

1. An AI system's human reviewers have overridden only 0.1% of decisions this quarter, against a testing baseline of 6%. What is the BEST interpretation?

  • A. The model has become nearly perfect
  • B. Reviewers may be rubber-stamping or the review step may be bypassed; investigate
  • C. The baseline was wrong; lower it
  • D. Remove the human review

2. Which log source is MOST essential for investigating an indirect prompt injection in a RAG assistant?

  • A. Retrieval logs showing which chunks were returned for each request, joined by trace ID
  • B. Firewall logs
  • C. HR records
  • D. DNS logs only

3. A unique string planted in a system prompt appears in a chatbot's output on a public forum. What does this MOST likely indicate?

  • A. A hallucination
  • B. Model drift
  • C. Normal behavior
  • D. The system prompt was extracted

4. A summarization agent suddenly starts calling a file-deletion tool shortly after reading external emails. What type of misuse is this MOST likely?

  • A. Access misuse by a stolen key
  • B. Resource exhaustion
  • C. Agency misuse from injected instructions
  • D. Concept drift

5. A jailbreak against a customer-facing agent was fixed with a new filter. It failed in one test attempt. When should the vulnerability be closed?

  • A. Immediately
  • B. After repeated testing with many attempts and variants shows it no longer succeeds
  • C. After a month without reports
  • D. Never

6. Attackers are injecting instructions through a single poisoned knowledge-base article. What is the BEST first containment step?

  • A. Quarantine the poisoned article (after snapshotting it as evidence) and tighten filters
  • B. Shut down the whole AI platform
  • C. Retrain the model
  • D. Wait for the investigation to finish

7. When stopping a compromised agent that has queued outbound payments, what should happen FIRST?

  • A. Delete the agent and its logs
  • B. Let queued actions complete
  • C. Restart the agent
  • D. Revoke the agent's credentials and hold pending actions for review

8. In what order should AI evidence be collected?

  • A. Model artifacts first, agent memory last
  • B. Alphabetical order
  • C. Most volatile first: agent memory and in-flight context, then short-retention logs, configuration, data stores, artifacts
  • D. Whatever is easiest

9. Poisoned data was used to fine-tune a model, which was then distilled into two smaller models. What must eradication include?

  • A. Only the original fine-tuned model
  • B. The data, the fine-tuned model, both derived models, and any indexes or caches built from them
  • C. Only the training data
  • D. Only the smaller models

10. Where should an AI kill switch operate?

  • A. Outside the model, for example by revoking credentials or disabling the endpoint at the gateway
  • B. As an instruction in the system prompt
  • C. Inside the model's weights
  • D. In the user interface only
Answers and explanations
Q Answer Explanation
1 B A collapse in override rate signals automation bias or a bypassed control, not perfection.
2 A Indirect injection arrives in retrieved content; only retrieval logs show which item carried it.
3 D A canary string appearing outside the system shows the prompt was extracted.
4 C An authorized agent acting outside its task after reading untrusted content suggests manipulation.
5 B Non-determinism means one failed attempt proves little; re-test with many attempts and variants.
6 A Use the least disruptive step that stops the harm, preserving evidence first.
7 D Revoke credentials so it cannot act, and hold pending actions rather than letting them run.
8 C Order of volatility: collect what disappears soonest first.
9 B Lineage shows all derived artifacts; each must be cleaned or rebuilt.
10 A A compromised model cannot be trusted to stop itself; the switch must act outside it.

References

Numbers in square brackets in the text refer to this list. Items [1] to [8] are on ISC2's suggested reference list for the AI Security pilot exam; [7] and [8] are recommended further reading for this domain.

ISC2 suggested references

  1. Brotherston, L., Berlin, A., and Reyor, W. F. (2024). Defensive Security Handbook, 2nd ed. O'Reilly Media.
  2. Sood, A. (2024). Combating Cyberattacks Targeting the AI Ecosystem. Mercury Learning and Information.
  3. Malik, V., Huang, K., and Dawson, A. (2025). AI-Native LLM Security. Packt Publishing.
  4. Wilson, S. (2024). The Developer's Playbook for Large Language Model Security. O'Reilly Media.
  5. Sotiropoulos, J. (2024). Adversarial AI Attacks, Mitigations, and Defense Strategies. Packt Publishing.
  6. Wendt, D. W. (2025). AI Strategy and Security. Apress.
  7. Ozkaya, E., and Aslaner, M. (2019). Hands-On Cybersecurity for Finance. Packt Publishing.
  8. Crossley, C. (2024). Software Supply Chain Security. O'Reilly Media.

Standards, frameworks and guidance

  1. Nelson, A., Rekhi, S., Souppaya, M., and Scarfone, K. (2025). Incident Response Recommendations and Considerations for Cybersecurity Risk Management: A CSF 2.0 Community Profile, NIST SP 800-61 Rev. 3. csrc.nist.gov/pubs/sp/800/61/r3/final
  2. MITRE Corporation. ATLAS: Adversarial Threat Landscape for Artificial-Intelligence Systems. atlas.mitre.org
  3. OWASP GenAI Security Project (2024). OWASP Top 10 for LLM Applications 2025. genai.owasp.org/llm-top-10
  4. Vassilev, A., et al. (2025). Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, NIST AI 100-2e2025. csrc.nist.gov/pubs/ai/100/2/e2025/final
  5. Brezinski, D., and Killalea, T. (2002). Guidelines for Evidence Collection and Archiving, RFC 3227. IETF. (Order of volatility.)
  6. Kent, K., Chevalier, S., Grance, T., and Dang, H. (2006). Guide to Integrating Forensic Techniques into Incident Response, NIST SP 800-86.
  7. OpenTelemetry. Semantic Conventions for Generative AI systems. opentelemetry.io
  8. Regulation (EU) 2016/679 (GDPR), Article 33.
  9. CERT-In (2022). Directions under section 70B(6) of the Information Technology Act, 2000, 28 April 2022.
  10. Regulation (EU) 2024/1689 (AI Act), Article 73 (reporting of serious incidents); Directive (EU) 2022/2555 (NIS2), Article 23.
  11. Cybersecurity and Infrastructure Security Agency, Joint Cyber Defense Collaborative (2025). AI Cybersecurity Collaboration Playbook. cisa.gov
  12. FIRST. Common Vulnerability Scoring System v4.0 and Exploit Prediction Scoring System (EPSS); CISA. Known Exploited Vulnerabilities Catalog. first.org · cisa.gov/known-exploited-vulnerabilities-catalog
  13. Swanson, M., et al. (2010). Contingency Planning Guide for Federal Information Systems, NIST SP 800-34 Rev. 1.
  14. ISO 22301:2019. Security and resilience — Business continuity management systems — Requirements.
  15. Beyer, B., Jones, C., Petoff, J., and Murphy, N. R. (eds.) (2016). Site Reliability Engineering: How Google Runs Production Systems, chapter on postmortem culture. O'Reilly Media.