1.R Review and references
In this lesson you will:
- Recap the main points of Domain 1
- Test yourself on ten exam-style review questions
- Look up the sources cited in this domain's sections
This page closes Domain 1, AI Concepts and Security Fundamentals. Citation numbers in square brackets in the sections, such as [3], refer to the reference list at the end of this page.
Domain summary
- An AI system is code plus data plus learned behavior. The training data, the training process and the model weights are all part of the attack surface.
- Generative AI is a subset of deep learning, which is a subset of machine learning, which is a subset of AI. Each learning paradigm has a typical weakness: label poisoning for supervised learning, shifting baselines for unsupervised, reward hacking for reinforcement, and absorbed poison and personal data for self-supervised pre-training.
- Overfitting is a privacy risk as well as a quality problem, because memorized data can be extracted.
- Self-attention lets any part of a transformer's input influence the output. That is why instructions hidden in documents, emails or tool results can steer an LLM.
- The CIA triad still applies, extended to weights, prompts and compute budgets, and AI frameworks add safety, fairness, transparency and accountability.
- Prompts are guidance, not controls. Enforce security outside the model: permission-aware retrieval for RAG, least-privilege tools and approval gates for agents.
- Use RAG for knowledge that changes or must respect permissions; fine-tuning bakes data into weights, can weaken safety, and cannot filter per user.
- Hallucination, non-determinism, drift and model collapse are inherent. Design controls that assume errors, monitor for drift, and keep data provenance.
- Accuracy misleads on imbalanced data. Use precision and recall, and set thresholds by the cost of each kind of error.
- Match human oversight to the impact and reversibility of the decision, and make it meaningful.
Review questions
Answer all ten before opening the answers.
1. A spam filter is trained on emails that users have marked "spam" or "not spam". Which learning paradigm is this, and what attack is it most exposed to?
- A. Unsupervised; baseline shifting
- B. Supervised; label poisoning
- C. Reinforcement; reward hacking
- D. Self-supervised; embedding inversion
2. A model scores 99% on its training data and 71% on new data. What is the MOST likely issue, and why does it matter for security?
- A. Overfitting; memorized training data may be extracted
- B. Underfitting; it is easy to evade
- C. Concept drift; outputs are stale
- D. Data drift; labels are poisoned
3. Why can an instruction hidden in a retrieved document change an LLM's behavior?
- A. Retrieved documents are executed as code
- B. The vector database rewrites the system prompt
- C. Embeddings are decrypted inside the model
- D. Self-attention lets any input text influence the output, and the model cannot reliably tell instructions from data
4. An attacker floods a public LLM endpoint with very long prompts, sharply increasing the provider's bill. Which security property is MOST directly affected?
- A. Confidentiality
- B. Integrity
- C. Availability
- D. Non-repudiation
5. An organization wants a management system for AI that an external auditor can certify. Which standard fits BEST?
- A. NIST AI RMF
- B. ISO/IEC 42001
- C. MITRE ATLAS
- D. NIST AI 100-2
6. A bank's assistant must answer from internal policies that change monthly, and each employee may see only documents they are entitled to. Which approach is BEST?
- A. RAG with permission-aware retrieval
- B. Fine-tune monthly on all policies
- C. A system prompt listing restricted topics
- D. Paste every policy into a large context window
7. A product team says its system prompt forbids the model from revealing customer data, so no other control is needed. What is the BEST response?
- A. Agree, if the prompt has been tested
- B. Strengthen the wording of the prompt
- C. Raise the temperature to reduce memorization
- D. Enforce data access outside the model, because prompts are guidance, not controls
8. An email agent can read mail, send mail and delete files, but it only needs to summarize incoming mail. Which risk does this MOST illustrate?
- A. Model collapse
- B. Hallucination
- C. Excessive agency
- D. Concept drift
9. A malware classifier flags 80 files, of which 60 are malicious. Another 40 malicious files were not flagged. What is its recall?
- A. 0.75
- B. 0.60
- C. 0.80
- D. 0.40
10. An AI system recommends whether to close customers' bank accounts. Its accuracy is 98%. Which oversight mode is MOST appropriate?
- A. Human-in-the-loop
- B. Fully automated with monthly sampling
- C. Human-on-the-loop
- D. None, because accuracy is above 95%
Answers and explanations
| Q | Answer | Explanation |
|---|---|---|
| 1 | B | Labeled examples mean supervised learning; whoever controls labels controls what the model learns. |
| 2 | A | A large gap between training and test performance indicates overfitting, and memorized data can be extracted. |
| 3 | D | Attention mixes all input text; the model has no built-in boundary between instructions and data. |
| 4 | C | Resource and cost exhaustion attacks availability ("denial of wallet"). |
| 5 | B | ISO/IEC 42001 is the certifiable AI management system standard; the others guide risk or describe attacks. |
| 6 | A | RAG handles changing knowledge, and access checks at retrieval prevent oversharing. Fine-tuning would expose all policies to everyone. |
| 7 | D | Prompt instructions can be bypassed; real access controls must enforce the rule. |
| 8 | C | The agent has more tools and permissions than its task requires. |
| 9 | B | Recall = 60 / (60 + 40) = 0.60. Option A is the precision (60 / 80). |
| 10 | A | The decision is high-impact and hard to reverse, so a person should approve each one, whatever the accuracy. |
References
Numbers in square brackets in the text refer to this list. Books [1] to [7] are on ISC2's suggested reference list for the AI Security pilot exam; the remaining sources are the primary frameworks and research papers behind specific points. The explanations in this domain's sections are our own synthesis; consult the sources for full treatment.
ISC2 suggested references
- Santos, O., and Radanliev, P. (2024). Beyond the Algorithm: AI, Security, Privacy, and Ethics. Addison-Wesley Professional.
- Ping, D. (2024). The Machine Learning Solutions Architect Handbook, 2nd ed. Packt Publishing.
- Nelson, C. (2021). Is Building Secure ML Possible? O'Reilly Media.
- Malik, V., Huang, K., and Dawson, A. (2025). AI-Native LLM Security. Packt Publishing.
- Wilson, S. (2024). The Developer's Playbook for Large Language Model Security. O'Reilly Media.
- Lin, B. (2026). Privacy and Security for Large Language Models. O'Reilly Media.
- Sotiropoulos, J. (2024). Adversarial AI Attacks, Mitigations, and Defense Strategies. Packt Publishing.
Frameworks, standards and law
- National Institute of Standards and Technology (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1. doi.org/10.6028/NIST.AI.100-1
- National Institute of Standards and Technology (2024). Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1. doi.org/10.6028/NIST.AI.600-1
- Vassilev, A., Oprea, A., Fordyce, A., Anderson, H., Davies, X., and Hamin, M. (2025). Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, NIST AI 100-2e2025. National Institute of Standards and Technology. csrc.nist.gov/pubs/ai/100/2/e2025/final
- ISO/IEC 42001:2023. Information technology — Artificial intelligence — Management system. International Organization for Standardization. iso.org/standard/81230.html
- ISO/IEC 23894:2023. Information technology — Artificial intelligence — Guidance on risk management. International Organization for Standardization.
- OWASP GenAI Security Project (2024). OWASP Top 10 for LLM Applications 2025. genai.owasp.org/llm-top-10. OWASP has since published a 2026 edition; check it for changes to entry names.
- MITRE Corporation. ATLAS: Adversarial Threat Landscape for Artificial-Intelligence Systems. atlas.mitre.org
- European Union (2024). Regulation (EU) 2024/1689 laying down harmonized rules on artificial intelligence (Artificial Intelligence Act). Official Journal of the European Union. eur-lex.europa.eu/eli/reg/2024/1689/oj
Research papers
- Vaswani, A., et al. (2017). Attention is all you need. Advances in Neural Information Processing Systems 30. arXiv:1706.03762
- Lewis, P., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems 33. arXiv:2005.11401
- Hu, E. J., et al. (2021). LoRA: Low-rank adaptation of large language models. arXiv:2106.09685
- Morris, J. X., Kuleshov, V., Shmatikov, V., and Rush, A. M. (2023). Text embeddings reveal (almost) as much as text. arXiv:2310.06816
- Shumailov, I., et al. (2024). AI models collapse when trained on recursively generated data. Nature 631, 755–759. doi.org/10.1038/s41586-024-07566-y
- Greshake, K., et al. (2023). Not what you've signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection. arXiv:2302.12173
- Goodfellow, I. J., Shlens, J., and Szegedy, C. (2015). Explaining and harnessing adversarial examples. International Conference on Learning Representations. arXiv:1412.6572
- Carlini, N., et al. (2021). Extracting training data from large language models. 30th USENIX Security Symposium. arXiv:2012.07805
- Model Context Protocol. Specification. modelcontextprotocol.io