3.R Review and references
In this lesson you will:
- Recap the main points of Domain 3
- Test yourself on ten exam-style review questions
- Look up the sources cited in this domain's sections
This page closes Domain 3, AI Data Management, Privacy and Model Security. Citation numbers in square brackets in the sections, such as [3], refer to the reference list at the end of this page.
Domain summary
- Data owners decide whether data may be used for AI; an AI provider that trains on customer data for its own purposes becomes a controller for that processing.
- Residency is where data is processed, sovereignty is whose law applies, localization is a legal requirement to keep data in-country.
- Classification must travel with data into chunks, embeddings and models; a derived artifact takes the highest classification of its inputs.
- Provenance says where data came from, lineage what happened to it, chain of custody who handled it. Lineage is what lets deletion and incident response reach every affected model.
- Pseudonymized data is still personal data; true anonymization is hard and must be risk-assessed.
- Differential privacy adds calibrated noise; smaller epsilon means stronger privacy and lower accuracy, and privacy loss adds up across queries.
- Poisoning comes in availability, targeted and backdoor forms; backdoors pass normal testing. Defend with provenance, validation, hashing and signed manifests, and use lineage to roll back.
- Extraction, membership inference and model inversion pull data out of models, especially overfitted ones; defend with deduplication, DP, output minimization and rate limits.
- DSPM finds data, CSPM checks infrastructure, DLP stops data leaving, AI-SPM covers AI assets. AI DLP needs both pattern and semantic detection.
- Encryption protects storage and transport, not what a model reveals. Cryptographic erase works only if every copy was encrypted and every copy of the key is destroyed.
- A model registry becomes a control when the pipeline deploys only registered, approved, signed models. Avoid pickle; prefer safe formats.
- Model watermarking is deliberately embedded; fingerprinting measures existing traits. Token-aware limits and budget caps protect availability.
- Pin external datasets and models by hash, use internal mirrors, and plan for compromised releases and inherited foundation-model risk.
Review questions
Answer all ten before opening the answers.
1. A company sends customer support transcripts to an AI provider's API. The provider's terms allow it to use the transcripts to train its own general models. What is the MOST significant change in the privacy position?
- A. None; the provider remains a processor
- B. The company becomes a processor
- C. The transcripts become anonymous
- D. The provider acts as a controller for its training use, which needs its own lawful basis
2. A dataset replaces names with consistent random IDs; a separate key table maps IDs back to names. Under the GDPR, the dataset is:
- A. Anonymized and outside the GDPR
- B. Synthetic data
- C. Pseudonymized and still personal data
- D. Encrypted data exempt from the GDPR
3. A data science team lowers epsilon in its differentially private training from 8 to 1. What is the expected effect?
- A. Weaker privacy, higher accuracy
- B. Stronger privacy, lower accuracy
- C. No change in privacy
- D. The model becomes fully anonymous
4. An image classifier performs normally in all tests, but any image containing a small yellow square is classified as "authorized". What attack is this?
- A. Backdoor poisoning
- B. Availability poisoning
- C. Model extraction
- D. Membership inference
5. An attacker queries a hospital's diagnostic model to determine whether a particular patient's record was used in training. What is this attack called, and which model property makes it easier?
- A. Model inversion; underfitting
- B. Data drift; high temperature
- C. Evasion; quantization
- D. Membership inference; overfitting
6. Which tool BEST answers "where are all the copies of customer data our AI teams have created across our cloud accounts?"
- A. DLP
- B. CSPM
- C. DSPM
- D. SIEM
7. A company must ensure data in a cloud AI platform is unrecoverable before ending the contract, without physical access to the disks. Which method is MOST appropriate?
- A. Clear by overwriting
- B. Cryptographic erase, destroying every copy of the encryption keys
- C. Deleting the files
- D. Degaussing
8. A deployment pipeline should allow only approved models into production. Which control BEST enforces this?
- A. Deploying only signed models whose hash matches an approved entry in the model registry
- B. A policy document
- C. Training developers
- D. Monthly audits
9. A team downloads a model from a public hub in pickle format. What is the main immediate risk?
- A. The model may be biased
- B. It will be slower
- C. It cannot be fine-tuned
- D. Loading it may execute malicious code
10. An organization wants to detect if a model it licenses to customers is leaked, and identify which customer leaked it. What should it use?
- A. Fingerprinting the original model
- B. Rate limiting
- C. A unique watermark in each licensed copy
- D. Differential privacy
Answers and explanations
| Q | Answer | Explanation |
|---|---|---|
| 1 | D | Using the data for its own training purposes makes the provider a controller for that processing. |
| 2 | C | Data that can be re-linked with separately held information is pseudonymized and remains personal data. |
| 3 | B | A smaller epsilon means more noise: stronger privacy, lower accuracy. |
| 4 | A | A hidden trigger causing chosen behavior, with normal behavior otherwise, is a backdoor. |
| 5 | D | Determining training-set membership is membership inference; overfitting makes it easier. |
| 6 | C | DSPM discovers and classifies sensitive data across environments. |
| 7 | B | Cryptographic erase is the practical purge method in cloud, provided all copies were encrypted and all key copies destroyed. |
| 8 | A | Making the registry and signatures a hard deployment gate enforces approval. |
| 9 | D | Pickle files can run arbitrary code on load; prefer safe formats such as safetensors and scan artifacts. |
| 10 | C | A unique watermark per licensee traces which copy leaked; fingerprinting identifies the model but not the licensee. |
References
Numbers in square brackets in the text refer to this list. Items [1] to [12] are on ISC2's suggested reference list for the AI Security pilot exam.
ISC2 suggested references
- Lin, B. (2026). Privacy and Security for Large Language Models. O'Reilly Media.
- Arbuckle, L., and El Emam, K. (2020). Building an Anonymization Pipeline. O'Reilly Media.
- Sotiropoulos, J. (2024). Adversarial AI Attacks, Mitigations, and Defense Strategies. Packt Publishing.
- Nelson, C. (2021). Is Building Secure ML Possible? O'Reilly Media.
- Sayles, J. (2024). Principles of AI Governance and Model Risk Management. Apress.
- Ping, D. (2024). The Machine Learning Solutions Architect Handbook, 2nd ed. Packt Publishing.
- Malik, V., Huang, K., and Dawson, A. (2025). AI-Native LLM Security. Packt Publishing.
- Wilson, S. (2024). The Developer's Playbook for Large Language Model Security. O'Reilly Media.
- Crossley, C. (2024). Software Supply Chain Security. O'Reilly Media.
- Buckwell, M., Van daele, S., and Horst, C. (2024). Security Architecture for Hybrid Cloud. O'Reilly Media.
- Brotherston, L., Berlin, A., and Reyor, W. F. (2024). Defensive Security Handbook, 2nd ed. O'Reilly Media.
- Manoj, R. K., et al. (2026). AI Trust, Risk, and Security Management. Wiley-Scrivener.
Standards, guidance and law
- Vassilev, A., et al. (2025). Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, NIST AI 100-2e2025. csrc.nist.gov/pubs/ai/100/2/e2025/final
- Near, J., Darais, D., Lefkovitz, N., and Howarth, G. (2025). Guidelines for Evaluating Differential Privacy Guarantees, NIST SP 800-226. csrc.nist.gov/pubs/sp/800/226/final
- Chandramouli, R., and Hibbard, E. (2025). Guidelines for Media Sanitization, NIST SP 800-88 Rev. 2. csrc.nist.gov/pubs/sp/800/88/r2/final
- Regulation (EU) 2016/679 (General Data Protection Regulation), including Articles 4(5), 22, 25 and Chapter V. eur-lex.europa.eu/eli/reg/2016/679/oj
- US Department of Health and Human Services. HIPAA Privacy Rule, de-identification standard, 45 CFR § 164.514.
- PCI Security Standards Council (2024). PCI DSS v4.0.1. pcisecuritystandards.org
Research papers
- Dwork, C., and Roth, A. (2014). The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science 9(3–4).
- Abadi, M., et al. (2016). Deep learning with differential privacy. ACM CCS. arXiv:1607.00133
- Shokri, R., Stronati, M., Song, C., and Shmatikov, V. (2017). Membership inference attacks against machine learning models. IEEE Symposium on Security and Privacy. arXiv:1610.05820
- Fredrikson, M., Jha, S., and Ristenpart, T. (2015). Model inversion attacks that exploit confidence information and basic countermeasures. ACM CCS.
- Tramèr, F., et al. (2016). Stealing machine learning models via prediction APIs. USENIX Security Symposium. arXiv:1609.02943
- Carlini, N., et al. (2021). Extracting training data from large language models. USENIX Security Symposium. arXiv:2012.07805
- Gu, T., Dolan-Gavitt, B., and Garg, S. (2017). BadNets: Identifying vulnerabilities in the machine learning model supply chain. arXiv:1708.06733
- Carlini, N., et al. (2024). Poisoning web-scale training datasets is practical. IEEE Symposium on Security and Privacy. arXiv:2302.10149
- Shumailov, I., et al. (2024). AI models collapse when trained on recursively generated data. Nature 631, 755–759. doi.org/10.1038/s41586-024-07566-y
Tools, specifications and advisories
- OpenSSF. SLSA: Supply-chain Levels for Software Artifacts. slsa.dev
- Sigstore project and OpenSSF model signing. sigstore.dev
- Hugging Face. Safetensors documentation. huggingface.co/docs/safetensors
- MITRE Corporation. ATLAS. atlas.mitre.org
- OWASP GenAI Security Project (2024). OWASP Top 10 for LLM Applications 2025, LLM10 Unbounded Consumption. genai.owasp.org/llm-top-10
- CERT-In (2026). Advisory CIAD-2026-0053: LiteLLM PyPI package and container image compromise, 23 October 2026, updated 28 October 2026. cert-in.org.in
- Kirchenbauer, J., et al. (2023). A watermark for large language models. International Conference on Machine Learning. arXiv:2301.10226
- Reserve Bank of India (2018). Storage of Payment System Data, circular RBI/2017-18/153, 6 April 2018.