The adversarial ML attack catalogue
In this lesson you will:
- Explain evasion attacks and adversarial examples in white-box and black-box settings
- Distinguish poisoning and backdoor attacks
- Explain model extraction, model inversion and membership inference
- Describe availability attacks including compute and token exhaustion
Adversarial machine learning studies how models can be attacked by manipulating their inputs, training data or access. These attacks apply to classic ML (fraud, malware, vision) as well as to generative models.
1. Attacker knowledge
- White-box — the attacker knows the model's architecture and weights; can compute gradients directly.
- Grey-box — partial knowledge (architecture, training data source, or a similar model).
- Black-box — only query access to inputs and outputs.
Black-box attacks are realistic for public APIs. Transferability — adversarial inputs crafted against one model often fool another — means a surrogate (Module 5) can turn black-box into effectively white-box.
2. Evasion attacks
An evasion attack modifies an input at inference time so the model makes the wrong decision, while the input still looks normal to humans or still achieves its malicious purpose.
- Adversarial examples — small, carefully computed perturbations (e.g. using gradient methods) that flip a classifier's output.
- Physical-world evasion — patches, stickers, glasses or clothing patterns that defeat vision systems.
- Problem-space evasion — modifying malware so it still works but a detector classifies it benign; rephrasing spam to bypass a filter.
Defences: adversarial training, input transformations and preprocessing, ensembles, detection of out-of-distribution inputs, rate-limiting query access (slows black-box search), and not relying on a single model for critical decisions.
3. Poisoning and backdoors
Covered in depth in Module 4 — here, the attacker's view:
| Attack | When | Goal | Detectability |
|---|---|---|---|
| Availability poisoning | Training | Degrade overall performance | Often visible in metrics |
| Targeted poisoning | Training | Misclassify specific inputs | Hard — overall metrics look fine |
| Backdoor / trojan | Training or model distribution | Hidden behaviour on a trigger | Very hard — normal tests pass |
| Clean-label poisoning | Training | Poison without changing labels | Hard — labels look correct |
4. Privacy attacks on models
| Attack | Question the attacker answers | Example |
|---|---|---|
| Model extraction | "Can I copy this model?" | Query an API to train a surrogate |
| Model inversion | "What do typical training examples look like?" | Reconstruct a recognisable face for a name class in a face-recognition model |
| Membership inference | "Was this specific record in the training set?" | Determine whether a patient's record was used to train a hospital model — itself a disclosure |
| Training-data extraction | "Can I make it repeat training data?" | Prompt an LLM to reproduce memorised personal data or secrets |
Defences: differential privacy in training, reducing overfitting, output minimisation (labels instead of confidences), rate limits and monitoring, deduplicating training data, and filtering outputs for memorised sensitive content.
5. Availability attacks
AI inference is expensive, so attackers target cost and capacity:
- Compute exhaustion — inputs that maximise processing (very long prompts, high-resolution images, inputs designed to maximise energy or latency — "sponge examples").
- Token exhaustion — prompts that trigger extremely long outputs or recursive reasoning; filling context windows to crowd out instructions.
- Agent loops — inducing an agent to call tools repeatedly or delegate endlessly.
- Denial of wallet — driving pay-per-use costs until budgets are exhausted.
- Quota abuse — many free or trial accounts consuming shared capacity.
Defences: input and output caps, per-identity quotas, timeouts, iteration limits, budget alerts and hard caps, anomaly detection on usage.
Worked example — evasion against a phishing classifier
A bank's email-security classifier flags phishing with 98% recall in testing. Attackers adapt:
| Attacker move | Effect | Defence added |
|---|---|---|
| Replace text with an image of the text | Text model sees almost nothing | OCR on images; image-based features |
| Insert invisible Unicode and homoglyphs | Tokens no longer match learned patterns | Unicode normalisation before classification |
| Probe with thousands of test emails from free accounts | Black-box search for bypasses | Rate limits on external submission; monitor for probing patterns |
| Use an LLM to rephrase lures into polite business language | Style features no longer match | Retrain with adversarial examples; add sender reputation and link analysis so the text model isn't the only signal |
Lesson: a test-set score measures performance against yesterday's attackers. Robustness needs adversarial testing and multiple independent signals.
🔐 Security lens
Any model exposed to adversaries — fraud, spam, malware, moderation, biometric — should be red-teamed for evasion, not only measured for accuracy.
Key takeaways
- Attacker knowledge ranges from white-box to black-box; transferability lets surrogates bridge the gap.
- Evasion manipulates inputs at inference; poisoning and backdoors manipulate training.
- Extraction copies models; inversion reconstructs representative data; membership inference reveals who was in the training set; extraction attacks can make LLMs regurgitate data.
- Availability attacks target compute, tokens, agent loops and budgets.
- Defend with adversarial training, DP, output minimisation, rate limits, caps and multiple independent signals.
Self-check
- Why does transferability matter for black-box attackers?
- Why can membership inference be a privacy breach on its own?
- Give two defences against token-exhaustion attacks.