AI Threat Modelling, Red Teaming & Assurance, 30 min read

The adversarial ML attack catalogue

In this lesson you will:

  • Explain evasion attacks and adversarial examples in white-box and black-box settings
  • Distinguish poisoning and backdoor attacks
  • Explain model extraction, model inversion and membership inference
  • Describe availability attacks including compute and token exhaustion

Adversarial machine learning studies how models can be attacked by manipulating their inputs, training data or access. These attacks apply to classic ML (fraud, malware, vision) as well as to generative models.

Adversarial attacks mapped to the ML lifecycle stage they target.
Figure 7.3 Where each attack strikes: training time (poisoning, backdoors) or inference time (evasion, extraction, inversion, membership inference, availability).

1. Attacker knowledge

  • White-box — the attacker knows the model's architecture and weights; can compute gradients directly.
  • Grey-box — partial knowledge (architecture, training data source, or a similar model).
  • Black-box — only query access to inputs and outputs.

Black-box attacks are realistic for public APIs. Transferability — adversarial inputs crafted against one model often fool another — means a surrogate (Module 5) can turn black-box into effectively white-box.

2. Evasion attacks

An evasion attack modifies an input at inference time so the model makes the wrong decision, while the input still looks normal to humans or still achieves its malicious purpose.

  • Adversarial examples — small, carefully computed perturbations (e.g. using gradient methods) that flip a classifier's output.
  • Physical-world evasion — patches, stickers, glasses or clothing patterns that defeat vision systems.
  • Problem-space evasion — modifying malware so it still works but a detector classifies it benign; rephrasing spam to bypass a filter.

Defences: adversarial training, input transformations and preprocessing, ensembles, detection of out-of-distribution inputs, rate-limiting query access (slows black-box search), and not relying on a single model for critical decisions.

3. Poisoning and backdoors

Covered in depth in Module 4 — here, the attacker's view:

Attack When Goal Detectability
Availability poisoning Training Degrade overall performance Often visible in metrics
Targeted poisoning Training Misclassify specific inputs Hard — overall metrics look fine
Backdoor / trojan Training or model distribution Hidden behaviour on a trigger Very hard — normal tests pass
Clean-label poisoning Training Poison without changing labels Hard — labels look correct

4. Privacy attacks on models

Model extraction, model inversion and membership inference compared.
Figure 7.4 Three ways attackers learn things they shouldn't from a model: copy it, reconstruct its data, or test who was in its training set.
Attack Question the attacker answers Example
Model extraction "Can I copy this model?" Query an API to train a surrogate
Model inversion "What do typical training examples look like?" Reconstruct a recognisable face for a name class in a face-recognition model
Membership inference "Was this specific record in the training set?" Determine whether a patient's record was used to train a hospital model — itself a disclosure
Training-data extraction "Can I make it repeat training data?" Prompt an LLM to reproduce memorised personal data or secrets

Defences: differential privacy in training, reducing overfitting, output minimisation (labels instead of confidences), rate limits and monitoring, deduplicating training data, and filtering outputs for memorised sensitive content.

5. Availability attacks

AI inference is expensive, so attackers target cost and capacity:

  • Compute exhaustion — inputs that maximise processing (very long prompts, high-resolution images, inputs designed to maximise energy or latency — "sponge examples").
  • Token exhaustion — prompts that trigger extremely long outputs or recursive reasoning; filling context windows to crowd out instructions.
  • Agent loops — inducing an agent to call tools repeatedly or delegate endlessly.
  • Denial of wallet — driving pay-per-use costs until budgets are exhausted.
  • Quota abuse — many free or trial accounts consuming shared capacity.

Defences: input and output caps, per-identity quotas, timeouts, iteration limits, budget alerts and hard caps, anomaly detection on usage.

Worked example — evasion against a phishing classifier

A bank's email-security classifier flags phishing with 98% recall in testing. Attackers adapt:

Attacker move Effect Defence added
Replace text with an image of the text Text model sees almost nothing OCR on images; image-based features
Insert invisible Unicode and homoglyphs Tokens no longer match learned patterns Unicode normalisation before classification
Probe with thousands of test emails from free accounts Black-box search for bypasses Rate limits on external submission; monitor for probing patterns
Use an LLM to rephrase lures into polite business language Style features no longer match Retrain with adversarial examples; add sender reputation and link analysis so the text model isn't the only signal

Lesson: a test-set score measures performance against yesterday's attackers. Robustness needs adversarial testing and multiple independent signals.

🔐 Security lens

Any model exposed to adversaries — fraud, spam, malware, moderation, biometric — should be red-teamed for evasion, not only measured for accuracy.

Key takeaways

  • Attacker knowledge ranges from white-box to black-box; transferability lets surrogates bridge the gap.
  • Evasion manipulates inputs at inference; poisoning and backdoors manipulate training.
  • Extraction copies models; inversion reconstructs representative data; membership inference reveals who was in the training set; extraction attacks can make LLMs regurgitate data.
  • Availability attacks target compute, tokens, agent loops and budgets.
  • Defend with adversarial training, DP, output minimisation, rate limits, caps and multiple independent signals.

Self-check

  1. Why does transferability matter for black-box attackers?
  2. Why can membership inference be a privacy breach on its own?
  3. Give two defences against token-exhaustion attacks.