Ready reckoner
The family and the attack surface
AI ⊃ machine learning ⊃ deep learning ⊃ generative AI. All production AI is narrow AI. An AI system = code + data + learned behavior: the data, the training process and the model file are attack surfaces that code review never sees.
| Paradigm | Learns from | Typical weakness |
|---|---|---|
| Supervised | Labeled examples | Label poisoning ("control the labels, control the model") |
| Unsupervised | Structure in unlabeled data | Shifting baselines, slow-burn evasion |
| Self-supervised | Labels made from the data (next token) | Absorbed poison and personal data |
| Reinforcement | Rewards | Reward hacking, feedback poisoning |
- Overfitting = memorization = privacy risk (extraction, membership inference).
- Self-attention lets any input token influence the output → root cause of prompt injection.
- A quantized, pruned or fine-tuned model is a new model: re-test it.
- CIA for AI covers weights, labels, prompts and compute budgets; trustworthy AI adds safety, fairness, transparency, accountability.
Generative AI building blocks
| Concept | Remember |
|---|---|
| Token | Unit of text; drives limits, cost, latency. Spikes = abuse signal |
| Sampling | Output chosen from a probability list → answers vary; test many times, log what the model saw |
| Context window | Everything the model sees; rules, questions and retrieved text are equal to it |
| Embedding / vector search | Finds what is relevant, not what is permitted → permission-aware retrieval |
| RAG | For changing or per-user knowledge |
| Prompting | For behavior instructions can fix |
| Fine-tuning | Only when behavior needs it and the data is governed |
| Agent | Model + tools + memory + loop. Reduce what it can do before detecting what it should not do |
Prompts guide; controls outside the model enforce.
Limitations and performance
- Hallucination is inherent → ground, cite, abstain, verify, human review for high impact.
- Non-determinism → one test proves little; re-asking is not reproducing; logs are evidence.
- Drift: data, concept, label, behavioral. Silent; monitor for it.
- Model collapse: training on model output erodes rare cases first → provenance, label synthetic data.
- Rare bad class → accuracy misleads. Use precision (of flagged, how many are bad) and recall (of bad, how many flagged). Set thresholds from the cost of each error.
- Oversight chosen by impact and reversibility, not accuracy. Watch automation bias and near-zero override rates.
Exam reflexes
- "Untrusted text acted as an instruction" → prompt injection, enabled by a shared context window.
- "Re-ran it and it was fine" → proves nothing; use logs.
- "99% accurate fraud model" with 0.5% fraud → ask for precision and recall.
- "The model will enforce access" → wrong; enforcement sits outside the model.