AI Concepts and Security Fundamentals, 10 min read

Ready reckoner

The family and the attack surface

AI ⊃ machine learning ⊃ deep learning ⊃ generative AI. All production AI is narrow AI. An AI system = code + data + learned behavior: the data, the training process and the model file are attack surfaces that code review never sees.

Paradigm Learns from Typical weakness
Supervised Labeled examples Label poisoning ("control the labels, control the model")
Unsupervised Structure in unlabeled data Shifting baselines, slow-burn evasion
Self-supervised Labels made from the data (next token) Absorbed poison and personal data
Reinforcement Rewards Reward hacking, feedback poisoning
  • Overfitting = memorization = privacy risk (extraction, membership inference).
  • Self-attention lets any input token influence the output → root cause of prompt injection.
  • A quantized, pruned or fine-tuned model is a new model: re-test it.
  • CIA for AI covers weights, labels, prompts and compute budgets; trustworthy AI adds safety, fairness, transparency, accountability.

Generative AI building blocks

Concept Remember
Token Unit of text; drives limits, cost, latency. Spikes = abuse signal
Sampling Output chosen from a probability list → answers vary; test many times, log what the model saw
Context window Everything the model sees; rules, questions and retrieved text are equal to it
Embedding / vector search Finds what is relevant, not what is permitted → permission-aware retrieval
RAG For changing or per-user knowledge
Prompting For behavior instructions can fix
Fine-tuning Only when behavior needs it and the data is governed
Agent Model + tools + memory + loop. Reduce what it can do before detecting what it should not do

Prompts guide; controls outside the model enforce.

Limitations and performance

  • Hallucination is inherent → ground, cite, abstain, verify, human review for high impact.
  • Non-determinism → one test proves little; re-asking is not reproducing; logs are evidence.
  • Drift: data, concept, label, behavioral. Silent; monitor for it.
  • Model collapse: training on model output erodes rare cases first → provenance, label synthetic data.
  • Rare bad class → accuracy misleads. Use precision (of flagged, how many are bad) and recall (of bad, how many flagged). Set thresholds from the cost of each error.
  • Oversight chosen by impact and reversibility, not accuracy. Watch automation bias and near-zero override rates.

Exam reflexes

  • "Untrusted text acted as an instruction" → prompt injection, enabled by a shared context window.
  • "Re-ran it and it was fine" → proves nothing; use logs.
  • "99% accurate fraud model" with 0.5% fraud → ask for precision and recall.
  • "The model will enforce access" → wrong; enforcement sits outside the model.