AI & GenAI Foundations for Security Professionals, 10 min read

Ready reckoner

The nesting

AI ⊃ ML ⊃ Deep learning ⊃ GenAI. Agentic AI = model + tools + memory + orchestration + autonomy.

Learning paradigms → weak spot

Paradigm Learns from Weak spot
Supervised Labelled examples Label poisoning, distribution shift
Unsupervised Unlabelled structure Baseline drift, slow-burn evasion
Self-supervised Labels made from the data itself (next token) Unvetted web-scale data, memorisation
Reinforcement Rewards from actions Reward hacking, feedback poisoning
Evolutionary Fitness-scored populations Objective gaming; attacker search tool

Architectures

Architecture Data Remember
Feed-forward / MLP Tabular No notion of order or space
CNN Images, grids Adversarial pixels, physical patches
RNN / LSTM / GRU Sequences Sequential, struggles with long range
Transformer Text, multimodal Self-attention; instructions + data share one channel → prompt injection
Diffusion / GAN Images, audio Deepfakes, synthetic media
Mixture-of-experts Large LLMs Routes inputs to sub-networks

Compute

  • GPU — general parallel accelerator. TPU — tensor ASIC. NPU — on-device inference. HBM/VRAM — holds weights + attention cache.
  • Risks: shared-GPU memory residue, side channels, checkpoint files that execute on load, resource exhaustion (denial of wallet), sanitisation of dense storage (use cryptographic erasure).
  • TEE / confidential computing = data and weights protected in use, with attestation.

GenAI building blocks

  • Token ≈ ¾ English word; drives limits, cost, latency. Spikes = abuse/exfiltration signal.
  • Embedding = meaning as a vector; compared by cosine similarity. Not anonymised — inversion possible.
  • Vector DB → need permission-aware retrieval, tenant isolation, write integrity, synced deletion.
  • Context window = all the model sees in one call. No hard trust boundary inside it.
  • Prompt engineering improves behaviour; it is not a security control. Assume the system prompt is public; no secrets in it.

Extending models

Approach Changes Pick when Main risks
Prompting Instructions Behaviour reachable by instructions Injection, extraction
RAG Knowledge at query time Data is current, private, must be deletable Indirect injection, KB poisoning, over-broad retrieval
Full fine-tune All weights Deep domain shift, big budget Memorisation, backdoors, safety regression
LoRA / PEFT Small adapters on frozen base Style, format, narrow skill cheaply Same as above + untrusted adapters

Agents

Loop: perceive → plan → act → observe → repeat.

  • MCP = standard way to expose tools/data to models. A2A = agents discover and delegate to other agents.
  • Top risks: excessive agency, injection → action, confused deputy, memory poisoning, rogue tools/servers, agent impersonation, cascading failure, runaway loops, attribution gaps.
  • Core controls: least-privilege tool scopes, per-agent identity, approval for high-impact actions, iteration/spend caps, sandboxing, allow-listed tools, full tool-call logging.

Limits

  • Hallucination — confident falsehood → ground, cite, validate, review. Watch package hallucination.
  • Non-determinism — pin model versions, fix sampling, log everything, test repeatedly.
  • Uncertainty — aleatoric (data noise) vs epistemic (lack of knowledge); models are often poorly calibrated.
  • Drift — data (inputs), concept (input→output relationship), model (performance), behavioural (after updates). Can be adversarially induced. Major drift = reassessment trigger.
  • Model collapse — training on AI-generated data erodes the tails → track provenance, keep human data.

Metrics

  • Precision = TP/(TP+FP) — are alerts real? Recall = TP/(TP+FN) — did we catch them? F1 balances both.
  • Accuracy misleads on imbalanced data. Costly misses → favour recall; costly false alarms → favour precision.
  • GenAI: perplexity, BLEU/ROUGE, benchmarks (beware contamination, saturation), human / LLM-as-judge, groundedness, jailbreak resistance.

Oversight modes

Mode Human role Use for
HITL Approves before effect High-impact, irreversible
HOTL Monitors, can override High-volume, moderate impact
Automated None in real time Low-risk, reversible, well-tested

Guard against automation bias: measure override rates, sample-audit approvals.

Exam-day reflexes

  • "Instructions hidden in a retrieved document" → indirect prompt injection.
  • "Data must be erasable on request" → prefer RAG over fine-tuning.
  • "Agent can do more than the task needs" → excessive agency → least privilege.
  • "Model fine at launch, worse six months later" → drift → monitor and retrain/rollback.
  • "Wrong but confident answer, no attacker" → model limitation, not a vulnerability.

Visual summary

Trusted and untrusted text share one context window
Remember One context, no trust boundary — the root of prompt injection.
Choosing RAG, fine-tuning or prompting
Remember Knowledge → RAG · behaviour → fine-tune · instructions → prompting.