AI Threats, Testing and Lifecycle Management, 10 min read

Ready reckoner

Threat modeling

Method Use when
STRIDE Systematic per-component/flow coverage
PASTA Tie threats to business impact (7 stages)
MAESTRO Agentic/multi-agent; threats within and between 7 layers
  • Cover the whole pipeline; use libraries: MITRE ATLAS, OWASP LLM/agentic Top 10, NIST AI 100-2.
  • Actors by motivation, capability, access; include non-human actors (your own manipulated agent).
  • Classic families: evasion (inference, integrity) · poisoning/backdoor (training) · extraction (confidentiality) · inversion/membership inference (privacy).
  • OWASP LLM 2025: injection, sensitive disclosure, supply chain, poisoning, improper output handling, excessive agency, system prompt leakage, vector/embedding weaknesses, misinformation, unbounded consumption.
  • Credible robustness claim = stated threat model + adaptive attacks + results across attack strengths.

Secure AIDLC and change

  • Security activity and automated gate at plan, data, build, evaluate, deploy, operate, retire. NIST SP 800-218A (SSDF for GenAI).
  • Environments: research (no prod creds) · dev · training (isolated, no internet) · prod (pipeline-only).
  • Behavior changes with model, data, retrieval, prompts, guardrails, tools → all versioned in a release manifest.
  • Rollout: shadow (users see old) → canary (some users see new) → pre-agreed triggers → whole-bundle rollback.

Supply chain and provenance

  • AI supplier risks: training data, silent changes, data use, dependency, opaque safety tuning, sub-processors.
  • Assurance: certificate scope, SOC 2 period/exceptions/user entity controls, test scope, model card method.
  • Licensing applies along the pedigree (base model + data terms).
  • Hash = integrity · signature = publisher · attestation = how it was built (SLSA).
  • AIBOM = component present · VEX = exploitable? (not affected / affected / fixed / under investigation).

Red teaming

  • Scope: objectives, scenarios, in/out of scope, measurable success criteria, signed RoE (data handling, safety stops, escalation).
  • Sandboxes and mock tools for agents.
  • Report ASR together with false refusal rate, over many attempts and variants.
  • Human creativity + automation (PyRIT, garak) in break-fix cycles; map to ATLAS/OWASP.
  • AI testing AI: keep in scope, validate the judge, protect generated data.
  • Jailbreak root causes: competing objectives, mismatched generalization. Jailbreak = model breaks its own rules; injection = attacker's instructions replace the application's.

Findings

  • Triage: severity, exploitability, reproducibility, business impact, reach, detectability.
  • Classify: security vulnerability · model limitation · safety issue · fairness issue → right owner.
  • Map to ATLAS, OWASP, CWE.
  • Reproduction package: versions, settings, inputs (all turns), outputs, success rate over N, trace IDs; protected.
  • Audience: engineers (repro), leaders (decision, no payloads), legal (obligations), vendors (responsible disclosure).

Mitigations

  • Target the mechanism, not the symptom (blocklists chase phrasing).
  • Tune for missed attacks and false positives, by risk tier.
  • Prioritize by risk reduction, feasibility, business context; interim controls; recorded, time-limited acceptance.
  • Regression suite (variants + utility, run N times) as a release gate.
  • Residual risk: KRIs, scheduled tests, canaries, reassessment triggers.
  • Empirical robustness (practical, adaptive attacks) vs certified (provable, narrow bounds). Guardrail models are one layer.