Neural network architectures and AI compute infrastructure
In this lesson you will:
- Explain how feed-forward networks, CNNs, RNNs/LSTMs and transformers process data
- Describe self-attention and why it enabled large language models
- Identify GPU, TPU and other accelerator roles in training and inference
- Recognise the security implications of AI hardware, memory and shared infrastructure
Architectures determine what data a model can handle and how it can be attacked. Hardware determines where the model runs, who shares that environment, and what an attacker might extract from it. This lesson covers both.
1. The basic building block
An artificial neuron multiplies each input by a weight, adds a bias, and passes the sum through an activation function (such as ReLU or sigmoid) that introduces non-linearity. Neurons are arranged in layers:
- Input layer — receives features (pixel values, token embeddings).
- Hidden layers — learn intermediate representations.
- Output layer — produces the prediction (a class probability, a next-token distribution).
A feed-forward network (multilayer perceptron) passes data in one direction only. It works well for fixed-size tabular input but has no notion of spatial layout or sequence order.
Training adjusts weights to reduce a loss function. Inference is running the trained model on new input. Training is expensive and occasional; inference is cheaper per call but happens constantly — and inference endpoints are what attackers usually reach.
2. Convolutional neural networks (CNNs)
CNNs are designed for grid-like data — images, video frames, spectrograms.
- Convolutional layers slide small filters across the input to detect local patterns (edges, textures).
- Pooling layers downsample, giving some tolerance to small shifts.
- Deeper layers combine local patterns into objects.
Where used: image classification, face recognition, document and CAPTCHA processing, malware visualisation (binaries rendered as images).
Security lens: CNNs are the classic target of adversarial examples — imperceptible pixel changes that cause confident misclassification — and of physical-world attacks such as patches or stickers that defeat a camera-based detector.
3. Recurrent networks (RNNs, LSTMs, GRUs)
RNNs process sequences one step at a time, carrying a hidden state forward like short-term memory. Plain RNNs struggle with long sequences because gradients vanish; LSTM (long short-term memory) and GRU cells add gates that decide what to keep or forget.
Where used: time-series forecasting, log-sequence anomaly detection, older speech and translation systems.
Limitations: strictly sequential processing is slow to train and still struggles with very long-range dependencies — the gap transformers filled.
4. Transformers
The transformer architecture (introduced in 2017) replaced recurrence with self-attention: every position in the input can directly weigh its relationship to every other position, in parallel.
Core ideas:
- Tokens and embeddings — input text is split into tokens, each mapped to a vector.
- Positional encoding — because attention itself is order-agnostic, position information is added.
- Multi-head self-attention — several attention "heads" learn different relationships (syntax, coreference, topic).
- Stacked layers — dozens to over a hundred layers deep in large models.
Variants:
| Variant | Typical use |
|---|---|
| Encoder-only | Understanding tasks: classification, search embeddings |
| Decoder-only | Generation: most chat and code models |
| Encoder–decoder | Translation, summarisation |
| Vision / multimodal transformers | Images, audio and text in a shared model |
Security lens: attention treats everything in the context window as material to reason over. The model has no built-in way to tell trusted instructions from untrusted data that happens to be in the same context. That architectural fact is the root of prompt injection.
5. Other architectures worth recognising
- Diffusion models — generate images/audio by learning to reverse a noise process. Relevant to deepfakes and synthetic media.
- Generative adversarial networks (GANs) — a generator and discriminator compete; historically used for face synthesis.
- Graph neural networks — reason over relationships; used in fraud rings and network analysis.
- Mixture-of-experts (MoE) — routes each input to a subset of specialised sub-networks, cutting compute per call.
6. AI hardware infrastructure
| Component | Role | Notes |
|---|---|---|
| GPU (graphics processing unit) | Massively parallel matrix maths for training and inference | Dominant general-purpose accelerator |
| TPU (tensor processing unit) | Application-specific chip optimised for tensor operations | Offered as a cloud service |
| NPU / other ASICs | On-device or specialised inference | Phones, laptops, edge devices |
| High-bandwidth memory (HBM) / VRAM | Holds weights, activations and the attention cache during inference | Memory capacity often limits model size |
| High-speed interconnects | Link accelerators for distributed training | Large clusters behave like one machine |
| Storage | Datasets, checkpoints, model artefacts | Very high density and volume |
Why hardware matters for security
- Shared, multi-tenant accelerators. Cloud GPUs may be shared or reused. Risks include residual data in GPU memory between tenants and side-channel leakage. Controls: dedicated or isolated capacity for sensitive workloads, memory scrubbing, and vendor attestations.
- Confidential computing. Some modern accelerators and CPUs support trusted execution environments (TEEs) that keep data and model weights encrypted while in use, with remote attestation.
- Model artefacts on disk. Checkpoints contain the full model. Some serialisation formats can execute code on load — treat model files as executable content.
- Availability and cost. Accelerators are scarce and expensive. Resource-exhaustion attacks against inference (very long prompts, recursive agent loops) translate directly into outage and cost — sometimes called denial of wallet.
- Physical and environmental security. High-density AI racks draw large amounts of power and need advanced cooling; data-centre controls (power redundancy, cooling, fire suppression, rack access) become availability controls for AI.
- Sanitisation. High-density storage and accelerator memory complicate secure disposal; cryptographic erasure is often the practical method.
Worked example — a KYC document reader on shared GPUs
A fintech uses a CNN to read identity documents uploaded during customer onboarding. A design review finds the following:
| Layer | Finding | Fix |
|---|---|---|
| Model artefact | Checkpoint downloaded from a public model hub in a pickle-based format | Convert to a safe tensor format, verify the publisher and hash, pin the version |
| Accelerators | Inference runs on shared multi-tenant cloud GPUs while processing identity images | Use dedicated or confidential-computing GPU instances; scrub memory between jobs |
| Inference endpoint | No per-client limits; very large images accepted | Per-client quotas, input size limits, request timeouts |
| Model behaviour | Tampered or adversarial images could pass as genuine | Add document-tamper and liveness checks; send low-confidence results to a human |
Takeaway: none of these findings concern the model's accuracy. Each sits in the infrastructure around the model — which is where most practical AI security work happens.
🔐 Security lens
Architecture tells you how a model can be fooled (pixels for CNNs, context for transformers). Infrastructure tells you where the model and its data are exposed (shared GPU memory, checkpoint files, inference endpoints). Assess both.
Key takeaways
- CNNs suit images and are classic adversarial-example targets; RNNs/LSTMs suit sequences; transformers dominate language and multimodal AI.
- Self-attention gives transformers their power — and means instructions and data share one channel, enabling prompt injection.
- Training is expensive and periodic; inference is constant and is the usual attack surface.
- GPUs, TPUs and other accelerators bring multi-tenancy, memory-residue, supply and availability risks.
- Treat model checkpoint files as sensitive and potentially executable.
Self-check
- Why does self-attention make prompt injection hard to solve architecturally?
- Name two controls for running sensitive inference on shared cloud GPUs.
- How can a single long-running agent loop become a financial risk?