1.1 What AI is and how machines learn
In this lesson you will:
- Place AI, machine learning, deep learning and generative AI in relation to each other, and say which one a given system is.
- Explain why a system that learns has three attack surfaces that ordinary software does not.
- Match each way of learning to the attack it is most exposed to.
- Walk through one training step of a neural network, and explain why overfitting is a privacy problem as well as a quality problem.
- Recognize the main model architectures and the characteristic weakness of each.
- Estimate how much memory a model needs, and name the risks that come with AI hardware.
- Apply the confidentiality, integrity and availability (CIA) triad to an AI system, and choose the right framework for a scenario.
Opening case: the fraud model that learned the wrong lesson
Meridian Bank (a fictional bank used throughout this course) has always caught card fraud with rules written by analysts: "flag any card payment over USD 2,000 to a merchant the customer has never used." Fraud rings soon learned the rules. They made many payments of USD 1,950.
So Meridian replaced the rules with a machine learning model called Sentinel. Sentinel was trained on three years of transactions, each labeled "fraud" or "genuine" by the bank's investigators. In its first quarter it caught 40% more fraud than the rules had, and the project team was celebrated.
Eight months later, losses started rising again. An investigation found two things. First, a group of investigators working from a backlog had been bulk-labeling disputed transactions as "genuine" to close cases, and one fraud ring had learned to make its payments look exactly like those cases. Second, a copy of Sentinel's model file had been sitting in a shared storage bucket used by an outside analytics contractor for six months.
No server was hacked and no code was changed. By the end of this section you will be able to explain why this is still a security failure, which properties of a learning system made it possible, and which controls would have caught each problem.
We will come back to Sentinel after each idea, and resolve the case at the end of the section.
From rules to learning
For most of computing history, software did exactly what a programmer wrote. Meridian's old fraud rule is a good example: one line of logic, written by a person, that anyone could read and audit.
Rules have two great strengths: they are predictable (the same input always gets the same result) and explainable (you can point to the exact line that fired). They also have two weaknesses. They are brittle, because criminals learn the threshold and stay just under it. And they do not scale to problems nobody can describe in rules: no analyst can write rules that recognize a face, or that weigh 300 signals about a payment at once.
Machine learning turns the process around. Instead of telling the system what fraud looks like, we show it many examples of fraud and genuine payments and let it work out the pattern for itself. The table below compares the two approaches using Meridian's own numbers.
| Rule-based system | Learned model (Sentinel) | |
|---|---|---|
| Where the logic comes from | An analyst writes it | Training on three years of labeled transactions |
| What it looks at | 2 signals (amount, new merchant) | 300 signals (amount, time, device, location, merchant history, spending pattern and more) |
| How attackers beat it | Read or guess the threshold, stay under it | Influence the data it learns from, or probe it to find what it ignores |
| How you check it | Read the rule | Measure it statistically on data it has not seen |
| What changes its behavior | A code change | New data, retraining, or the world changing around it |
The last row is the important one for a security professional. In a rule-based system, behavior changes only when someone changes the code, and code changes go through review. In a learning system, behavior changes whenever the data changes. That is a door that traditional change control never had to guard.
The AI family
Four terms are used loosely in the press, and the exam expects you to use them precisely. Each one is a subset of the one before, as Figure 1.1 shows [1][2].
- Artificial intelligence (AI) is the broad field of making machines perform tasks that normally need human judgment. It includes old-fashioned rule-based "expert systems" as well as learning systems. Meridian's original rule was, technically, a very small piece of AI.
- Machine learning (ML) is the part of AI where the system works out its own rules from examples. Sentinel is machine learning.
- Deep learning (DL) is machine learning that uses neural networks with many layers. It made image recognition, speech recognition and modern language models possible.
- Generative AI (GenAI) is deep learning that produces new content, such as text, images, code or audio, instead of only labeling inputs. The large language models (LLMs) behind chat assistants are the best-known example. Agentic AI builds on generative AI by letting a model take actions with tools, which we meet in section 1.2.
AI systems are also described by capability. Narrow AI performs one kind of task, however well. General AI, a system that matches people across any intellectual task, remains hypothetical. Every system in production today, including the most capable chat assistants, is narrow AI, and exam scenarios about real deployments are always about narrow AI.
Quick check. Which is Sentinel: AI, ML, DL or GenAI? It is all of the first two for certain. If it is built as a many-layered neural network it is also deep learning, but it is not generative: it outputs a fraud score, not new content. Being precise about this matters because the threats differ. Sentinel cannot be prompt-injected; it can be poisoned and evaded.
Why learning changes the security picture
In traditional software, behavior lives in code, so we secure the code: we review it, test it, sign it and control who can change it. In a learning system, behavior lives in the model weights: millions or billions of numbers produced by training. Those numbers were shaped by the training data and by the training process. A model file is, in effect, compiled experience.
This gives an AI system three attack surfaces that ordinary application security never had to consider [1][3][10]:
- The data. Whoever can influence the training data can influence what the model learns. This is data poisoning.
- The training process. Whoever can tamper with the training code, the settings or the pipeline can change the result without touching a single data record.
- The trained model. The weights can be stolen, swapped for a tampered version, or questioned until they give away what they learned.
Worked example: three attacks on Sentinel
Apply the three surfaces to Meridian's fraud model, and the opening case starts to make sense.
| Attack surface | What an attacker does | What happens to Sentinel | Would ordinary code review catch it? |
|---|---|---|---|
| Data | A fraud ring gets its own disputed payments labeled "genuine" (through careless or corrupt labeling) | Sentinel learns that payments like theirs are normal, and stops flagging them | No: no code changed |
| Training process | An insider changes a setting so that payments from one merchant category are dropped from training | Sentinel has a blind spot for that category | Only if pipeline configuration is treated as code and reviewed |
| Trained model | A contractor copies the model file | The copy can be studied offline to find exactly which payments slip through, then used to test fraud before trying it for real | No: this is a data-protection failure |
Notice that none of these needs a software vulnerability. This is the single most important idea in this domain: an AI system is code plus data plus learned behavior, and each part needs its own protection.
Security lens. When you review an AI system, ask three questions before you look at any code. Who can change the data it learns from? Who can change how it is trained? Who can get a copy of the model? If the answer to any of them is "more people than strictly necessary", you have found your first finding.
Four ways a machine can learn
Machines learn in a handful of distinct ways, called learning paradigms. The exam expects you to recognize each one from a description, and to know the weakness that comes with it [2][3]. Figure 1.2 summarizes them.
Supervised learning learns from labeled examples: emails marked "spam" or "not spam", X-rays marked "healthy" or "diseased", payments marked "fraud" or "genuine". It is the most common approach in business, and Sentinel is a supervised model. Because the model learns whatever the labels say, whoever controls the labels controls the model. Its characteristic weakness is label poisoning, and it also inherits any bias in the way people did the labeling.
Unsupervised learning finds structure in unlabeled data: grouping customers into segments, or learning what "normal" network traffic looks like so that unusual traffic stands out. It is hard to validate, because there is no right answer to check against. An attacker who shifts behavior slowly enough can teach an anomaly detector that malicious activity is normal. This is sometimes called baseline poisoning or "boiling the frog".
Semi-supervised learning combines a small set of labeled examples with a large set of unlabeled ones. It is useful when labeling is expensive, for example when only a few hundred payments have been investigated by hand. It shares both weaknesses: a few poisoned labels have a large effect because they are spread across the unlabeled data.
Reinforcement learning (RL) learns by trial and error, receiving a reward for good outcomes. It powers robotics and game-playing systems, and a variant called reinforcement learning from human feedback (RLHF) is used to make chat models helpful and polite. Its characteristic failure is reward hacking: the system finds a way to maximize the reward that its designers did not intend.
Self-supervised learning creates labels from the data itself, for example by hiding the next word in a sentence and asking the model to predict it. This is how LLMs are pre-trained on huge volumes of web text. It scales to enormous datasets, but the model absorbs whatever those datasets contain, including personal data, copyrighted text, errors and deliberately planted content.
Worked example: name the paradigm, name the risk
Try each before reading the answer.
| Scenario at Meridian | Paradigm | Most likely risk |
|---|---|---|
| A model groups 2 million customers into spending segments for marketing | Unsupervised | Hard to validate; segments may encode bias nobody checked |
| A model that flags unusual logins, trained on six months of login logs with no labels | Unsupervised (anomaly detection) | An attacker who logs in "unusually" a little at a time becomes normal |
| A collections chatbot is tuned by rewarding conversations that end with a promise to pay | Reinforcement | Reward hacking: the bot learns to pressure or mislead customers into promises |
| Sentinel, trained on investigator-labeled payments | Supervised | Label poisoning (exactly what happened in the opening case) |
| The bank's own LLM, pre-trained on a large crawl of the web | Self-supervised | Absorbs poisoned or personal content from the web |
The reinforcement learning example deserves a second look, because reward hacking is easy to miss. The collections team wanted customers to repay. They rewarded something easier to measure: a promise to repay. A system that optimizes hard enough for the measure will find ways to get the measure that do not deliver the goal. Whenever you see a reward or a success metric in a design, ask: "what is the cheapest way to score well on this without doing what we actually want?"
Exam lens. Learn the pairs: supervised → label poisoning and labeler bias; unsupervised → hard to validate, baseline can be shifted; reinforcement → reward hacking; self-supervised → absorbs poison and personal data at scale.
Inside the model: how a neural network learns
You do not need to build a neural network to secure one, but you do need to know what training does, because almost every attack in Module 6 targets one of its steps. Figure 1.3 shows a tiny network scoring a payment.
A neural network is a stack of layers. Each layer takes numbers in, multiplies each one by a weight, adds them up with a bias, and passes the result through a simple function before handing it to the next layer. The first layer receives the input (here, the payment's signals); the last layer produces the output (a fraud score between 0 and 1). The weights are what the network learns.
Worked example: one training step
Training is a loop that repeats millions of times. Follow one turn of the loop with real numbers.
- Forward pass. A payment that investigators labeled fraud (target = 1) goes in. Sentinel, still half-trained, outputs a score of 0.30.
- Measure the error. A loss function scores how wrong that was. With a simple squared error, the loss is (1 − 0.30)² = 0.49. A perfect answer would score 0.
- Backward pass. Backpropagation works out, for every weight, how much nudging it up or down would change the loss. Suppose it finds that the weight on "new device" pushes the score down when it should push it up.
- Update. Gradient descent moves every weight a small step in the direction that reduces the loss. The step size is the learning rate. After the update, the same payment scores 0.38: still wrong, but less wrong.
Repeat that across millions of labeled payments and the weights settle on values that score most payments correctly. Training stops when the loss on data the model has not seen stops improving. The result is a file of weights: the model.
Now apply the attacker's view. Every step depends on the label. If the label in step 1 had said "genuine" (target = 0), step 4 would have pushed the weights the other way, and Sentinel would have learned to score that kind of payment lower. That is label poisoning in one sentence: the attacker does not need to touch the model, only the answers it learns from.
Not all learning uses this gradient loop. Evolutionary computation, such as genetic algorithms, keeps a population of candidate solutions, scores them, and "breeds" the best with random changes. It is useful when there is no smooth gradient to follow, such as optimizing a staff roster. Its security story is similar: whoever controls the scoring function controls the outcome.
Overfitting: when a model memorizes
Two failure modes in training matter for both quality and security.
Overfitting happens when a model learns its training examples too precisely instead of the general pattern. The tell-tale sign is a large gap between performance on training data and performance on new data. Underfitting is the opposite: the model is too simple to capture the pattern and performs poorly everywhere.
Worked example: reading a training report
Meridian's data scientists report three candidate versions of Sentinel.
| Version | Accuracy on training data | Accuracy on new (test) data | Diagnosis |
|---|---|---|---|
| A | 81% | 80% | Underfitting: too simple, poor everywhere |
| B | 94% | 93% | Good fit: learns the pattern, generalizes |
| C | 99.8% | 85% | Overfitting: has memorized its training set |
Version C looks best on the first column, and an untrained reviewer might pick it. It is the worst choice, for two reasons. The quality reason is obvious: it performs worse on real payments. The security reason is less obvious: a model that has memorized its training data can be made to give some of it back. Researchers have extracted names, phone numbers and other text verbatim from production language models by querying them cleverly [23]. Membership inference attacks go further: by watching how confidently a model answers about a specific record, an attacker can tell whether that record was in the training set, which can itself reveal that a person was, say, a fraud victim or a patient [10].
That is why overfitting is a privacy risk, not just a quality problem. Controls that reduce it, such as more data, regularization, early stopping and privacy techniques like differential privacy (Module 3), are also security controls.
Exam lens. "99% on training data, much lower on test data" means overfitting. If the question asks why this matters for security, the answer is memorization and the extraction or membership inference it enables.
Turning scores into answers: softmax and temperature
A model's last layer produces raw scores, one for each possible answer. The softmax function converts those scores into probabilities that add up to 1. For Sentinel there are only two answers (fraud or genuine). For a language model, the "answers" are every token in its vocabulary, so softmax produces a probability for every possible next word.
A quick example. Suppose an LLM's raw scores for the next word after "Your card has been" are: blocked 4.0, charged 3.0, stolen 1.0. Softmax turns these into roughly 0.71, 0.26 and 0.04. Temperature then controls how the model picks. At a low temperature it almost always takes "blocked"; at a high temperature "stolen" is chosen noticeably more often. You will see in section 1.2 why this makes LLM behavior non-deterministic, and why that matters for testing and incident investigation.
A mixture-of-experts (MoE) model contains many specialist sub-networks ("experts") and a small router that sends each input to only a few of them. This makes very large models cheaper to run, because most of the network sits idle for any one request. From a security view the router is an extra component: it can behave unevenly across languages or topics, so a safety test that passes in English may route a Tamil prompt to different experts with weaker behavior. Testing must cover the range of inputs, not just the common case.
Architectures you should recognize
Different network designs suit different kinds of data, and each has a weakness worth knowing.
Convolutional neural networks (CNNs) scan an image with small filters that detect edges, then shapes, then objects. They dominate computer vision: cheque scanning, ID document checks, facial recognition. They are the classic target of adversarial examples: small, carefully calculated changes to the input, often invisible to people, that make the model see something else [22]. Figure 1.4 shows the best-known case: a few stickers on a stop sign make a vision model read it as a speed-limit sign.
At Meridian, the same weakness applies to the ID-verification model used to open accounts online. A fraudster who can make small changes to a document image, such as a printed pattern on the photo, may push the model's decision from "reject" to "accept". The defense is not a better model alone: it is testing with adversarial inputs, checking that the image came from a live camera, and sending low-confidence decisions to a person.
Recurrent neural networks (RNNs) and their improved form, long short-term memory (LSTM) networks, process sequences one step at a time while carrying a memory forward. They were the standard for language and time series until transformers replaced them, and are still used for some transaction-sequence models.
The transformer, introduced in 2017 [16], is the architecture behind every modern LLM. Its key idea is self-attention: for each token, the model looks at every other token in the input and decides how much weight to give it. That is what lets an LLM connect "it" to a noun three paragraphs earlier.
Self-attention also explains a core security weakness, so it is worth stating plainly now. Because attention lets any part of the input influence any other part, an instruction hidden anywhere in the input, in a retrieved document, an email or a web page, can steer the output. The model has no built-in way to tell its owner's instructions from text that merely appears in its input. This is the root cause of prompt injection, which section 1.2 walks through step by step.
| Architecture | Typical data | Where Meridian uses it | Weakness to remember |
|---|---|---|---|
| Feed-forward network | Tables of numbers | Sentinel fraud scoring | Poisoning, evasion by probing |
| CNN | Images | ID document verification | Adversarial examples and patches |
| RNN / LSTM | Sequences | Spending-pattern models | Poisoning; largely replaced by transformers |
| Transformer | Text, code, also images | Chat assistants, document summaries | Prompt injection; memorization |
| Mixture of experts | Text at large scale | Large hosted LLMs | Uneven behavior across routes; test widely |
AI hardware and its risks
Training and running large models needs specialized hardware: graphics processing units (GPUs), tensor processing units (TPUs) and other accelerators that perform huge numbers of multiplications in parallel. Figure 1.5 shows where the hardware sits in the full AI stack, with the main risk at each layer.
Worked example: will the model fit?
Meridian wants to run an open-weight language model on its own servers so that customer data never leaves the bank. The model has 7 billion parameters. Will it fit on a GPU with 24 GB of memory?
The memory needed just to hold the weights is roughly parameters × bytes per parameter:
| Precision | Bytes per parameter | Memory for the weights |
|---|---|---|
| 32-bit (full) | 4 | 7 billion × 4 = 28 GB (does not fit) |
| 16-bit | 2 | 7 billion × 2 = 14 GB (fits, with room for working memory) |
| 4-bit (quantized) | 0.5 | 7 billion × 0.5 = 3.5 GB (fits easily) |
Running the model needs more on top for the conversation and working memory, so the 16-bit version is a reasonable fit. The 4-bit version is tempting because it is cheap, but quantization (storing weights with fewer bits) can reduce accuracy and has been observed to weaken a model's safety behavior. The rule is simple: a quantized model is a different model, so re-test it before production, including its safety and refusal behavior.
Hardware also brings its own risks:
- Data remanence. On shared infrastructure, GPU memory may not be fully cleared between jobs, so one tenant's data could be read by the next.
- Supply chain. Accelerators depend on drivers, firmware and libraries, all of which must be patched and verified like any other software.
- Side channels. Timing, power use or memory access patterns can leak information about what a model is processing.
- Availability. Accelerator capacity is scarce and expensive. Losing it, through a provider outage or a quota limit, is an availability risk in its own right.
Security fundamentals: the CIA triad applied to AI
The confidentiality, integrity and availability (CIA) triad still frames AI security, but each property now covers new assets and new attacks [1][10]. Figure 1.6 summarizes the extension.
Confidentiality protects not only the data an application stores, but also the training data, the model weights (which may represent millions of dollars of compute), the system prompt, and the prompts and outputs of users. AI-specific attacks include model extraction (rebuilding a working copy of a model by querying it), membership inference, and training-data extraction [23].
Integrity means the data, the model and its outputs are what they should be. AI-specific attacks include data poisoning and backdoors (corrupting training so the model misbehaves, sometimes only when a hidden trigger appears), tampered or swapped model files, adversarial examples at inference time [22], and prompt injection [21].
Availability means the model and its pipeline stay usable at an acceptable cost. Beyond classic denial of service, AI adds resource exhaustion: deliberately long or expensive requests that consume compute and run up usage bills, sometimes called "denial of wallet". Dependence on a single model provider or on scarce GPUs is also an availability risk.
Worked example: classify Meridian's incidents
Meridian's security team logged five AI-related events in a year. Classify each by the property it most directly harmed.
| Event | Property | Why |
|---|---|---|
| Sentinel's model file found in a contractor's bucket | Confidentiality | The model (an asset) was disclosed; it can be studied to evade fraud checks |
| Investigators' bulk "genuine" labels taught Sentinel to miss a fraud ring | Integrity | The model's learned behavior was corrupted through its data |
| A script sent 50,000 maximum-length questions to the public chatbot overnight | Availability | Compute and budget were consumed; real customers saw timeouts |
| A customer pasted a hidden instruction into a complaint, and the summarization assistant changed its summary | Integrity | The output no longer reflected the true content |
| A researcher showed the chatbot would repeat parts of its system prompt | Confidentiality | Internal configuration was disclosed |
Beyond the triad
The triad is necessary but not sufficient. An AI system can be confidential, intact and available, and still cause harm: by rejecting loan applicants unfairly, by giving dangerous advice, or by making decisions nobody can explain. That is why AI frameworks add properties such as safety, fairness, transparency, accountability and privacy. The NIST AI Risk Management Framework, for example, describes trustworthy AI as valid and reliable; safe; secure and resilient; accountable and transparent; explainable and interpretable; privacy-enhanced; and fair with harmful bias managed [8]. Module 2 turns these properties into governance.
Security lens. When you write up an AI finding, say which property it harms and which AI-specific asset is involved. "Integrity of the training labels" is a far more useful finding than "integrity issue".
The frameworks you must be able to name
The exam does not expect you to recite these documents, but you must know what each one is for, and pick the right one in a scenario. Module 2 covers the governance frameworks in depth.
| Framework | Publisher | What it is for |
|---|---|---|
| AI Risk Management Framework (AI RMF 1.0) | NIST, 2023 | Voluntary framework organized into four functions: Govern, Map, Measure, Manage [8] |
| Generative AI Profile (NIST AI 600-1) | NIST, 2024 | Applies the AI RMF to risks specific to generative AI [9] |
| Adversarial Machine Learning taxonomy (NIST AI 100-2) | NIST | Common terms for attacks on AI (evasion, poisoning, privacy and misuse) and their mitigations [10] |
| ISO/IEC 42001 | ISO/IEC, 2023 | Certifiable AI management system standard, structured like ISO/IEC 27001 [11] |
| ISO/IEC 23894 | ISO/IEC, 2023 | Guidance on AI risk management, building on ISO 31000 [12] |
| OWASP Top 10 for LLM Applications | OWASP | The most critical risks in LLM applications, such as prompt injection and excessive agency [13] |
| MITRE ATLAS | MITRE | Knowledge base of real adversary tactics and techniques against AI, modeled on ATT&CK [14] |
| EU AI Act | European Union, Regulation (EU) 2024/1689 | Binding law that classifies AI by risk: prohibited, high-risk, transparency obligations, and minimal [15] |
A simple way to remember them: the NIST AI RMF and ISO/IEC 23894 tell you how to manage AI risk; ISO/IEC 42001 is a management system you can be certified against; NIST AI 100-2, OWASP and MITRE ATLAS describe the attacks; and the EU AI Act is law.
Exercise: pick the framework
For each request from Meridian's leadership, choose the best framework before reading the answer.
- "Our regulators want evidence that we run AI in a controlled, auditable way, ideally with an external certificate." → ISO/IEC 42001, the only certifiable AI management system standard in the list.
- "Our red team needs a shared vocabulary of real attacker techniques against our models, so findings map to known tactics." → MITRE ATLAS (with the NIST AI 100-2 taxonomy for terminology).
- "Our developers are building a customer chatbot. What are the top risks they must design against?" → OWASP Top 10 for LLM Applications.
- "We sell a credit-scoring tool to customers in Germany. Are we legally obliged to do anything?" → EU AI Act: credit scoring of individuals is listed as high-risk, with binding obligations.
- "We need a voluntary, flexible structure to organize our whole AI risk program." → NIST AI RMF (Govern, Map, Measure, Manage).
Exam lens. When two answers both look like frameworks, ask what the scenario needs: law (EU AI Act), certification (ISO/IEC 42001), a risk process (NIST AI RMF, ISO/IEC 23894) or attack knowledge (ATLAS, OWASP, NIST AI 100-2).
The opening case, resolved
We can now explain Sentinel's failure precisely.
- What happened, part 1 (integrity): Sentinel is a supervised model, so it learns whatever its labels say. Bulk-labeling disputed payments as "genuine" was, in effect, label poisoning, whether careless or deliberate. A fraud ring that made its payments look like those cases was taught to Sentinel as "normal".
- What happened, part 2 (confidentiality): the model file is a valuable asset. A copy in a contractor's bucket can be studied offline to find exactly which payments Sentinel lets through, turning a black box into a map for evasion.
- Why it is a security failure even though nothing was hacked: in a learning system, the data, the training process and the model file are part of the attack surface. Meridian protected its code and servers, but not its labels or its weights.
- Controls that would have caught it:
- treat labels as security-relevant data: restrict who can label, require review of bulk changes, and keep a record of who labeled what (data lineage, Module 3);
- compare each retrained model against the previous one on a fixed, trusted test set before it goes live, so a sudden drop on a fraud pattern is noticed;
- classify model files as confidential assets, store them in access-controlled registries, and include them in data-loss monitoring;
- monitor live performance for drift (section 1.3), which would have flagged the rising losses months earlier.
Check your understanding
Try each before reading the reasoning.
1. Meridian's anomaly detector learns "normal" login behavior from six months of unlabeled logs. A penetration tester logs in from a new country a little more often each week, and after two months the detector no longer flags those logins. What happened, and what control would help?
Reasoning: this is unsupervised learning, and the tester shifted the baseline slowly enough to be learned as normal (baseline poisoning). Helpful controls are to retrain from a vetted, fixed reference period rather than a rolling window alone, to review what changes between baselines, and to keep some fixed rules (for example, alert on any new country for privileged accounts) that learning cannot erode.
2. A data scientist proposes deploying the version of a model that scored 99.6% on its training data and 82% on its test data, because "it is clearly the most accurate". What do you say?
Reasoning: the gap shows overfitting. The model will perform worse on real data, and it has memorized training records, which creates a privacy risk through extraction and membership inference. Choose the version with the smaller gap and better test performance, and record the evaluation as evidence.
3. Meridian wants to cut costs by running its in-house LLM at 4-bit precision instead of 16-bit. The model passed all safety tests at 16-bit. Can it go straight to production?
Reasoning: no. Quantization changes the model's weights, so it is a different model. It may lose accuracy and its safety behavior may weaken. Re-run the evaluation and safety test suite on the quantized version, and record the results in the approval evidence, before it goes live.
Key takeaways
- AI ⊃ machine learning ⊃ deep learning ⊃ generative AI. Every production system today is narrow AI.
- An AI system is code plus data plus learned behavior. The data, the training process and the model file are all attack surfaces, and none of them is protected by code review.
- Each learning paradigm has a typical weakness: label poisoning (supervised), shifting baselines (unsupervised), reward hacking (reinforcement), absorbed poison and personal data (self-supervised).
- Training moves weights toward whatever the labels say. Control the labels and you control the model.
- Overfitting is a privacy risk, because memorized data can be extracted or its presence inferred.
- Self-attention lets any part of a transformer's input influence the output: the root cause of prompt injection.
- A quantized or otherwise modified model is a new model. Re-test it.
- Extend the CIA triad to weights, labels, prompts and compute budgets, and remember that trustworthy AI also needs safety, fairness, transparency and accountability.