2.7 AI security education and awareness
In this lesson you will:
- Explain why AI literacy is a security control, and when it is a legal duty.
- Run a skills assessment and prioritize the gaps by risk.
- Design training that fits each audience, from the board to the reviewers who oversee AI decisions.
- Measure whether training works, using the Kirkpatrick levels and leading and lagging indicators.
- Recognize a risk-aware AI culture, and the measurements that fake one.
Opening case: 100% completion
After the TalentMatch audit, Meridian launches a mandatory 20-minute e-learning module called "Responsible AI at Meridian" for all 18,000 staff. Six weeks later, the dashboard shows 100% completion and an average satisfaction score of 4.6 out of 5. The program is reported to the board as a success.
In the same quarter:
- DLP alerts for customer data pasted into public chatbots fall by only 3%;
- a developer builds an internal agent with write access to the HR system and no approval step;
- underwriters reviewing the AI mortgage tool still override it in 0.3% of cases;
- a finance clerk pays a USD 240,000 invoice after a video call with what looked and sounded like the CFO, but was a deepfake.
By the end of this section you will be able to explain why 100% completion told the board almost nothing, and design a program whose success would show up in exactly these numbers.
Literacy as a control and a legal duty
People are part of every AI control. A reviewer who does not understand automation bias rubber-stamps decisions. A developer who does not understand prompt injection builds an exploitable agent. An employee who does not know the acceptable use policy pastes client data into a public chatbot. A clerk who does not know what a real-time deepfake can do trusts a video call. AI literacy, the skills and understanding needed to use AI appropriately and to recognize its risks, is therefore a security control.
It is also a legal requirement. Article 4 of the EU AI Act requires providers and deployers to take measures to ensure a sufficient level of AI literacy among staff dealing with AI systems, taking account of their technical knowledge, experience, education and training, the context of use, and the people affected. It has applied since 2 February 2025 [14]. ISO/IEC 42001 similarly requires the organization to determine and ensure the competence of people whose work affects AI performance [12].
Notice the wording: "taking account of ... the context". A single generic module for everyone is hard to defend as meeting that duty for the people who build, approve and oversee high-risk AI.
Skills assessment and gap analysis
A program starts by finding out where people are. A skills assessment maps each role to the AI knowledge it needs, then measures current capability using self-assessment, short tests, manager input, and evidence from incidents and audits. The gap analysis compares the two.
Not every gap matters equally. Prioritize gaps by risk: the people who approve high-risk uses, build agents with tool access, or oversee automated decisions about customers come first.
Worked example: Meridian's priority gaps
| Role group | Needed | Found | Risk if not closed | Priority |
|---|---|---|---|---|
| Developers building agents (about 120) | Prompt injection, excessive agency, secure tool design | 15% could explain indirect injection | An exploitable agent with write access (already happened) | 1 |
| Mortgage underwriters reviewing AI output (about 300) | System limits, automation bias, when and how to override | Most believed the tool "is more accurate than us" | Unlawful or unfair decisions approved without scrutiny | 1 |
| Finance and treasury staff (about 400) | Deepfake and AI-enabled fraud, call-back verification | Few knew real-time video deepfakes exist | Direct financial loss (already happened) | 1 |
| AI governance committee (12) | EU AI Act classification, risk appetite, questions to ask | Uneven | Wrong approvals | 2 |
| All staff | Acceptable use, what data never goes into AI tools | Policy known by name, not content | Data leakage | 2 |
Role-based training
One generic module for everyone is the weakest option. Effective programs tailor content to each audience [2][5]. Figure 2.34 shows a training matrix: which topics are core and which are awareness-level for each group.
| Audience | Focus |
|---|---|
| Board and executives | AI risk appetite, regulatory exposure, accountability, the questions to ask management |
| AI builders (developers, ML engineers) | Secure AI development, prompt injection and agent risks, data handling, testing, the AI standards |
| Data teams | Data governance, privacy, provenance, bias in data, labeling quality |
| Security operations | AI threats and telemetry, detecting AI misuse, AI incident playbooks |
| Legal, privacy and compliance | AI Act classification, DPIAs and FRIAs, IP and contract issues |
| Human overseers and reviewers | System limitations, automation bias, when and how to override and escalate |
| All staff | Acceptable use, what data never goes into AI tools, AI-enabled phishing and deepfakes, how to report concerns |
Ethics should be woven into each track, not taught as a separate module. Builders should practice spotting proxy variables; reviewers should discuss cases where the AI's answer was legal but unfair; finance staff should rehearse saying "I'll call you back on the number we have on file" to a convincing executive on video.
Formats matter as much as content. Short scenario-based modules, hands-on labs (developers attacking a deliberately vulnerable agent), simulations (AI-generated phishing and deepfake voice calls), and just-in-time prompts (a DLP coaching message the moment someone pastes customer data into a chatbot) change behavior far more than a single annual video.
Measuring whether training works
Completion rates measure attendance, not effect. The four levels of the Kirkpatrick model give a better structure [22], shown in Figure 2.35.
- Reaction: did participants find it relevant and engaging? (satisfaction surveys)
- Learning: did their knowledge or skills improve? (pre- and post-tests, lab results)
- Behavior: do they act differently at work? (fewer policy violations, better reviews, secure patterns in code)
- Results: did organizational outcomes improve? (fewer AI incidents, faster detection, clean audit findings)
Meridian's dashboard measured completion (not even level 1) and satisfaction (level 1). The four numbers in the opening case are levels 3 and 4, which is where the board's real question ("are we safer?") is answered.
Combine leading indicators, which predict future performance, with lagging indicators, which record what already happened:
| Leading indicators | Lagging indicators |
|---|---|
| Share of high-risk roles trained and assessed | Data-leakage incidents through AI tools |
| Results of AI-generated phishing and deepfake-call simulations | Losses from AI-enabled fraud |
| Share of AI uses registered before rather than after launch | Shadow AI discovered |
| Override rate and quality of overrides in sampled reviews | Audit findings on AI oversight |
| Secure-design findings per agent in code review | Security findings in AI red-team tests |
Worked example: a scorecard for the board
| Measure | Level | Baseline | Target in 6 months |
|---|---|---|---|
| Developers passing the agent-security lab | 2 | 15% | 90% |
| DLP alerts for customer data in public AI tools, per 1,000 staff | 3 | 2.2 per month | Below 0.5 |
| Underwriter override rate in sampled cases where the AI was wrong | 3 | Rarely caught | At least 80% caught |
| Finance staff passing a simulated deepfake call (call-back performed) | 3 | 30% | 95% |
| AI-enabled fraud losses | 4 | USD 240,000 per quarter | Zero |
| AI uses registered before launch | 3 | 40% | 95% |
Signs of a risk-aware AI culture
Culture shows in behavior. In a healthy AI culture:
- teams register AI uses before building them;
- people report AI errors and near misses without fear of blame;
- reviewers override the AI when they disagree, and their overrides are valued;
- security is invited into design discussions early;
- staff feel able to say "I'll verify that" to a senior voice on a call.
Watch for common measurement traps:
- treating 100% completion as success;
- measuring only reaction surveys;
- celebrating zero reported incidents, which may mean people are not reporting;
- rewarding the speed of AI adoption without any measure of safe adoption.
The opening case, resolved
- Why 100% completion told the board almost nothing: it measured attendance and, at best, Kirkpatrick level 1. The four numbers that matter (DLP alerts, an unsafe agent, rubber-stamp overrides and a deepfake payment) are level 3 and level 4 measures, and they barely moved.
- What was wrong with the program: one generic module for 18,000 people, not role-based, not prioritized by risk, with no hands-on practice and no behavior measures. Arguably it does not meet the AI Act's requirement to take account of context and of the people affected.
- What would work: a skills gap analysis; priority tracks for agent developers, AI-output reviewers and finance staff; labs and simulations; just-in-time DLP coaching; and a scorecard of leading and lagging indicators reported to the board instead of completion.
Check your understanding
Try each before reading the reasoning.
1. Which is the BEST evidence that training for human reviewers of an AI credit tool is working?
Reasoning: a behavior (level 3) measure, such as sampled cases where the AI was wrong and the reviewer caught and overrode it, together with the quality of override reasons. Completion and satisfaction are weak evidence; a post-test is level 2.
2. An organization reports zero AI-related incidents for the year and proposes to cut the awareness budget. What should you consider?
Reasoning: zero reported incidents may mean under-reporting, not safety. Check discovery data (shadow AI, DLP alerts, simulation failure rates) and whether reporting channels are trusted and used before concluding the risk is low.
3. A deployer of a high-risk AI system asks whether its general AI awareness module meets the EU AI Act's literacy duty for the staff who oversee the system. What would you advise?
Reasoning: Article 4 requires literacy measures that take account of staff knowledge, the context of use and the people affected. Overseers of a high-risk system need role-specific training on its limitations, automation bias and how to override and escalate; a generic module alone is unlikely to be sufficient.
Key takeaways
- AI literacy is a security control and, under Article 4 of the EU AI Act, a legal duty since 2 February 2025.
- Start with a skills gap analysis and prioritize by risk: builders of agents, overseers of AI decisions and staff exposed to AI-enabled fraud come first.
- Train by role, weave ethics into every track, and favor labs, simulations and just-in-time coaching over annual videos.
- Measure with the Kirkpatrick levels; completion is not a level. Report leading and lagging indicators.
- Healthy culture shows in early registration, blame-free reporting, valued overrides and verification under pressure. Zero reported incidents is a question, not an answer.