The writing has been on the wall, and not exactly new. Five years ago, risk management was retrospective: audits, point-in-time assessments and quarterly reporting. Today, with AI, this practice has moved that needle to continuous assurance. The OWASP AI Testing Guide is not another checklist. It is the industry's first open, community-driven standard for trustworthiness testing of AI systems, and its central argument is one that most security teams are not yet ready to hear: security is not sufficient. AI trustworthiness is the real objective.

The data backs this up. According to Adversa AI's 2025 security report, 35% of real-world AI security incidents were caused by simple prompts, with some leading to losses exceeding $100,000 per incident. These are not exploits in the traditional sense. They are not buffer overflows or SQL injections. They are carefully crafted sentences that cause a model to hallucinate, leak training data, or bypass its own guardrails. A system can pass every security scan in your CI/CD pipeline and still fail catastrophically when a human tries to break it on purpose.

There are three essential frameworks in this discourse: the OWASP AI Testing Guide, the Cyber Security Agency of Singapore's Guidelines on Securing AI Systems, and IMDA's Agentic AI Governance Framework. Together, they form a practical stack that moves organisations from "we tested the API" to "we can defend the system."

The Blind Spots of Conventional Penetration Testing in AI

Traditional application security testing was built for deterministic systems. The same input produces the same output. A bug either exists or it does not. AI systems break this model entirely. They learn, adapt, generalise, and fail probabilistically. The failure modes are not just about confidentiality, integrity, and availability. These extend to the trustworthiness failures depicted below.

Failure Mode	What It Looks Like in Practice
Adversarial manipulation	Prompt injection, jailbreaks and model evasion attacks that use language, not code, as the attack vector.
Bias and fairness failures	A model that is technically secure but systematically discriminates against certain groups.
Sensitive information leakage	Training data extraction through carefully crafted queries.
Hallucinations and misinformation	Confidently stated falsehoods (confabulations) that carry real-world consequences.
Supply chain poisoning	Compromised training data, models or plugins that introduce backdoors.
Excessive or unsafe agency	Autonomous systems that take actions beyond their intended scope.
Model drift	Degradation over time as the world changes around a static model.

Table 1: AI-specific failure modes that conventional penetration testing was never designed to catch

The OWASP AITG's foundational move is to codify this difference. It establishes that AI systems require a testing discipline that evaluates trustworthiness properties across four layers: the AI Application Layer, the AI Model Layer, the AI Infrastructure Layer, and the AI Data Layer. Each layer carries distinct risks and demands distinct test cases. Table 2 contrasts the two testing paradigms.

Dimension	Traditional Penetration Testing	AI Security & Trustworthiness Testing
System behaviour	Deterministic — same input, same output; a bug either exists or it does not	Probabilistic — a test that passes once may fail on a reworded input
Attack vector	Code and configuration (e.g. buffer overflows, SQL injection)	Language and data (prompt injection, jailbreaks, data poisoning
Objective	Confidentiality, integrity, availability (CIA)	Trustworthiness — security plus fairness, robustness, explainability and alignment
Cadence	Point-in-time assessments, quarterly reporting	Continuous assurance at every model update, prompt change and new data source
Scope	Application, network, infrastructure	Four layers: AI application, model, infrastructure and data
Team	Security practitioners	Cross-functional: security, data science and ML engineerin

Table 2: Traditional penetration testing versus AI security and trustworthiness testing

This is not a theoretical refinement. The EU AI Act requires adversarial testing as part of the risk management system for high-risk AI systems, with full compliance required by August 2, 2026. Organisations that treat AI testing as an extension of AppSec are already behind.

The OWASP AI Testing Guide: Trustworthiness as the North Star

The OWASP AITG v1 operationalises a principle that the security community has been circling for years: AI systems must be tested not just for security vulnerabilities but for the broader properties that make them trustworthy. The guide provides a standardised, repeatable methodology that is technology-agnostic and globally relevant.

The Four-Layer Testing Model

The guide organises testing across four layers, each with its own threat profile and test methodology:

AI Application Layer. This is the layer most familiar to AppSec teams. It covers the user-facing interface, APIs, and integration points. Testing here addresses prompt injection, jailbreak attempts, output filtering, and access control. The key insight is that application-layer testing for AI must account for the model's non-deterministic behaviour. A prompt injection test that passes once may fail on the next attempt with a slightly reworded input.

AI Model Layer. This is where the guide departs most sharply from traditional security testing. Model-layer testing evaluates the model itself: adversarial robustness, bias and fairness, explainability, and alignment with intended behaviour. The guide introduces the DIE Triad (Distributed, Immutable, Ephemeral) as a complement to the traditional CIA triad, shifting the focus from hardening individual components to making the entire system resilient. If a model node is compromised, it should be killed and replaced instantly, not patched.

AI Infrastructure Layer. The infrastructure that serves AI models carries familiar risks (container escapes, misconfigured storage, exposed endpoints) but with AI-specific amplification. A compromised model registry can poison every downstream deployment. An exposed vector database can leak embeddings that reveal sensitive training data. The guide defines infrastructure as a trust boundary and requires teams to harden it against both conventional and AI-specific threats.

AI Data Layer. Data is the most overlooked attack surface in AI systems. The guide covers training data provenance, data pipeline integrity, and the specific risks of retrieval-augmented generation (RAG) architectures where external data retrieval can introduce indirect prompt injections. Data poisoning, whether at training time or through ongoing fine-tuning, is one of the hardest attacks to detect and one of the most consequential.

What the Guide Changes in Practice

For security teams, the OWASP AITG changes the testing cadence. You cannot run a one-time penetration test on an AI system and call it secure. The guide implies continuous testing, with adversarial robustness evaluated at every model update, every prompt template change, and every new data source integration. It also demands cross-functional participation: data scientists, ML engineers, and security practitioners must collaborate on test design because no single function understands all four layers.

CSA's Lifecycle Approach: Secure by Design, Secure by Default

Nearer to home in Singapore, the Cyber Security Agency of Singapore released its Guidelines on Securing AI Systems in October 2024, alongside a Companion Guide developed with industry practitioners. The guidelines are voluntary but carry Singapore's characteristic pragmatism: they are designed to be actionable, not aspirational.

The CSA's foundational principle is that AI should be secure by design and secure by default, as with all digital systems. The guidelines take a lifecycle approach, arguing that hardening only the AI model is insufficient. Security must be embedded across five stages:

Planning and Design

Security begins before the first line of code. The planning phase requires organisations to anticipate potential threats, including adversarial machine learning and supply chain attacks. Threat modelling at this stage must account for the specific capabilities of the AI system being built. A customer-facing chatbot and an autonomous trading agent face fundamentally different threat profiles, and the design phase is where those differences should shape architecture decisions.

Development

The development phase emphasises securing AI assets and ensuring robust protection across the AI supply chain. This includes model provenance verification, training data integrity checks, and dependency auditing for AI frameworks and libraries. The Companion Guide references the MITRE ATLAS database and the OWASP Top 10 for LLM Applications as practical resources for identifying development-stage threats.

Deployment

This phase focuses on deploying AI systems in secure environments with strong infrastructure safeguards. The guidelines call for hardened serving infrastructure, network segmentation, API authentication, and runtime monitoring. The deployment phase is also where organisations should implement the guardrails that constrain model behaviour in production.

Operations and Maintenance

This is the longest phase and the one where most AI security failures occur. The guidelines require continuous monitoring for anomalies, regular model evaluation, and vulnerability disclosure processes. Model drift, data drift, and emergent behaviours all surface during operations, and the security posture must be dynamic enough to catch them. The Companion Guide curates practical measures and controls from industry and academia, recognising that AI security is a developing field where mitigation controls continue to evolve.

End of Life

Proper disposal of AI-related data and artefacts at the end of their lifecycle ensures that outdated systems do not become a target. This includes model weights, training data, embeddings, and any derived artefacts. The end-of-life phase is often neglected in AI security discussions, but a decommissioned model that still contains sensitive training data or proprietary weights is a liability.

The Agentic AI Addendum

In late 2025, CSA released an addendum on securing agentic AI systems, recognising that AI agents capable of autonomous planning, reasoning, and tool use introduce risks that the original guidelines did not fully anticipate. The addendum outlines how risks can be identified and assessed based on agentic capabilities, maps agentic workflows to identify where threat actors could exploit vulnerabilities, and provides practical controls across the development lifecycle. It includes case studies across app development assistants, automated client onboarding systems, and fraud detection systems. 

The Agentic AI Frontier: IMDA's Governance Framework

If CSA's guidelines address the technical security of AI systems, IMDA's Model AI Governance Framework for Agentic AI, unveiled at the World Economic Forum in Davos in January 2026 and updated in May 2026, addresses the governance challenge. It is the world's first governance standard designed specifically for agentic AI: systems that can autonomously reason, plan, and take actions in pursuit of goals without human intervention at each step.

The framework is structured around four core dimensions, and each one has direct implications for security testing:

1. Assessing and Bounding the Risks

Agentic AI systems introduce cascading failure modes. An agent that can browse the web, execute code, and make API calls multiplies the attack surface with each capability. The framework requires organisations to map agentic workflows explicitly, identifying every decision point where an agent could take an unintended action. For security testers, this means red-teaming must cover not just what the model says but what the agent does. A prompt that causes an agent to execute an unauthorised API call is a security failure, even if the model's textual response looks benign.

2. Ensuring Meaningful Human Accountability

The framework rejects the "human in the loop" checkbox. It requires meaningful human accountability: clear assignment of responsibility when agentic systems cause harm, and governance structures that ensure humans can actually intervene, not just rubber-stamp automated decisions. For testing, this translates to evaluating whether human oversight mechanisms are real or performative. Can a human override the agent? Is there a kill switch? Who is accountable when the agent acts?

3. Implementing Technical Controls and Processes

The May 2026 update to the framework added substantial depth here, incorporating industry feedback on multi-agent systems, third-party agents, and automation bias. Multi-agent systems introduce coordination risks where emergent behaviours can arise from the interaction of individually safe agents. Third-party agents bring supply chain risks that are harder to audit than traditional software dependencies. The framework recommends controls including sandboxed execution environments, capability boundaries, and continuous monitoring of agent behaviour against declared objectives.

4. Enabling End-User Responsibility

The framework recognises that end users interacting with agentic systems need to understand what they are dealing with. Transparency about agentic capabilities, clear disclosure when users are interacting with an autonomous agent, and education about the limits of agent reliability are all part of the governance picture. For security testing, this dimension surfaces the need to test not just the system but the user experience around it. Does the interface make the agent's autonomy clear? Are users warned before the agent takes high-stakes actions?

Connecting Testing to Governance: The Bigger Picture

Security testing does not exist in isolation. It is one pillar of a broader AI governance structure that includes risk management, compliance, and board-level oversight. Two frameworks help connect the testing work to the governance whole.

ISO 42001: The Management System Backbone

ISO/IEC 42001:2023, the international standard for AI management systems, provides the organisational scaffolding that makes testing programs sustainable. It defines an AI Management System (AIMS) as a set of interconnected elements that create policies, objectives, and processes for responsible AI development and use. Its 38 controls across 10 control objectives cover governance, risk management, data quality, third-party oversight, and lifecycle management.

The standard's Plan-Do-Check-Act methodology maps directly onto the testing cadence that the OWASP AITG and CSA guidelines demand. Plan: identify AI risks and define testing scope. Do: execute tests across all four layers. Check: evaluate results against trustworthiness criteria. Act: remediate and update the testing program. Organisations that have implemented ISO 42001 have a ready-made process for absorbing the testing requirements these newer frameworks introduce.

PDCA Stage	ISO 42001 Management Activity	AI Testing Programme Equivalent
Plan	Identify AI risks; define AIMS policies and objectives	Identify AI risks and define testing scope across the four AITG layers
Do	Implement controls and processes	Execute tests across the application, model, infrastructure and data layers
Check	Monitor, measure and audit performance	Evaluate results against trustworthiness criteria; maintain the audit trail
Act	Correct, improve and update the AIMS	Remediate findings and update the testing programme

Table 3: Mapping ISO 42001's Plan-Do-Check-Act cycle onto the AI testing cadence

The standard also provides the documentation and record-keeping discipline that regulators will expect. Every test result, every risk assessment, every model evaluation becomes part of an auditable trail. For organisations facing the EU AI Act's August 2026 compliance deadline, that audit trail is not optional.

Toward a Unified Testing and Governance Posture

The three frameworks profiled here are complementary, not competing. The OWASP AITG provides the testing methodology. CSA's guidelines provide the lifecycle structure and practical controls. IMDA's agentic AI framework provides the governance lens for the most dangerous class of AI systems emerging today. ISO 42001 provides the management system that makes the whole thing auditable and sustainable. Table 4 summarises the stack.

Framework	Issued by	Role in the Stack
OWASP AI Testing Guide v1 (2025)	OWASP Foundation	The testing methodology — trustworthiness testing across four layers
Guidelines & Companion Guide on Securing AI Systems (2024), with Agentic AI Addendum (2025)	CSA Singapore	The lifecycle structure and practical controls — secure by design, secure by default
Model AI Governance Framework for Agentic AI (2026)	IMDA Singapore	The governance lens for autonomous, agentic AI systems
ISO/IEC 42001:2023	ISO/IEC	The management system — makes the programme repeatable, auditable and sustainable
EU AI Act (Reg. (EU) 2024/1689)	European Union	The regulatory driver — adversarial testing mandatory for high-risk AI by August 202

Table 4: The unified AI testing and governance stack

An organisation that wants to get serious about AI security testing should start with these steps:

Adopt the OWASP AITG as your testing baseline. Map your AI systems to the four-layer model. Identify which layers you are testing and which you are not. The gaps are your risk.

Implement CSA's lifecycle approach. Secure by design means security is present at planning, development, deployment, operations, and end of life. If security only shows up at deployment, you are already too late.

Assess your agentic exposure. If your organisation is deploying or planning to deploy autonomous agents, the IMDA framework's four dimensions are the governance minimum. Map agentic workflows. Identify every decision point. Test what happens when prompts push the agent beyond its intended scope.

Build the management system. ISO 42001 certification is not required for every organisation, but the discipline it enforces is. Document your testing program. Make it repeatable. Make it auditable.

Educate the board. Directors do not need to understand adversarial prompt engineering. They do need to understand the residual risk the organisation is carrying and what is being done about it. The testing program must produce board-ready metrics.

The moment a client or regulator can trust your controls, doors open. Deals will close faster, audits will pass cleaner, and new markets become more accessible. Cybersecurity testing looks like a cost centre when measured in isolation from the business outcomes it enables.

References

[1] OWASP Foundation, “OWASP AI Testing Guide, Version 1.0”, November 2025. Available at: https://owasp.org/www-project-ai-testing-guide/

[2] Adversa AI, “AI Security Incidents Report 2025”, 2025. Available at: https://adversa.ai/

[3] Cyber Security Agency of Singapore (CSA), “Guidelines and Companion Guide on Securing AI Systems”, 15 October 2024. Available at: https://www.csa.gov.sg/resources/publications/guidelines-and-companion-guide-on-securing-ai-systems

[4] Cyber Security Agency of Singapore (CSA), “Securing Agentic AI — An Addendum to the Guidelines and Companion Guide on Securing AI Systems”, 2025–2026. Available at: https://www.csa.gov.sg/resources/publications/addendum-on-securing-ai-systems/

[5] Infocomm Media Development Authority (IMDA), “Model AI Governance Framework for Agentic AI”, unveiled at the World Economic Forum, Davos, January 2026; updated May 2026. Available at: https://www.imda.gov.sg/

[6] International Organization for Standardization / International Electrotechnical Commission, “ISO/IEC 42001:2023 — Information technology — Artificial intelligence — Management system”, December 2023. Available at: https://www.iso.org/standard/81230.html

[7] European Parliament and Council, “Regulation (EU) 2024/1689 (EU Artificial Intelligence Act)”, Article 9 and Annex-related obligations for high-risk AI systems; full applicability for high-risk systems from 2 August 2026. Available at: https://eur-lex.europa.eu/eli/reg/2024/1689/oj

[8] MITRE Corporation, “MITRE ATLAS — Adversarial Threat Landscape for Artificial-Intelligence Systems”. Available at: https://atlas.mitre.org/

[9] OWASP Foundation, “OWASP Top 10 for Large Language Model Applications”. Available at: https://genai.owasp.org/