Penetration Testing
AI & LLM Penetration Testing and AI Red Teaming
AI red teaming is manual, adversarial testing of your AI and large language model applications, from chatbots and copilots to RAG search and agents that call tools. We test for prompt injection, data leakage, jailbreaks and excessive agency against the OWASP Top 10 for LLM Applications and, as an ISO 27001-certified firm, handle your prompts and data securely.
Prefer to talk? Call +44 161 523 5276
- OWASP Top 10 for LLM Applications
- Prompt injection, jailbreaks and data leakage
- Agent and tool-use testing
Trusted by banks, payment providers and global brands
On this page
- LLM security testing: what's included
- AI penetration testing: what we test for
- AI red teaming vs AI penetration testing vs LLM red teaming
- Which frameworks guide AI red teaming?
- What does an AI security assessment cover beyond the model?
- Testing agents, tools and MCP servers
- Prompt injection testing
- How is an AI penetration test carried out?
- Why choose Sovereign Secure for AI and LLM penetration testing?
- Frequently asked questions
Overview
What is AI red teaming?
The short answer
AI red teaming, or LLM penetration testing, is adversarial testing of an AI-enabled application to find what it can be made to say, reveal or do. It covers prompt injection, jailbreaks, data leakage, unsafe output handling and excessive agency, tested against the OWASP Top 10 for LLM Applications, plus the web and API layer around the model.
AI red teaming, also called AI or LLM penetration testing, means attacking an AI-enabled application the way a real adversary would, to find out what it can be made to say, reveal or do. This is testing of your AI and LLM applications. It is different from an “AI-powered” penetration testing tool, which uses AI to scan conventional systems.
AI and LLM applications introduce risks that traditional testing does not cover. A model can be manipulated through crafted input, tricked into revealing data it should protect, or, when connected to tools and data, driven into taking actions it should not. Our AI security testing covers these specific weaknesses alongside the conventional flaws in the surrounding application.
Our coverage follows the 2026 OWASP Top 10 for LLM Applications, from prompt injection (LLM01) and sensitive information disclosure (LLM02) to excessive agency (LLM03), unbounded consumption (LLM06) and hidden context exposure (LLM08), formerly system prompt leakage. Our OWASP LLM Top 10 glossary entry explains each risk in plain English, and the groups below show how each one maps to what we test.
Because an LLM feature is still part of a wider application, we test the web application and API layer around it too. For objective-led exercises that combine AI and conventional attack paths, see our red team assessments. Findings support ISO 27001 evidence, as set out on our ISO 27001 penetration testing page. New to testing? Our guides to what penetration testing is and the types of penetration testing show where this test fits.
What's included
LLM security testing: what's included
An AI engagement covers the model, its integrations and the application around it.
- 01
Prompt injection testing
Direct and indirect prompt injection attempts to bypass instructions, guardrails and safety controls.
- 02
Data leakage testing
Attempts to extract sensitive data, system prompts, training data or context the model should not reveal.
- 03
Output handling review
Assessment of how model output is used downstream, where unsafe handling can lead to injection or code execution.
- 04
Agent and tool-use testing
Testing of models connected to tools, functions and data sources for unsafe or excessive actions.
- 05
Authorisation and access checks
Verification that the AI feature respects the same access controls as the rest of the application.
- 06
Guardrail and abuse testing
Testing of content filters and safety controls, and of how the system resists misuse.
- 07
Surrounding application testing
The web and API layer around the AI feature, tested for conventional vulnerabilities.
- 08
Retest
A retest of remediated findings, scoped in your fixed-price proposal, with the report updated to show them closed.
What we test for
AI penetration testing: what we test for
Each group shows the risks it covers from the OWASP Top 10 for LLM Applications (2026), the list our LLM Top 10 glossary entry summarises.
Prompt injection and jailbreaks (LLM01)
Crafted input that overrides intended instructions and controls.
- Direct prompt injection
- Indirect injection via retrieved content
- Jailbreaks and guardrail bypass
- Instruction and role confusion
- Injection through files, images and web pages
Sensitive information and hidden context exposure (LLM02, LLM08)
Extraction of data and instructions the model should keep protected.
- Hidden context exposure: system prompts, tool definitions and instructions
- Disclosure of confidential context
- Extraction of connected data sources
- Leakage of other users’ data
- Secrets embedded in prompts
Excessive agency and tool use (LLM03)
Models connected to tools and actions with too much freedom.
- Unsafe function and tool invocation
- Actions beyond the intended scope
- Insufficient human oversight of actions
- Over-broad permissions granted to the model
- Chaining of tools towards harmful outcomes
Supply chain, data and model poisoning (LLM04, LLM05)
Risks in the data, components and services the system depends on.
- Training and retrieval data poisoning
- Vulnerable model or plugin components
- Insecure third-party model services
- Untrusted content added to knowledge bases
- Model and dependency provenance
Resource abuse, misinformation and vector stores (LLM06, LLM07, LLM09)
Weaknesses in cost controls, answer reliability and retrieval.
- Unbounded token and request consumption
- Denial-of-wallet exposure
- Confidently wrong answers in high-risk flows
- Cross-tenant leakage through embeddings
- Access control on vector databases
Improper output handling (LLM10)
Model output trusted downstream without proper handling.
- Cross-site scripting via generated output
- Injection into downstream systems
- Code or command execution from output
- Unsafe rendering of model responses
- Missing validation of generated content
Surrounding application
The conventional application around the AI feature.
- Authentication and authorisation flaws
- API vulnerabilities behind the feature
- Injection in the wider application
- Insecure configuration and secrets
- Rate limiting and abuse controls
AI red teaming vs AI penetration testing vs LLM red teaming
The three terms overlap, and providers use them loosely. The useful distinction is between testing what a model can be made to do and testing the whole application that uses it.
| Term | What it usually means | Typical focus |
|---|---|---|
| AI red teaming | Adversarial testing of an AI system’s behaviour, trying to make it break its rules | Jailbreaks, harmful or misleading output, misuse and safety controls |
| AI penetration testing | A scoped security test of an AI-enabled application, run and reported like any other penetration test | Prompt injection, data leakage, unsafe output handling, agents and tools, plus the surrounding web and API layer |
| LLM red teaming | AI red teaming aimed specifically at large language models and the applications built on them | Prompt injection, system prompt leakage, jailbreaks and data extraction |
Our engagements cover the security side of all three: what the model can be made to reveal or do, and the conventional application it sits in.
Which frameworks guide AI red teaming?
Three public frameworks shape most AI red teaming work. None is a test method on its own, so each does a different job: one lists the risks to test, one describes how attackers behave, and one turns the results into managed risk.
| Framework | What it is | How it is used in AI red teaming |
|---|---|---|
| OWASP Top 10 for LLM Applications (2026) | A ranked list of the ten most important security risks in applications built on large language models, from prompt injection (LLM01) to improper output handling (LLM10) | The checklist for test coverage, and the reference each finding is reported against |
| MITRE ATLAS | A public knowledge base of the tactics and techniques used against AI systems, drawn from real incidents and research and modelled on MITRE ATT&CK | A source of realistic attack scenarios when planning tests, and a shared vocabulary for describing them |
| NIST AI Risk Management Framework | A voluntary framework for managing AI risk through four functions (Govern, Map, Measure and Manage), with a Generative AI Profile that calls for pre-deployment testing | Placing red team findings in the Measure function as evidence, and feeding the fixes into Manage |
AI pen testing covers the application around the model as well as the model itself, so the conventional OWASP Top 10 and the OWASP API Security Top 10 still apply to the web and API layer.
What does an AI security assessment cover beyond the model?
Many of the most serious AI weaknesses sit around the model rather than in it: in what it is connected to, what data it can reach and how its output is used. An AI security assessment therefore looks at the whole system.
| Area | What is checked |
|---|---|
| Retrieval (RAG) and vector stores | Whether users can retrieve documents they should not see, and whether planted content can steer answers |
| Tools, plugins and agents | What actions the model can trigger, with whose permissions, and whether approval steps hold |
| Identity and access | Whether the AI feature enforces the same access controls as the rest of the application |
| Output handling | Whether generated text, code or links are rendered or executed safely downstream |
| Third-party model services | What data is sent to external model providers, and how keys and settings are protected |
| Logging, rate limits and cost controls | Whether misuse is noticed, and whether request floods could run up cost or deny service |
| The surrounding application | The web front end and APIs behind the feature, tested like any other application |
Testing agents, tools and MCP servers
When a model can call tools, the question changes from what it might say to what it might do. We test whether an agent can be driven, through direct instructions or content it reads, to call tools outside its intended scope, use permissions broader than the task needs, or skip a human approval step.
Many agents now reach tools through the Model Context Protocol (MCP). Testing covers the servers an agent connects to: the permissions and tokens they hold, tool descriptions and results that could carry injected instructions, and server packages from untrusted sources. Our guides to agentic AI security and MCP security set out the threat models in more detail.
Prompt injection testing
Prompt injection is input crafted to make a model ignore its instructions. Direct injection comes from the user; indirect injection hides in content the model reads, such as a web page, an email, a document or a tool result. There is no complete fix, so testing checks two things: how easily the model can be steered, and how much damage a successful injection could do with the data, tools and permissions the model has. The second answer usually shapes the fix: least privilege, careful output handling and human approval for sensitive actions.
How it works
How is an AI penetration test carried out?
- 1
Scoping
We agree the AI feature, its integrations, the data it can reach and the surrounding application, and fix the price.
- 2
Understanding
We map how the model is used, what it connects to, and where its input and output flow.
- 3
Testing
Manual testing for prompt injection, data leakage, unsafe output and excessive agency, plus the conventional application layer. Testers commonly use open-source tools such as PyRIT and garak to send large sets of attack prompts, then confirm and extend each result by hand.
- 4
Reporting
An executive summary and technical report with reproduction steps, ratings and remediation advice.
- 5
Debrief
A walkthrough with your team covering the findings, the risk and the fixes.
- 6
Retest
Once remediated, we retest and reissue the report showing findings closed.
Every test follows the five stages of penetration testing: planning and reconnaissance, scanning, gaining access, maintaining access, and analysis and reporting.
Why Sovereign Secure
Why choose Sovereign Secure for AI and LLM penetration testing?
Established in 2012, we deliver PCI, ISO and Cyber Essentials work for banks, payment processors, card manufacturers and fast-growing companies.
Read client testimonials →- Testing aligned to the OWASP Top 10 for LLM Applications, the recognised framework for this domain.
- Coverage of the AI-specific attack surface and the conventional application around it.
- Manual testing that reflects how these systems are actually attacked.
- Fixed scope and price, with any retest and its window agreed in your proposal.
FAQ
Frequently asked questions
Can't find your answer? Call us on +44 161 523 5276 or send us a message.
Why do AI and LLM applications need their own testing?
They introduce risks that conventional testing does not cover, such as prompt injection, data leakage through the model, and unsafe actions when a model is connected to tools. These need to be tested alongside the usual application vulnerabilities, using techniques suited to how AI systems fail.
What is prompt injection?
Crafted input that makes a model ignore its instructions or safety controls; our prompt injection glossary entry explains direct and indirect attacks, and prompt injection testing is a core part of every AI engagement we run.
Do you test AI agents that can take actions?
Yes. Where a model is connected to tools, functions or data, we test for excessive agency: whether it can be driven to take actions beyond its intended scope, and whether human oversight and permissions are adequate. Our agentic AI security page sets out the agent threat model in more detail.
What do you need from us for an AI or LLM penetration test?
Access to the AI feature, details of the model and how it is integrated, what data and tools it can reach, and test accounts. Documentation of the system prompts and guardrails helps us test thoroughly. We confirm the detail during scoping.
Do you test the application around the AI feature too?
Yes. An AI feature is part of a wider application, so we test the surrounding web and API layer for conventional vulnerabilities as well as the AI-specific risks.
How do you test LLM security?
We map how the model is used, what it can reach and where its input and output flow (a short threat modelling exercise), then test by hand against the OWASP Top 10 for LLM Applications: prompt injection and jailbreaks, sensitive data and hidden context exposure such as system prompts, unsafe output handling and excessive agency. We also test the web and API layer around the model, and report each finding with reproduction steps and a fix.
Does this support ISO 27001 and AI governance?
Yes. Testing supports ISO 27001:2022 Annex A 8.8 and 8.29 and aligns with AI risk frameworks such as the NIST AI Risk Management Framework, giving you evidence that AI features have been assessed. The report can also serve as evidence for an AI management system under ISO 42001, and for the robustness and cybersecurity testing the EU AI Act expects of high-risk AI systems. Our AI governance guide explains where that evidence fits in a wider programme.
Is AI red teaming the same as an AI-powered penetration test?
No. An AI-powered penetration test uses AI tools to test ordinary systems. AI red teaming tests the AI system itself: whether your chatbot, copilot or agent can be manipulated, made to leak data or pushed into unsafe actions. We do the second, by hand.
How much does AI and LLM penetration testing cost?
The price depends on how many AI features are in scope, what data and tools each can reach, whether agents can take actions, and how much of the surrounding web and API layer we test. Our penetration testing cost guide explains the drivers, and our AI penetration testing quote form returns a fixed price.
Last reviewed October 2026 by the Sovereign Secure assessment team.
Related services
- Web application testingWeb application penetration testing in the UK: manual OWASP Top 10, access-control and business logic testing by a PCI QSA company. Get a quote.
- API penetration testingAPI penetration testing services for REST, GraphQL and SOAP web services, tested against the OWASP API Security Top 10 by a PCI QSA company. Get a quote.
- Red team assessmentsRed teaming services and red team assessments: objective-led adversary simulation across technical, human and physical routes, under written rules of engagement.
- ISO 27001 penetration testingISO 27001 penetration testing mapped to Annex A 8.8 and 8.29: what auditors ask for, how often to test and an audit-ready evidence pack. Get a fixed quote.
- Penetration testing quoteScope your test types and asset counts and get a fixed-price penetration testing proposal.
Free consultation
Talk to an assessor
We aim to reply within one working day, and send a fixed quote once your scope is confirmed.
- PCI SSC-approved QSA, QPA, CPSA, P2PE and 3DS assessors
- IASME-licensed Cyber Essentials certification body
- Clients across the UK, US, Europe, Middle East and Asia since 2012
“The Cyber Security Services were delivered on time and price was competitive.”
Call +44 161 523 5276
Email [email protected]