A support bot that answers from your docs is a search box with good manners. A support bot that can read a customer's account and send an email is a new employee who will follow instructions from anyone who writes to the shared inbox. Most LLM features shipped this year are the second kind, and most were tested as if they were the first.
LLM application penetration testing is not a contest to make the model say something rude. It asks a narrower question: when someone else's text reaches the model, what can the model touch, and how does data get out?
The model is an untrusted caller. Test it like one, and judge every finding by the tools it holds, not by how clever the wording was.
Why prompt injection is a design problem, not a bug to patch
A language model reads instructions and data in the same stream. There is no equivalent of a bound SQL parameter. The UK National Cyber Security Centre made the point bluntly in Prompt injection is not SQL injection (it may be worse): there is no clean fix, so you reduce likelihood and impact through design. OWASP LLM01:2025 says the same thing in standards language. It is unclear whether any method fully prevents it.
The vendors agree. OpenAI told TechCrunch that prompt injection is unlikely ever to be fully solved for browser agents. Anthropic's prompt injection research reports large drops in attack success from training and classifiers, and still says no browser agent is immune. If the companies training the models plan for a residual rate, your feature should too.
The direct form is a user typing "ignore your instructions." The form that hurts is indirect. Greshake et al. (2023) showed attackers planting instructions in content an app would later retrieve: a web page, an email, a document. The user never sees the payload. The model reads it while doing its job.
Map the lethal trifecta before you test anything
Simon Willison's lethal trifecta is the best scoping tool we know for this work. An agent can be turned into a data thief when it has three things at once: access to private data, exposure to untrusted content, and a way to communicate outward. Remove any one leg and the theft path breaks.
FOR EACH LLM FEATURE, ANSWER THREE QUESTIONS
[A] Private data
What can it read? CRM rows, tickets, files,
other tenants' records, env vars, the system prompt
[B] Untrusted content
What text reaches it that you did not write?
Emails, support tickets, uploads, web pages,
GitHub issues, form fields, RAG documents
[C] A way out
How can it send data somewhere?
Email or webhook tools, links, Markdown images,
PR comments, allowlisted domains, its own logs
A + B + C -> test exfiltration end to end
Two of three -> test whether a third leg can be added
(a new tool, a renderer, an integration)Leg C is the one teams miss. A chat UI that renders Markdown images has an outbound channel: the browser fetches the image URL, and the URL can carry whatever the model put in it. No tool call, no click.
What the incidents teach
EchoLeak (CVE-2025-32711) in Microsoft 365 Copilot is the reference case. One crafted email, worded to avoid tripping the injection classifier, was retrieved into Copilot's context during ordinary business questions. Copilot put internal data into a URL. A reference-style Markdown image slipped past link redaction, and an allowed Microsoft Teams proxy carried the request out. The user did not click anything. The academic write-up and Checkmarx's analysis walk through each step. Microsoft fixed it server-side in June 2025.
Slack AI (PromptArmor, 2024) followed the same shape. An attacker posted instructions in a public channel only they were in. When a victim asked Slack AI a question, the answer pulled in the victim's private API key and rendered a "reauthenticate" link with the key as a URL parameter. The attacker's message was not cited, so the source was hard to trace.
Developer agents widen this. Invariant Labs planted an issue in a public repository. An agent asked to review open issues then read the user's private repositories and opened a public pull request containing salary data. The tools were working as designed. General Analysis showed a support ticket steering a coding assistant into running SQL as Supabase's service_role, which bypasses row-level security, and pasting integration tokens back into the ticket for the attacker to read.
The 2026 cases moved to business agents. VentureBeat reported payloads submitted through a public SharePoint form and a public lead form that got Copilot Studio and Agentforce agents to email out list and CRM data. DLP stayed quiet because the agent used a legitimate send action.
Every one of these is the trifecta. None needed a jailbreak that would impress anyone.
Tool permissions are where impact lives
OWASP LLM06:2025 Excessive Agency splits the problem three ways: tools the feature does not need, permissions broader than the task, and high-impact actions taken with no check. Each one is testable.
Excessive functionality is a tool list that grew. A summarizer that can also delete records, or a generic "run query" or shell tool where three narrow functions would do. Excessive permissions is the shared service account. If the agent acts as a superuser instead of as the signed-in customer, a prompt injection becomes a cross-tenant read. Excessive autonomy is an agent that sends, deletes, or pays without anyone confirming.
The fix that works is the boring one. Authorization is enforced in the API behind the tool, using the end user's identity, exactly as if a browser had made the call. The model never decides who is allowed to see what. OWASP LLM07 adds the corollary: the system prompt is not a secret and not a control. If extracting it reveals an API key or the rules for who is an admin, that is the finding.
What an LLM feature pentest should cover
LLM FEATURE PENTEST: WHAT IT SHOULD PROVE
[1] Every ingestion path tested for indirect injection
tickets, emails, uploads, RAG sources, form fields
[2] Tool calls authorized as the end user
User A's session cannot make the agent read User B
[3] No shared superuser behind any tool
read-only where reads are enough, scoped tokens
[4] Outbound channels closed or allowlisted
Markdown images, auto-fetched links, webhooks,
email tools, redirects through allowed domains
[5] High-impact actions need a human
send, delete, pay, change permissions
[6] System prompt holds no secrets or authz rules
[7] Model output treated as untrusted input
escaped before it hits HTML, SQL, or a shellItem 7 is OWASP LLM05, improper output handling. When model output lands in a page unescaped, it is ordinary cross-site scripting with an unusual author. When it lands in a query, it is ordinary injection. Classic web testing still applies; the model is just one more place attacker text comes from.
Defenses worth testing for are architectural. The design patterns paper from researchers across industry and academia, summarized by Willison, comes down to one rule: once an agent has read untrusted input, it must not be able to trigger consequential actions. Google DeepMind's CaMeL enforces that by deriving control flow from the trusted user request only, and pays for it with somewhat lower task completion. Classifiers and stronger prompts help at the margin. They are not a boundary.
Put the AI feature in the same pentest as the app
The AI feature is not a separate product. It calls your APIs, reads your database, and renders in your frontend. The bugs that make prompt injection expensive are usually the old ones: an endpoint that skips the ownership check, a service token with admin scope, an output path that is not escaped. Testing the model alone misses them. Testing the app without the model misses how they chain.
GhostProbe tests both in the same job. The Pentest is $1,490 for up to four staging or preproduction addresses you own, with optional source from up to 2 GitHub repositories, so the review can see which tools an agent holds and what identity they run as. Recon is $490 for a single address. A security engineer reproduces every finding, removes the noise, writes the fix, and signs the dossier.
Point it at the staging build where the agent has its tools, and find out which leg of the trifecta to cut before someone else does.