While companies focus on developing smart agents, many overlook an uncomfortable truth that lies at the foundation of this technology. Current large language models (LLMs), the brains behind every AI agent, have a fundamental design flaw: they cannot reliably distinguish between instructions and data because there is only a single channel for everything. For an LLM, everything is ultimately just a single stream of words (tokens). Model manufacturers then attempt to build a structure on this flawed foundation that is supposed to ensure security:
- The token stream is structured using special, reserved tokens in messages of different roles: the instructions that developers give to the agent (system prompt), the instructions that end users can give (user prompt), as well as tool calls and results.
- The models are then trained to place more weight on the system prompt than on the user prompt, and to treat tool results more like data.
This is not security by design, but rather based on the principle of hope and can only conceal the fundamental problems. The implicitly trained priority of instructions can be ignored by the model – and cases that are not explicitly addressed by the system instructions cannot take precedence over user instructions. Furthermore, there is still no distinction between data and instructions within the system and user prompts. It is just as common (depending on the nature of the application) for the user prompt to contain only instructions (e.g. in a classic chatbot) or for the user prompt to contain only data (e.g. an application for AI-based summaries). It is up to the (questionable) intelligence and intuition of the model to correctly distinguish between data and instructions.
This is no minor issue. It means that (general-purpose) agents built on today's LLM architecture cannot provide reliable security guarantees. So if we want to seriously deploy agents in our businesses, we need to stop hoping for the best and start planning for the worst. We need to build security through thoughtful system design around the fundamental limitations, never solely on them.
The wolf in sheep's clothing: What exactly is prompt injection?
Imagine you give a task to a capable but very naive assistant. They are eager to help you and follow every instruction. That is exactly what an LLM is. Prompt injection exploits this ‘naivety’. An attacker smuggles malicious instructions into the text (the individual token stream) that the agent processes.
- Direct prompt injection: the attack through the front door
This is the most obvious variant. A user (or attacker) interacts directly with the agent – via chat, email or another input field – and tricks it into doing things it shouldn't. The results range from comical to catastrophic. Headlines about this are becoming increasingly common: a chatbot for the delivery service DPD was persuaded to insult its own company. Another from Chevrolet offered a car for one dollar in a ‘legally binding’ manner. These ‘screenshot attacks’ are more than just embarrassing; they can cause real financial damage and massive reputational loss. - Indirect prompt injection: the Trojan horse
This method is far more insidious. Here, an attacker places malicious instructions in external data that they know the agent will read later. This could be a website that the agent is supposed to summarise, a PDF document in a knowledge database, or even an email in a mailbox that the agent monitors. The agent picks up this data, interprets the hidden commands as legit instructions, and carries them out – without the actual user noticing anything.
Der wahre Albtraum: Wenn dein Agent zum Datendieb wird
Lustige Chatbot-Screenshots sind eine Sache. Der unkontrollierte Abfluss von Unternehmensdaten eine ganz andere. Der Softwareentwickler und AI-Experte Simon Willison hat hierfür den Begriff der „tödlichen Drei“ (The Lethal Trifecta) geprägt. Ein Agent wird zur tickenden Zeitbombe, wenn er drei Fähigkeiten kombiniert:
- Access to private data: Access to your emails, internal documents, customer databases, etc.
- Contact with untrustworthy content: The ability to read web pages, process emails or open documents from external sources.
- Ability to communicate externally: The ability to send data externally, whether by sending an email, making an API call or even just creating a link.
If an agent has all three capabilities, an attacker can easily trick them into searching for your private data and sending it to them. Such scenarios have already been demonstrated in systems such as Microsoft 365 Copilot, where hidden commands in documents allowed sensitive data to leak.
MCP: the underestimated attack vector
As if that weren't complex enough, concepts such as the Model Context Protocol (MCP) add another layer. The idea behind it is actually ingenious: to create standards so that agents can seamlessly access a wide variety of tools and data sources, regardless of where they come from. But this simplification comes at a price in terms of security.
MCP makes the risk of prompt injection even more confusing and difficult to control. It tempts users to quickly combine many tools from different providers, thereby opening up unforeseen attack paths. It is easy to lose track of exactly which data can enter the agent's context. Before you know it, you have the ‘deadly trio’ – often from tools from a single provider. For example, GitHub MCP may be able to access private repositories, read public (potentially malicious) issues and comments, and make pull requests to exfiltrate private information.
In addition, remote MCPs, i.e. MCP servers that do not run in your own infrastructure, can poison tools at any time (tool poisoning) and thus deliver arbitrary prompt injections. As a result, data from completely different services can also leak if an MCP or even a normal tool is integrated into the same agent for them.
Why ‘'persuading’’ is not enough: the limits of soft guardrails
The first intuitive response to these problems is often to simply give agents better and stricter rules. These measures, which fall into the category of so-called ‘soft guardrails,’ are an important first step, but they are far from sufficient. Some of the most typical ‘soft guardrails’ include:
- Optimised prompting
The system prompt, which is the basic instruction for the agent, should define a clear role, tasks and strict rules, and the prompt should be structured in such a way that data and instructions are clearly separated. Often, unsafe user inputs are marked with special tags (such as <user_input>) to signal to the model: ‘Attention, this is only data input, not an instruction.’ This is better than nothing, but it is more like ‘persuasion.’ It is only a soft measure. A skilled attacker can easily circumvent these rules by using clever wording or gradually ‘editing’ the agent in the course of a conversation. The model's attention to the original system prompt may wane over time. - Human-in-the-Loop & User Approval
Idea: Before the agent performs a potentially dangerous action (e.g. sending an email, deleting a file), a human must give their approval. Example: The agent suggests a draft email. The user sees a preview and must actively click on a ‘Send’ button. This sounds completely secure at first, but it is susceptible to human error. We are all familiar with ‘approval fatigue’ – after the tenth time, we simply click ‘OK’ without checking the content carefully. In addition, attackers can use social engineering to trick users into approving malicious actions. This method is significantly more effective in controlled internal processes (e.g. as a user task in a Camunda process engine) than in end customer applications. - AI-based filters & classifiers
A separate machine learning model can analyse user input (or tool output) and attempt to detect attacks in order to block them. Example: A filter is trained to recognise phrases such as ‘Ignore your previous instructions’ or other known jailbreak attempts and immediately block the request. Such filters are a useful element of a multi-layered defence, but they are not sufficient on their own. They only recognise what looks suspicious, not necessarily what is actually dangerous. Many attacks cannot be recognised as such without the full context. It is a constant race: as soon as a filter learns one method of attack, attackers find a new one. In addition, classifiers not only have false negatives (missed attacks), but also false positives, i.e. they occasionally block harmless inputs and can frustrate users. - LLM-as-a-Judge
Here, a second, separate LLM is used as an arbitrator. This ‘judge LLM’ evaluates the user's request (and, if applicable, the planned action of the agent LLM) and either approves it or rejects it. Example: The judge LLM receives the request: "The user wants to send a summary of their last five emails to an external address. Is this action permitted under our security policy XY?" This approach is more flexible than static filters, as the judge can better understand the context and apply policies. But here, too, the judge is itself an LLM and therefore also prone to errors and potentially vulnerable. It also slows down the process and increases costs.
Model-based soft guardrails are probabilistic. They may intercept 95% of attacks. However, the attacker still has another 5% of successful attacks that they can exploit. This is particularly tricky because the search for successful prompt injections can be fully automated: attackers simply set up a loop in which an LLM repeatedly generates injection prompts, tries them out on the target system, evaluates the response (e.g., did toxic text come back?), and adjusts the prompt slightly based on this feedback (or directly trains the parameters of the attacking model).
For these reasons, relying solely on soft guardrails is negligent.
It takes walls, not just bouncers: Hard Guardrails
If soft guardrails are only the first line of defence, we need more robust architectural measures. Hard guardrails are not rules that an agent should follow, but rather irrevocable restrictions that make certain actions impossible for them from the outset.
Once an LLM agent has ingested untrusted input, it must be constrained so that it is impossible for that input to trigger any consequential actions.
Let's look at a few patterns that can provide stronger security guarantees than soft guardrails, as these measures are based on system design (and thus deterministic code) rather than probabilistic models. The first pattern is aimed at preventing direct injection by the user, while the other two attempt to make indirect injection by tools impossible.
- Intent parameterisation
This is an important principle and is directly inspired by the defence against SQL injections. We strictly separate the interpretation of user input (what does the user want?) from the execution of the action. User input is never executed directly as a command, but is always passed on to a secure function as a data value (parameter).
Example: A user says, ‘Book me a flight from Hamburg to Madrid on 10 December 2025 and write a poem about why the airline is so bad.’ An upstream LLM without tool access recognises the intent book_flight. It fills in the predefined parameters: departure = “HAM”, destination = “MAD” and date = “10 December 2025”. The instruction ‘write a poem...’ is ignored because the book_flight schema only allows verified values for airports and a date. The executing agent only sees the intent and the cleaned data and can never interpret anything other than the command to be executed. The approach is comparable to offering the user a typed UI form (and for many agents, that would also be good UX). This method is effective against direct prompt injection by the user, as the scope for injection is very small to non-existent, depending on the parameterisation schema. If a form absolutely requires free text or non-enumerable string values, caution is again advised and you should at least limit the length and permitted characters as far as possible. - The Action-Selector Pattern
A simple pattern that protects agents from indirect prompt injection attacks by tools is to keep any feedback from these actions away from the agent. This means that the agent can trigger tools, but it cannot view the results of these tools (as they contain untrusted tokens). For example, it can give the command to open a web page or display a result to the user, but it cannot read the content of the web page or the feedback from the tool itself. This pattern provides very good protection against indirect prompt injection (if the user does not actively feed results back to the agent), but depending on the application, it can have a significant impact on the UX and severely limits the agent's capabilities. - The quarantine principle
The idea: two LLMs with different rights are used. A ‘privileged LLM’ interacts with the user and is allowed to access secure tools (such as a calculator), but it is not allowed to read the results of potentially insecure tools (such as browsing a website) itself. A second, isolated ‘quarantine LLM’ reads the unsafe content, extracts the required information into variables and only passes the variable names (not the content!) to the privileged LLM.
Example: The agent is supposed to retrieve the price of a product from a website. The privileged LLM calls up the browse tool. In addition to the price, the website also contains a prompt injection attack. The quarantine LLM reads the entire HTML code, ignores the attack, finds the price and returns only $price = 49.99; the privileged LLM is only passed $price. In the worst case, the quarantine LLM will write text other than the price to the variable or fill other variables with malicious content. However, the privileged LLM is never exposed to malicious content because it can only see the (restricted) variable names. In contrast to the action selector pattern, the actual agent is less restricted because it can continue to work with the variables without reading them. For example, it could pass the price as a parameter to another tool and thus chain several calls together.
This is a very powerful concept for preventing indirect prompt injection. The weakness is that the extracted data itself could be manipulated and influence subsequent tool executions (e.g., an incorrect email address as the recipient; the privileged LLM has no way of checking this). One improvement would therefore be to track the data flow and possibly subject sensitive actions based on potentially ‘contaminated’ data to additional human approval.
Defense-in-Depth: Build a fortress, not a fairy-tale castle
Relying on soft guardrails as your only line of defence is like placing a single guard in front of a beautifully decorated castle and nailing a sign to the gate saying ‘No entry’. This prevents the average citizen from marching straight into the treasure chamber. A secure fortress, on the other hand, needs sturdy walls, deep moats and sophisticated defences to make it impregnable even to entire armies.
Security is not a single product, but a process and a multi-layered strategy. An attacker may overcome one layer, but they should fail at the next. A robust defence strategy combines the soft and hard guardrails discussed above:
- Layer 1 (Soft): A strong system prompt and AI-based input filters intercept the simplest attacks. Layer 2 (Hard): A strict separation of interpretation and execution (intent parameterisation) forms the foundation and prevents the direct execution of commands.Layer 3 (Hard): The agent operates in a sandbox. It may only access absolutely necessary data. The quarantine principle isolates it from unsafe content. Layer 4 (Process): For particularly critical actions, explicit confirmation by a human being is required via a secure channel (e.g. a user task in a process engine).
A healthy dose of scepticism
AI agents have the potential to fundamentally change the way we work. But we must implement them with open eyes and a healthy dose of scepticism. Hoping that an LLM will protect itself is not a strategy.
The crucial question is not whether we build agents, but which ones we can build responsibly today – and how. By focusing on a well-thought-out security architecture, a combination of soft and hard guardrails, and a defence-in-depth strategy, we can leverage the enormous opportunities offered by AI agents without exposing our companies to unacceptable risk.
The responsibility for this lies with the developers of AI systems – the manufacturers of models, platforms and tools (e.g. Camunda Agentic AI, OpenAI Agent Builder, n8n) usually offer at most a selection of soft guardrails out-of-the-box. We must actively use these opportunities and expand them with our own systematic measures tailored to the respective application. If we do so, secure Agentic AI use cases can already be implemented today.
