More engineering, less magic: How Agentic AI is finding its way into production

If we are honest, 2023 and large parts of 2024 and 2025 were the years of demos. We have all experienced that ‘aha’ moment when an AI agent solves a complex task seemingly effortlessly. It feels like magic. But when you try to bring that magic into an enterprise environment – into processes that need to be auditable, reliable and scalable – the magic often evaporates faster than you'd like.

In my last post, I talked about security – about prompt injection and data exfiltration. But even if an agent is completely secure, that doesn't mean it's useful or reliable.

At the end of 2025, the model is often not the central problem when it comes to reliability. GPT-5.1, Claude 4.5 or Gemini 3 are smart enough. The problem is the lack of adequate engineering around the model, as well as the necessary systematic approach to development.

If you build an agent today and your strategy is, ‘We'll write a good prompt and hope for the best,’ then you're not building an enterprise system, you're building an experiment.

We've seen enough of those in recent years – that's not to say they aren't helpful from time to time, but we need to think one step further.

So what does it take to bring agentic AI from the playground to production? Three important aspects distinguish demos from reliable systems: evaluation, systematic prompt engineering, and architecture. Combined with the most important mechanism of all (even in the age of prompts and foundation models) – the data flywheel – they form a solid basis for production-ready agentic AI.

Why traditional software testing is not sufficient for agents

In traditional software development, we live in a deterministic world. Input A in function B always results in output C. If not, it's a bug. We write unit tests, we achieve 100% coverage, we go live.

Agents don't work quite that simply.

LLMs, and even more so agents, are probabilistic. They make decisions based on probabilities. They use tools, they have a ‘memory state,’ they make plans that can change in the middle of execution.

An agent that works today may fail tomorrow with exactly the same input because the sampling turned out differently or there was an update in the inference stack.

The “vibe check” problem

The biggest risk in current AI projects is relying on the ‘vibe check’. A developer writes or adjusts the prompt, manually tests five examples, nods contentedly (‘Looks good!’), and deploys.

What has long been a no-go in classic software development does not suddenly become acceptable again with agentic AI (or AI in general).

In the world of enterprise AI, we are seeing entirely new classes of errors:

1. Subtle logical errors: The agent sounds extremely convincing, but the facts or the decisions derived from them are incorrect.
2. Lack of alignment: The agent makes understandable decisions, but they do not comply with the company's specific business or compliance rules.
3. Infinite loops & costs: An agent gets caught in a loop and burns through API budget without delivering a result.
4. Tool misuse: The agent calls the correct API, but with parameters that are syntactically correct but semantically nonsensical.

Anyone with a background in machine learning or data science knows that if there is no explicit evaluation stack for an agent and AI components, you are flying blind. A change to the prompt that satisfactorily solves a specific problem can simultaneously ruin various other scenarios. That is why regression testing is, of course, also essential for AI – and it comes with a few special features.

Measuring what matters

A robust strategy does not consist of a single ‘test’, but rather of layers:

1. Unit Level (Deterministic): Before we even talk about AI: Do your tools work? Do your Pydantic schemas validate the outputs correctly? This is classic software testing.
2. Component Level (RAG & Tasks): When your agent searches for documents, how good is the retrieval? When it needs to extract data, how high is the precision/recall? Frameworks such as DeepEval or Ragas can help here.
3. End-to-End Agent Scenarios: Here, scenarios with expected results are defined that test the entire agent loop: "Given support ticket X and customer data Y, the expected result is: ticket classified as “high priority”, refund of over £50 initiated, customer informed by email."

Offline vs. online evaluation

Offline evaluation (offline = static/batch) is our regression suite. Before a new prompt version goes live, it runs against 200 historical cases (‘golden dataset’). We measure success rate, costs, latency and everything else that is important. Initially, such an evaluation provides reliable, more or less objective figures as to whether the system is ready for production at all. Does it work reliably enough and with the necessary quality? Does an AI agent even work for this use case, or have we bitten off more than we can chew? Here, we need figures in black and white with clearly defined thresholds for KPIs. Once these have been achieved, they must be continuously re-checked with every change.

But even that is not enough. In production, online evaluation (online = live/real-time) is required. You cannot read every chat. You need LLM-as-a-Judge systems that run in the background and evaluate conversations: Did the agent comply with the policy? Was it helpful?

If we combine this with hard business metrics (was the ticket really closed?), we finally have transparency.

Without observability – i.e. detailed logging of every thought step (chain of thought), every tool call and every state change – error analysis is not possible. You have to see the "trace", not just the input and output.

Prompts: saying goodbye to the "Magic Strings"

Prompts are free text with virtually unlimited degrees of freedom. The performance of the LLM depends largely on the structure and wording of the prompts, and at the same time, it is often a matter of guesswork and a lot of time-consuming, manual experimentation by ‘prompt engineers’ that determines what a prompt ultimately looks like. With luck, the manufacturer will provide a rough prompting guide for the respective model. Nevertheless, in the end, it sometimes depends on small formulations or additions whether the model performs as it should.

A prompt (if developed correctly) is always tailored to a very specific model and often cannot be transferred directly to other models without side effects.

In many companies, the code for LLM applications also looks scary: huge Python files in which f-strings are buried with hundreds of lines of prompt text. Somewhere, opaque variables are being reformatted.

Now let's imagine that our company is scaling the adoption of LLMs and AI agents. Hundreds of manually written prompts attached to different models, spread across various codebases.

These prompts must be modified regularly to improve the system and fix certain errors in production. Sooner or later, they must or should be completely migrated to a new model generation (typical lifetime in the API: only 2-3 years), from one provider to another, or from proprietary to open-weight.

If treated as before, prompts thus become massive technical debt.

To address these issues, we must instead treat prompts like code – or better yet, like compiled artefacts.

Centralisation: Prompts should not be hardcoded into the business logic. They belong in a registry. Each prompt has an ID and a version.

Specification instead of text: Automatic prompt optimisers (such as in DSPy) change the paradigm: Instead of spending hours fine-tuning the wording (‘Be nice and think step by step’), you define the signature (input -> output) of the AI component, the relevant metrics and a gold set. An optimiser then compiles the perfect prompt for the specific model.

Versioning: When GPT-6 comes out, your hand-crafted prompt for GPT-5 will probably no longer be optimal. With a systematic approach (prompt registry + eval suite + optimiser), you swap the model, run the optimiser, validate against your gold set, and go live with a new version. Thanks to the registry and versioning, a rollback is possible at any time.

The architecture rule is: separation of concerns.

• The workflow (BPMN) specifies when and where an AI decision is required.
• The prompt service provides the implementation (the specific text) based on the version and model.

Bounded autonomy: agents need guidelines

There is the idealised notion of a ‘fully autonomous agent’ that you can throw a problem at and it will somehow solve it. In an enterprise context, this is usually a bad idea.

We need bounded autonomy.

Workflow First, Agent Second

The most stable systems I can imagine reverse the relationship. They do not use agents to control the process. They use a process (e.g. in Camunda or similar orchestration tools) to control agents.

The framework (the orchestration) is deterministic (BPMN). AI functionality is used within tasks (classification, extraction). Within small, clearly defined sub-processes, a limited agent is allowed to ‘think’ and use tools, provided that this is really necessary for the process.

The boundaries are hard-coded: Which tools is it allowed to use? How much budget does it have? And above all: Where are the human-in-the-loop checkpoints? For reasons of reliability and governance, many decisions should ultimately be made by a human being. The agent only delivers or handles defined low-risk cases independently.

The goal is not to restrict AI, but to manage risk. Only then can agents evolve from a ‘nice gimmick’ to a business-critical tool.

It is also important to avoid forcing agents into processes for the sake of technology, but rather to be honest and see where they can really help. Demo use cases can often be broken down step by step so that they can be handled by a normal workflow with AI tasks – with major advantages in terms of reliability, governance and costs. Use cases that really benefit from agents are often exploratory in nature (research, analysis, etc.). Unstructured data alone does not justify the use of agents; it is the LLM that is needed.

The data flywheel: the engine that drives it all

Now we have evaluation, clean prompt management and process guidelines. But the system is still static.

The secret of the tech giants (and why their models are getting better and better) is the data flywheel.

An AI or agent system is never "finished" The first day in production is the worst day for the system. From that point on, it should learn.

How the Flywheel works:

1. Data collection: Every trace, every user feedback thumb, every correction made by employees in the human-in-the-loop process is stored.
2. Curation: We filter out the ‘hard cases’ from this data – the cases where the agent was uncertain or failed.
3. Enriching evaluation: These hard cases are immediately transferred to our ‘Golden Tasks’ test suite. This means that our tests become tougher and better with reality.
4. Optimising: We use this data to improve the prompts or even the model (via DSPy or fine-tuning).
5. Deploying: The new version goes live and generates even better data.

That is the difference between a project and a product. Without this cycle, you only accumulate errors. With this cycle, you can build a competitive advantage (data moat).

A roadmap for implementation

That sounds like a lot of work. And yes, it is. But we don't have to implement everything from day one. Here's a pragmatic suggestion:

Step 1: Instrumentation (ending blind flight)

Before you optimise anything: log everything. Inputs, outputs, tool calls, latencies. Build a dashboard. Know how often your agent actually fails.

Step 2: the first regression suite

Take 50 real examples from your logs and build an automated test. Use rule-based metrics where possible, LLM-as-a-Judge where necessary. If you change the prompt, run the test. That's the beginning of professionalism.

Step 3: Bounded Autonomy & Structure

It is acceptable to start with a large agent to prove the validity of the concept. However, then systematically move away from monolithic "super prompts": break down the process into workflows. Encapsulate the AI tasks. Introduce versioning for prompts.

Step 4: Closing the flywheel

Establish processes for systematically feeding user feedback back into development. Ideally, make re-optimising prompts part of the CI/CD cycle.

How good does it feel to adjust a parameter in a configuration on the evening of the GPT-6 release, initiate a pipeline, and make an informed decision based on hard data as to whether migration is worthwhile (and complete it directly with another pipeline)? Migration to a new model generation with two clicks on the evening of the release. Anyone standing here has made it and can confidently scale Agentic AI in their company.

Conclusion: the questions you need to ask

If you want to avoid continuing to build only demos, ask these questions:

1. How do we know that the AI agent is mature? (Evaluation)
2. How do we ensure that tomorrow's update doesn't ruin everything from yesterday? (Regression testing & versioning)
3. What happens in the worst case scenario? (Guardrails & bounded autonomy)
4. How does the system improve over time and avoid previous mistakes? (Data flywheel)

Secure agents are a must, reliable, operational agents are a bonus – but that is precisely where the money will be made in the coming years.