The step that should not be an agent

There is a specific way agentic automation projects go wrong, and it is not the one people expect. It is not that the model hallucinates a customer name or invents a policy. Those happen, and they are catchable. The common failure is quieter: a model is put in charge of a step that was never a judgement call, and the whole process inherits its variance, latency, and cost for no benefit at all.

A worked example

An invoice arrives. The process needs to decide which supplier it came from.

The tempting implementation is to hand the document to a model and ask. It works in testing. It works on most real invoices. It fails, occasionally, in ways that are hard to predict — a subsidiary name, a shared address, a trading-as line — and each failure costs a reconciliation.

The better implementation is a lookup. Extract the VAT number, the bank account, and the supplier name; match against your supplier table with exact match first, then a normalised string match; produce a confidence score; and only involve a model when all of that fails to resolve. Now the common path is deterministic, instant, free, and identical every time, and the model handles the residue — which is exactly the shape of problem models are good at.

The general rule: use a model where the input space is genuinely open and the output requires judgement. Everything else is a rule you have not written down yet.

How to tell which steps are which

Ask three questions about the step.

Could a competent new hire do it correctly from a written instruction, without asking anybody? If yes, it is a rule. Write the rule. The fact that nobody has written it down is a documentation problem, not a reason to reach for inference.

Does the same input always have the same correct output? Classification of free-text intent: no, genuinely ambiguous. Deciding whether an invoice total matches the sum of its lines: yes, always. The second one must never be a model call, because a model can be wrong about arithmetic and a comparison cannot.

What does being wrong cost, and would you notice? A wrongly categorised support ticket gets re-routed by a human in ten seconds. A wrongly matched purchase order posts money to the wrong supplier. High-consequence steps want determinism, or a gate, or both.

The architecture that follows

Once the split is made, the shape of a reliable automation is fairly consistent:

• A deterministic spine: fetch, validate, look up, transform, write. Ordinary code, ordinary tests, ordinary error handling.
• Model calls as leaf nodes on that spine, each with a narrow job, a tool allow-list, and a schema its output must satisfy. If the output does not parse, that is a failure, not something to coerce.
• A gate before anything irreversible. Sending, paying, publishing, deleting. The gate can be a confidence threshold, a human approval, or both — but it is a structural element, not a setting somebody can turn off in a hurry.
• Idempotency everywhere external. Every write carries a key so a retry cannot double-post. This matters more with models in the loop, not less, because retries become more common.

Evaluation is what makes the model part safe

The deterministic spine is tested the way all code is tested. The model calls need something else: a fixed set of real cases with known-correct answers, run on every change to a prompt, a tool, or a model version, with a threshold that blocks release.

This sounds heavy. It is about fifty cases and an afternoon for most processes, and it converts the single riskiest property of the system — that a prompt edit on Thursday silently changes behaviour on Friday — into a number that either passes or does not.

Without it, you have no way to answer the only question that matters when you want to upgrade a model: did that make anything worse?

The uncomfortable version

A large amount of what is currently sold as AI automation is a scheduled job with good error handling, wrapped around two or three genuine judgement calls. That is not a criticism of the category — that shape is enormously useful, and the judgement calls are often the ones that used to require a person to read something.

But it does mean the honest version of an automation proposal usually includes a sentence like: most of this is a rules engine, and here are the four places where a model earns its keep. If a proposal you are reading does not contain that sentence, or something like it, it is worth asking which steps are models, and why each one has to be.