← Playbook

We gave an AI write access to a production ERP. Here is everything we did first.

Not a prompt. Eight structural controls, none of which depend on the model behaving well, and all of which you can build before writing a single line of agent code.

2026-07-10/7 min read/Levelbrook AI Practice

Every meaningful AI deployment eventually arrives at the same conversation: to be worth anything, this thing has to change something in a real system. And the person who owns that system reasonably wants to know what stops it doing something catastrophic.

"We wrote a careful prompt" is not an answer. Here is the actual list, in the order we build it.

1. The model has no credentials

The foundational one. The model cannot reach the ERP, the billing system or the database. It produces a proposal — a typed object — and a separate execution service holds every credential and decides what runs.

The security property this buys is worth stating plainly: a successful prompt injection produces a rejected proposal, not a production incident. An attacker who fully controls the model's input can, at worst, get a proposal into a queue where a human looks at it. That is a different universe from an agent holding an API key.

2. Actions are a closed, typed vocabulary

There is no generic "run this update". There is a finite list of action types — nineteen, in one deployment — each with a schema, validation rules, and a declared blast radius. Anything outside the vocabulary cannot be expressed, let alone executed.

This is also what makes accuracy measurable. "The AI is 96% accurate" is meaningless. "Short-shipment credits are approved 99.1% of the time over 1,200 decisions; address changes 94.2%" is a control panel.

3. Every action declares its reversal

An action type without a defined reverse operation does not ship. Not "we could probably undo that" — a named, tested, one-click reverse handle stored with the action. Reversibility is designed in, at the schema level, or it does not exist when you need it at 2am.

4. Values are recomputed, never trusted

The model decides that a credit is owed. Deterministic code computes how much, from structured data, and the proposal fails validation if the two disagree by more than a defined tolerance.

This one control eliminated the largest single error category we have measured. Language models are good at classification and bad at being the final authority on arithmetic. Do not ask them to be.

5. Blast radius sets the gate, not confidence

A high-confidence action with a large blast radius still requires approval. Some things stay behind a human permanently regardless of how accurate the system becomes: contract changes, credit limits, account termination, anything discretionary, anything with legal exposure.

Confidence tells you how likely it is to be right. Blast radius tells you what happens when it is not. The gate is set by the second one.

6. Structural rules that override the model entirely

Some things are not probabilistic judgements and should never be modelled as one. Hard rules, evaluated outside the model, that stop an action regardless of confidence:

  • More than two prior AI contacts on the same thread → human
  • Any date field without an explicit year → human
  • Address change to a location with no delivery history → human
  • Account flagged as contract, legal hold, or in collections → human
  • Amount above a per-action-type ceiling → human

Each of these came from a real failure. That is how the list should grow: incident, then rule.

7. Rate limits and circuit breakers on the actions themselves

Caps per action type per hour, and an automatic halt when the rejection rate on any type spikes above its baseline. The nightmare is not one wrong action — it is four hundred wrong actions in six minutes because something upstream changed shape. Rate limiting turns an unbounded event into a bounded one, and the circuit breaker means nobody has to be watching for it to stop.

8. The kill switch is per action type

Any authorised person can disable one action type immediately, from the queue interface, without engineering involvement and without affecting anything else. Tested during rollout, not discovered during an incident.

The all-or-nothing kill switch is what ends AI programmes. One bad incident, one binary lever, and nobody ever turns it back on. Granular disablement means an incident costs you one action type for a week instead of the entire investment.

What is deliberately not on this list

Prompt engineering. It matters — it is most of how you get quality — but it is not a control. A control is something that holds when the model is wrong, or manipulated, or has been quietly swapped for a new version. Every item above holds under all three.

The test for any AI safety measure Assume the model is adversarial, or simply broken, or has been replaced without your knowledge. Does the measure still hold? If it depends on the model behaving well, it is a preference, not a control.

Build the eight, then give it write access. In that order.

Keep reading

This is what we do all day.

Support automation, AI phone agents, n8n back-office work, and the engineering loop itself — always behind a gate you control.