A support department that closes its own tickets

The situation

A regional wholesale distributor: building products and industrial supply, roughly 4,000 active trade accounts, eleven people in customer service split across order support and returns. Ticket volume ran about 1,150 a week across email, a web form and a chat widget, with heavy Monday and month-end peaks.

The team had already been sold an AI chatbot by their helpdesk vendor. It had been switched off after five months. The reason, in the support manager's words, was that "it answered the easy ones and then told the customer to email us anyway, so we got the same ticket twice with a paragraph of apology on top."

The actual constraint was not answering. It was doing. Sixty-eight percent of tickets required somebody to change something in a system: reissue a delivery, correct a ship-to address, credit a short shipment, open an RMA, change a quantity on an open sales order. No one was willing to let a model near any of that, and they were right not to be.

Week one: counting, not building

We sampled 2,000 consecutive resolved tickets and classified them by the action the agent ultimately took, not by the customer's subject line. That produced nineteen action types covering 94% of volume, ranked by annual hours and by blast radius.

The bottom row of that table is the important one. We told them up front that price and contract disputes would never leave human hands in this engagement, and that credit limit changes and account terminations were out of scope entirely. Naming what you will not automate on day one is what buys you the latitude to automate the rest.

What got built

The retrieval layer

Grounding sources, in priority order: the 40,000 resolved tickets already in the helpdesk (by far the richest source — it encodes how this company actually talks to trade customers), the product catalogue and its 190,000 SKUs, the freight and returns policy documents, and live reads from the ERP for order, shipment and account state. Every generated sentence carries its citations through to the reviewer's screen. Answers that cannot be grounded are not produced; the ticket abstains to a human.

The action layer

The model has no credentials to anything. It emits proposed actions — typed, schema-validated objects — and a separate execution service holds the ERP and helpdesk credentials. That service checks the proposal against policy and the current trust level for its action type, then either executes it or parks it.

{
  "type": "credit.short_shipment",
  "target": { "invoice": "INV-224871", "account": "TRD-3390" },
  "change": { "amount_cents": 18400, "lines": [{"sku":"HX-4410","short":4}] },
  "evidence": [
    "ticket:118422",
    "pod:1Z9V...:signed_qty=8",
    "invoice:INV-224871:qty=12",
    "policy:credits#short-shipment-under-250"
  ],
  "confidence": 0.96,
  "reversible": true, "reverse_with": "credit.void",
  "requires": "level>=3 OR approval"
}

Two consequences fall out of that shape and they are the whole reason we build it this way. First, a prompt injection in a customer email cannot reach the ERP — the worst it can do is produce a proposal that fails schema validation or gets rejected by a human. Second, every action is reversible by construction, because the reverse operation is declared alongside it.

The queue

Built as a dedicated review surface, keyboard-first: J/K to move, A to approve, R to reject with a one-key reason code, E to edit and approve. The proposal, the diff against current system state, and the cited evidence sit on one screen with no scrolling. Runs of identical low-risk proposals can be approved as a block.

Rejection reasons came from a fixed taxonomy of nine, and this turned out to be the single most valuable data structure in the build. "Wrong policy applied" and "right action, wrong amount" are completely different failures and demand completely different fixes. A queue that records only yes and no learns nothing.

The ledger

Append-only, hash-chained, one row per event: proposed, approved, rejected, executed, reversed. Each row carries the actor (agent version or the human's identity), the evidence bundle, the resulting system state, and the reverse handle. It is exportable to CSV and PDF for the annual financial audit, which is what got the CFO from tolerant to enthusiastic.

The rollout

Weeks 1–2 — shadow. The system proposed against live traffic and executed nothing. We compared its proposals with what the humans did. Initial agreement on read-only action types was 91%; on write actions it was 74%, which was worse than we expected and traced to two things: the model did not know that trade accounts on payment terms have different credit rules than card accounts, and it was reading the freight policy as though it applied to will-call orders. Both were retrieval problems, not model problems, and both were fixed by week three.

Weeks 3–5 — draft. Proposed replies appeared as pre-filled drafts inside the existing helpdesk. Agents edited and sent. Send-with-minor-or-no-edit rate crossed 80% in week four for tracking replies and proof-of-delivery, and those two graduated.

Weeks 5–11 — queue. Write actions started reaching the approval queue. Median time-to-decide settled at 8 seconds. The support manager's initial fear — that reviewing would be slower than just doing the work — did not survive contact with the data: doing a short-shipment credit by hand in the ERP took her team a median of 4 minutes 20 seconds.

Weeks 11–14 — graduation. Six action types cleared the level-3 gate (≥98% approval over ≥200 decisions): tracking replies, proof-of-delivery, address correction pre-pick, short-shipment credits under $250, RMA creation, and reorder-link replies. Each was given a 4-hour recall window, visible on a dashboard the team actually watched for the first fortnight and then stopped watching, which is the correct trajectory.

What broke

Where it landed

Headcount did not change, and that was the agreed goal going in. The eleven people stopped typing tracking numbers and moved onto the two things the business had never had capacity for: proactive outreach on late orders, and working the returns backlog that had been ageing for two years. We think that framing is also why the team cooperated with the rollout instead of quietly sabotaging it.

What it cost, and the shape of the maths

Fourteen weeks of build, plus inference running at roughly $0.11 per handled ticket at the volumes above — call it $6,600 a year in model spend on 1,150 tickets a week. Against 11 FTE in support, of which the measured recovered capacity was around 3.4 FTE-equivalent of time, the payback on the build sits inside the first year on capacity alone, before counting the resolution-time improvement.

Try the interface

The approval queue in this build is the same pattern as the browser demo on this site. Same proposal shape, same keyboard flow, same ladder promoting action types as they earn it.

Open the live approval queue →