None of them are technical. All of them are answerable by anyone who has actually deployed this in a real company. Print them out and bring them to the demo.
The demo will be excellent. Demos are always excellent, because they are built from cases that work. These questions are about what happens on the other 6%, on a Tuesday, eight months from now.
1. In week four of production, what action does it take without a human clicking anything?
If the answer is a form of "it drafts" or "it suggests", the ROI is deferred to a phase that may never
arrive. Read-only pilots are where AI programmes go to die.
2. Where do the credentials live? Can the model reach my systems directly?
You want to hear that the model emits proposals and a separate execution layer holds the credentials.
If the model has API keys, a prompt injection in a customer email is a production incident waiting for a
date.
3. Show me a real proposal object.
It should contain the action type, the target, the exact change, the evidence, a confidence value, and how
to reverse it. If they cannot show you one, actions are being taken as side effects and there is nothing
to review.
4. What is your abstention rate, and what triggers it?
A system that always answers is a system that will be confidently wrong on a schedule. You want explicit
triggers: thin retrieval, conflicting sources, high stakes, angry customer, novel case.
5. Show me the calibration curve on data like mine.
Stated confidence against measured accuracy. If it is flat, confidence is decorative and you cannot set a
threshold, which means you cannot safely automate anything.
6. Give me your failure taxonomy from a real deployment, with counts.
Not the accuracy number. The categories of error. Anyone who has run this in production has this list. If
they do not, nobody has been watching.
7. It just did the wrong thing to forty customers. Walk me through the next ten minutes.
You are listening for three specific things: an undo, a log that explains what happened, and a kill switch
that disables one action type without turning off the whole system. Vendors without a containment
story have not had an incident yet, which tells you how much production traffic they have seen.
8. Can I set the approval threshold per action type, myself, without a support ticket?
Your risk appetite is not uniform. Sending tracking information and cancelling a contract should never
share a setting.
9. How does an action type earn more autonomy? What is the published gate?
There should be a numeric threshold and an agreed process. "We'll monitor it and let you know" means
somebody's judgement, unlogged.
10. What is in the ledger and can I export it for an auditor?
Proposals, approvals, rejections, executions, reversals, with evidence and actor identity. CSV and PDF.
Self-service.
11. If I end this contract in eighteen months, what do I keep?
The single most revealing question on the list. You should keep your action taxonomy, your policy
configuration, your evaluation set, your accuracy history and your entire ledger — in portable
formats. If the answer is that your data is exportable but your configuration and evaluation history are
not, you are renting the part you paid to create.
A vendor with good answers may still not be right for you. A vendor who is annoyed by the questions has answered them for you already.
Support automation, AI phone agents, n8n back-office work, and the engineering loop itself — always behind a gate you control.