If your AI vendor won't show you the rejections, walk.

There is one question that reliably tells you whether an AI vendor has run their product in a real company with real customers, and it takes ten seconds to ask.

Show me your failure taxonomy from a live deployment. The categories of error, with counts.

Not the accuracy number. The accuracy number is a marketing artefact measured on a benchmark they selected. The failure taxonomy is an operational artefact that only exists if somebody was watching.

Why it is the perfect question

It cannot be improvised. A convincing failure taxonomy requires hundreds of reviewed decisions, a rejection-reason taxonomy someone designed in advance, and months of watching. You cannot generate one in the meeting.

It is unflattering, so it is honest. Nobody puts this on a slide voluntarily. A vendor who produces it readily is telling you something about how they operate.

It predicts your experience. Their failure categories will be roughly your failure categories. It is the closest thing to a preview of month four that exists.

It tells you what they built. A vendor whose failures are all "the model was confused" has built a wrapper. A vendor whose failures are "right action, wrong amount — 19; missing account context — 14; missed escalation signal — 11" has built a system, and has fixed things.

What a real answer looks like

Categories, counts, and for each one the structural change made in response. Something like: the largest category was right-action-wrong-amount, so monetary values are now recomputed deterministically from system data and the proposal fails validation if the model's figure disagrees.

Note that the fix is not "we improved the prompt". Real operators fix failures with architecture, because architecture holds when the model changes and prompts do not.

The three bad answers

"We're at 96% accuracy." You asked what kind, they told you how many. Ask again.

"Every customer is different, we'd have to look at your data." True and irrelevant. The failure categories generalise enormously. This answer means they do not have one.

"Our system doesn't really make errors in that sense." End the meeting.

The follow-ups, if the first answer is good

1. How does a rejection get captured — free text, or a fixed taxonomy in one keystroke?
2. What did you change as a result of the largest category?
3. What is your abstention rate, and what triggers it?
4. Walk me through the worst incident you have had in production.

That last one is the real test. Everyone who has run this at scale has an incident. They will usually tell you about it with a certain rueful enthusiasm, because it is the thing they learned most from. Someone who claims never to have had one has either not deployed or was not watching, and both are disqualifying.