How to read a number in an AI vendor’s case study
Ask four questions: what system did the figure come from, over what date range, computed by what method, and who can independently verify it. A claim that survives all four is evidence. A claim that survives none is an advertisement with a number in it.
The evidence ladder
- T0 Assertion — we said so. Worth nothing.
- T1 Artifact — the thing exists and you can look at it.
- T2 Architecture — the system is documented and inspectable.
- T3 Instrumented — measured before and after in the client’s own systems.
- T4 Attributed — a named client, on the record.
- T5 Verified — countersigned by the client as accurate.
The questions that separate the tiers
Most published case study numbers sit at T0 or T1 while being presented as though they sit at T4. The four questions above collapse that distinction quickly, and a vendor who cannot answer them usually stops claiming the number rather than defending it.
The tell nobody expects
Ask for the failure metrics. Escalation rate, correction rate, incorrect-action rate. A vendor who has genuinely measured can produce these in a minute, because they came from the same instrumentation as the wins. A vendor who cannot has not measured anything — they have collected impressions.