Orkivanta
← All posts
7 min read

How to read an AI automation case study honestly

A vendor case study is a marketing document with a number in it. Read one well and it tells you whether an automation could work for you. Read one badly and you buy a headline that was never yours to expect.

The trick is to separate the mechanism — what the system did, which usually transfers — from the metric — a specific result in a specific context, which usually does not. Ask who published it, whose result it is, what baseline it beat, and what is not claimed.

Whose result is it, and who published it

A vendor page reports the vendor's client's result. That is evidence the thing can work. It is not proof it repeats for you. And it is not an audited figure. It is a claim on the page of the company selling the tool. The first move is to name the vendor, name the customer, and say out loud that you are reading a marketing asset.

Mechanism versus metric

The transferable part of a case study is what moved and why. "Messaging beat calling for routine follow-up." "Reading every call instead of a sample found that half the logged complaints were not complaints." Those mechanisms tend to hold across companies.

The non-transferable part is the exact figure. It is bound to that company's baseline, market, volume, and margin. A 40% recovery rate is real for the store that reported it and meaningless for you until you rebuild it on your own inputs. Borrow the mechanism. Leave the number where you found it.

The baseline question

Every metric is a comparison, and the comparison is usually hidden. "40% cart recovery" — against nothing, email, or SMS? "$400k saved" — whose hours, and were they removed or moved? A metric with no baseline is a number floating free. Reconstruct the baseline before you believe the improvement.

What is not claimed, and where it is from

Honest evidence names its limits, and the omissions tell you where to look. A page that reports capacity up but not accuracy, or ROI but not the baseline, is showing you the weak spot. An anonymized end-client carries less weight than a named one. Absence is information. Read it.

Then there is the context the number lives in. A US collections result runs under different recovery law than an Indian one. A figure from another regulatory context is a mechanism you can borrow and a metric you cannot inherit.

Turn a case study into your own estimate

The only number worth acting on is the one you can defend. Take the mechanism from the case study. Plug in your baseline, your volume, your margin, and your cost per unit of work. If you can reconstruct it, you have a forecast you can stand behind. If you cannot, you have a story, not a projection.

Before you talk to anyone

Score your workflow first.

One number, already counted in your systems, that should move — and a switch to stop the thing if it misbehaves. Our readiness test checks exactly that, in a few minutes, with the result shown immediately.

Take the readiness test