A vendor case study is a marketing document with a number in it. Read one well and it tells you whether an automation could work for you; read one badly and you buy a headline that was never yours to expect. The trick is to separate the mechanism — what the system did, which usually transfers — from the metric — a specific result in a specific context, which usually does not. Ask who published it, whose result it is, what baseline it beat, and what is conspicuously not claimed. Here is the checklist we use, the same one behind our own public-case-study teardowns at /public-case-studies.
Whose result is it, and who published it
A vendor page reports the vendor's client's result. That is evidence the thing can work; it is not proof it repeats for you, and it is not an audited figure — it is a claim on the page of the company selling the tool. The first move on any case study is to name the vendor, name the customer, and say out loud that you are reading a marketing asset. That is not cynicism; it is the correct weight to give a self-published number before you do anything else with it.
Mechanism versus metric
The transferable part of a case study is what moved and why. 'Messaging beat calling for routine transactional follow-up.' 'Reading every call instead of a sample found that half the logged complaints were not complaints.' Those mechanisms tend to hold across companies, because they describe how the work changed.
The non-transferable part is the exact figure, because it is bound to that company's baseline, market, volume, and margin. A 40% recovery rate or a 21× ROI is real for the store that reported it and close to meaningless for you until you rebuild it on your own inputs. Borrow the mechanism; leave the number where you found it.
The baseline question
Every metric is a comparison, and the comparison is usually hidden. '40% cart recovery' — against nothing, against email, against SMS? '$400k saved' — whose hours, and were they removed or just moved somewhere else? 'Time-to-hire cut in half' — from what, and measured how? A metric with no baseline is a number floating free, and a lift measured against doing nothing is a different animal from a lift measured against the process you already run. Reconstruct the baseline before you believe the improvement.
What is not claimed, and where it is from
Honest evidence names its limits, and the omissions tell you where to look. A page that reports capacity up but not accuracy, or ROI but not the baseline, is quietly showing you the weak spot. An anonymized end-client — a BPO's unnamed insurer, say — carries less weight than a named one. 'Date not stated,' a redirected source link, a metric that only appears on a video tile and not in the article body: each of these lowers how much a claim can bear. Absence is information; read it.
Then there is the context the number lives in. A US collections result runs under different recovery law than an Indian one; a gaming operator's KYC runs under a different regulator than a fintech's. A figure from another regulatory context is a mechanism you can borrow and a metric you cannot inherit — especially where DPDP, the RBI, or TRAI change what is even permitted. This is exactly how our teardown library treats third-party evidence: every metric attributed to the vendor who published it, never restated as an Orkivanta result.
Turn a case study into your own estimate
The only number worth acting on is the one you can defend. So take the mechanism from the case study, plug in your baseline, your volume, your margin, and your cost per unit of work, and produce a figure that is yours. If you can reconstruct it, you have a forecast you can stand behind and a way to check it after you buy. If you cannot — if the case study gives you a headline but no way to rebuild it on your inputs — then you have a story, not a projection, and a story is not something to spend on.