An eval set you keep
Your queries, your judgments, your recall target. It outlives the engagement and it re-runs every time someone proposes a change to the index.
services · search & retrieval POV
Stop estimating. Measure your own corpus.
You are about to make a six-figure decision on somebody's estimate. The measurement is about a week of work, it costs a few hundred dollars in embeddings, and it replaces every vendor benchmark with your own numbers.
// the honest version
You can run this yourself.
The protocol is below, in full, and it is the same one I use. It is also in the talk, and the cost model behind it is open source. If your team has a week and someone who enjoys this, take it and go — that is a good outcome and I would rather you had the number than not.
Most teams don't have the week. That's the engagement.
// the protocol
Sample, index three ways, measure, check the slope.
- 01
Stratify the sample
By source, length, age, tenant. Not the first N documents — that measures your oldest, cleanest data and tells you nothing about the corpus you actually serve.
- 02
Index it three ways
Sparse (BM25), dense, and dense + quantized. Multi-vector too, if you're in that world. Real indexes on real hardware, not a notebook.
- 03
Record five things
Bytes on disk · wall-clock build time · embedding spend · p99 at your concurrency · recall@k with filters on. Filtered recall is the one that surprises people.
- 04
Run it at two sizes
20 GB and 100 GB. One measurement is a point; two is a slope. The slope is what survives extrapolation to 40× your sample.
- 05
Extrapolate the linear, trust the slope for the rest
Bytes per vector and embedding cost per GB are ratios and scale cleanly. Compaction, graph build time and recall decay do not — those you read off the slope.
Why 20–100 GB and not 1: a toy corpus tells you the ratios and lies about everything else. Compaction, HNSW build time, filtered recall and p99 under a real index are invisible below ~10 GB and brutal above it.
// what you get
Four artifacts, all yours.
Measured write amplification
Bytes at rest per byte of source, for each index shape, on your corpus — not the 6× a calculator assumed for you.
A three-year cost curve
Storage, writes and embedding across the index lifespan at your real re-index cadence, on a basis procurement will accept.
The vendor call, argued
Which store, and the number it wins on. Including the case where the answer is the database you already run.
// shape and price
Fixed fee, a few weeks, no standing retainer.
Scoped as a fixed-fee proof of value against a question you can state in a sentence — should we move off what we have, can we hit this recall target, what does this actually cost at 40× today's corpus. Priced from the standard rate card.
I work on your data under your terms, and the engagement ends with a system your team runs, not a document about one.