Decision Models as Rerankers

A guest talk for students of AI-Powered Search, Doug Turnbull and Trey Grainger’s Maven course, on Monday November 23rd, 2026.

What you’ll learn

What a decision model is, and why it fits reranking. You send typed questions and get a calibrated probability per answer. No text is generated, so calls are fast.

Four decision models on one benchmark. Jev, Perplexity, Clef and OpenAI against Cohere, Voyage and Mixedbread, all on the same BEIR shortlists.

Request shape matters as much as the model. Pair, batch or choice can take a model from beating BM25 to losing to it, or raise its bill nearly 30x.

Calibrated scores let you prune as well as sort. A probability threshold cuts weak context before your LLM pays for it, if the model’s probabilities hold up.

How to choose, and how to build your own. Weigh quality, price, tail latency and calibration, using an open wrapper and the full results.

Why this talk matters

Rerankers have been a boutique category, where a few vendors each sell one model and you get nothing to tune. A general-purpose decision model, asked one yes-or-no question per document, lands next to the purpose-built rerankers untuned, at a quarter of Cohere’s price. You write the question yourself, so you can tune the reranker to your own domain instead of buying one off the shelf.

What’s behind it

The numbers come from Hev meets Jev, Jev has company and OpenAI makes four, and from the reranker comparison, which has 51 systems on SciFact, NFCorpus and FiQA. Every reranker reorders the same BM25 top 30, which scores 0.404 nDCG@10 averaged over the three corpora.

• Jev, untuned, scores 0.501 for $0.54 per 1,000 queries, beside Voyage (0.504, $0.50) and above Cohere (0.486, $2.00). Mixedbread’s listwise model leads at 0.511.
• Perplexity, asked to choose among the 30, matches Jev at 0.501 for $0.40. Asked 30 separate questions in one call, it costs $11.59, because its usage receipts count the shared context once per question.
• Clef-flash reaches 0.498 scoring one document per call. Handed all 30 at once, it falls to 0.283, below the BM25 order it was supposed to improve.
• OpenAI’s Decisions API (public beta) trails at 0.472. Its probabilities are too sharp to prune with: the 0.1 threshold that works for Jev drops a quarter of the relevant documents.

The wrapper is open source at github.com/hev/reranker and on PyPI as hev-rerank. The lesson closes on what comes after one generic question per document, including Doug’s bag of decisions, which asks an LLM to write a rubric of questions for each query.