Reranking just got cheaper

In September, Mixedbread’s v3.1 listwise reranker was the best system in our benchmark and cost $1.42 per 1,000 queries at list price. Perplexity’s first decision model came close for $0.40. This month Perplexity’s pplx-decider-v1.2 caught up: it reranks a 30-document shortlist as well as Mixedbread v3.1 for $0.20 per 1,000 queries, from billed usage at Perplexity’s list price on October 24. Mixedbread has cut its rate too, to about $0.03 per million input tokens, which comes to roughly $0.27 per 1,000 queries by my count. I can’t say why Mixedbread cut its price. But in the month a general decision model matched the best specialist, the cost of top-quality reranking fell by about four fifths. If you buy reranking, you win either way.

Mixedbread’s new price is an estimate, not a receipt: I counted what v3.1 was sent with the Qwen3.5 tokenizer, at the rate Mixedbread gave us on October 24. We have no billed usage for v3.1, so the gap between the two new prices is small and approximate. On quality, over SciFact, NFCorpus and FiQA it averages 0.511 nDCG@10. The paired difference from Mixedbread is +0.0001, with a 95% interval from −0.005 to +0.005, so the two are statistically indistinguishable in this test. That is not proof they are equivalent. It also beats Jev batch by 0.010, and that interval excludes zero.

The short version:

• The best reranking got cheaper. The top score in our test cost $1.42 per 1,000 queries in September. Today it costs $0.20 from Perplexity, or about $0.27 from Mixedbread (estimated).
• It’s level with the top of the table. Asked to pick among the 30 documents, the 27B model scores 0.511, level with Mixedbread v3.1. Per corpus it wins SciFact, loses NFCorpus and ties FiQA, so neither system is better across the board. It costs $0.198 per 1,000 queries. Mixedbread’s is estimated at about $0.27 (see below).
• Everything else in the table scores lower. It’s ahead of Jev batch by 0.010, Jev pair by 0.009, Voyage rerank-3 by 0.006 (just) and Cohere v3.5 by 0.025, and each of those paired intervals clears zero.
• Flash matches Jev for a fifth of the price. The smaller pplx-decider-flash-v1.2 scores 0.500, statistically level with Jev batch, for $0.10 per 1,000 queries.
• It improved on v1. Paired query by query against October’s pplx-decider-v1-27b, the choice shape gains 0.009, and the price per token halved.
• Batch got cheap but not better. Thirty yes-or-no questions in one request fell from $11.59 to $0.25 per 1,000 queries. Quality didn’t move, and batch now trails choice by 0.013.
• The run is incomplete. Flash batch hit rate limits after 16 to 20 queries per corpus, and the 30-calls-per-query pair shape only ran on 10 queries per corpus. Neither appears in the comparisons below.

This is the fourth post in the series. In Hev meets Jev I used Jev, TypeSafe’s decision model, as a reranker. In Jev has company, Perplexity’s first decision model, pplx-decider-v1-27b, edged Jev by a hair, and OpenAI made four. Perplexity has since shipped v1.2 in two sizes: the 27B model and a smaller flash model, built on Qwen3.5-4B and distilled from the 27B models. I ran both through the same test, with the same shortlists and the same prompt, and no tuning. (I skipped v1.1, which came out in between.)

The test

Nothing changed. Every system reranks the same BM25 top 30 for SciFact (300 queries), NFCorpus (323) and a fixed 300-query sample of FiQA. The score is nDCG@10, averaged with equal weight per corpus, and differences are paired bootstrap intervals over queries. The models are asked in the same three ways as before:

The prompt is the one Clef, v1 and OpenAI used, picked on 20 SciFact training queries before any of those test runs. For v1.2 I ran one training query per model and shape to check the requests worked, then went straight to the test sets. Nothing about the prompt or the choice of shapes was changed after seeing test scores.

One thing did change on Perplexity’s side. Its API now sends requests for pplx-decider-v1-27b to v1.2 and bills them at the v1.2 price. The v1 numbers in this post come from the October 3–4 receipts. Anyone who reruns that model id today gets v1.2.

Quality against price

*Estimate, explained below.

Prices are each run’s recorded usage at list rates, normalized to 1,000 queries, before credits or discounts, except Mixedbread’s, which is estimated. The Perplexity rows use today’s rates: $0.02 per million input tokens for the 27B model and $0.01 for flash, with output free, down from the $0.04 that v1 cost in October. Every other row carries over from the earlier posts at the list rate when it ran. Mixedbread’s figure is an estimate. We have no billed usage for it. Mixedbread told us on October 24 that it now charges about $0.03 per million input tokens and suggested counting with the Qwen3.5 tokenizer, so I counted exactly what each request sent (the query once, then the 30 documents as title and text) with that tokenizer: about 10,300 to 10,400 tokens per query on SciFact and NFCorpus and 6,500 on FiQA, or $0.27 per 1,000 queries averaged over the three. That leaves out whatever Mixedbread’s server wraps around the documents, which I can’t see. Each extra token per document would add about $0.001 per 1,000 queries, and counting the query once per document instead adds $0.01. Choice requests are the same size they were in October, within a few tokens, so the drop from $0.40 to $0.20 is the price cut alone.

Above a fixed quality line, most of what’s left is price. Seven full runs score above 0.5 mean nDCG@10, and they cost between $0.10 and $0.87 per 1,000 queries.

Against Mixedbread v3.1, 27B choice wins SciFact by 0.012 and loses NFCorpus by 0.009, and both of those intervals exclude zero. On FiQA and on the three-corpus mean they are statistically indistinguishable. Against Jev batch it gains 0.020 on FiQA, ties on NFCorpus, and is ahead on SciFact by 0.011 with an interval that crosses zero. Its lead over Voyage rerank-3 on the mean is 0.006, with an interval from +0.0004 to +0.013, which is about as thin as a lead can be and still count.

Flash choice doesn’t clearly separate from Jev batch, Jev pair, Clef-flash pair or v1 choice on any corpus or on the mean. It trails the 27B chooser by 0.010, mostly on FiQA, and Mixedbread by the same amount. For a much smaller model (Perplexity puts it at about a seventh the size), at half the per-token price, that’s a good place to land.

What changed from v1

Choice got better and batch didn’t. Against v1, 27B choice gains 0.011 on NFCorpus and 0.014 on FiQA, and both intervals exclude zero. 27B batch moves by less than 0.01 on every corpus, in both directions, and none of those changes clears its interval.

That matters because of how v1.2 reads a request. The 27B model card says the model reads “the state, then every question” as one sequence, and that “every question attends to the state and to every other question.” It adds that grouping “helps when the questions are facets of one decision, and hurts when they are independent candidates.” Reranking is the second case. Thirty yes-or-no questions about thirty different documents are thirty candidates competing for the same query.

The numbers are consistent with that. In October, batch trailed choice by 0.002. Now it trails by 0.013 (interval −0.019 to −0.006). Measured query by query, that gap widened by 0.011, with an interval from +0.004 to +0.018, and most of it is on NFCorpus (+0.019), where a BM25 shortlist holds 3.8 relevant documents on average, against about one on SciFact and FiQA. But v1.2 also changed its training data and model weights, so this doesn’t isolate the joint reading as the cause. The clean test would compare batch with pair on the same model, and the pair run only covered 10 queries per corpus. On those 10, batch and pair don’t separate. The flash card is more direct: “one question per request is the safer default.”

What a request is billed for

The batch price is the bigger change. In October, Perplexity’s usage receipts counted the 30 shared documents once per question: a batch request over SciFact billed about 327,000 input tokens, against about 11,000 for a choice request over the same documents. That is why v1 batch cost $11.59 per 1,000 queries. The 27B v1.2 model now bills about 13,600 tokens for the same request, a little more than choice, which puts batch at $0.25.

Flash batch didn’t get that change. Its requests averaged 216,000 to 321,000 input tokens per corpus, close to v1’s figures. It was also the shape that hit Perplexity’s rate limit (too_many_requests) on every corpus, after 16 to 20 queries. I can’t tell from the error whether the limit was on tokens or on requests, so I’m not reading anything into it. If you want flash, use choice.

Latency

The v1.2 runs came from a Linux server at concurrency 4, with at most five requests started per second, timed from sending each request to parsing the response:

These are averages of the three corpus medians and 95th percentiles, over successful requests only. The earlier posts timed their runs from a laptop and from a Mac mini, on other days and at other concurrency, so I’m not ranking v1.2 against those numbers. Within this run, flash choice was the fastest one-call shape and flash batch the slowest by far, which fits a flash batch request billing more than twenty times the tokens of 27B batch.

Are rerankers dead?

Not from this result. What it does is take away the quality reason to pay a specialist for one job: putting a 30-document shortlist in order. On these three corpora, a general decision model asked a single question does that job as well as the best reranker we’ve run, at $0.20 per 1,000 queries. If ordering a short list is all you use a reranker for, that’s worth a test on your own data.

A reranker does more than order a list, though, and none of the rest was tested here:

• Independent scores. Choice spreads one unit of probability across the 30 documents, so it tells you which document is likeliest, not whether any of them is good. A reranker returns a relevance score for each document, and Mixedbread says v3.1’s scores depend on the content so they can set threshold cutoffs. That is what lets you drop the weak ones, set a threshold, or notice that nothing in the list answers the question. This test didn’t measure how well v3.1’s thresholds work. The first post pruned Jev’s candidates at a probability of 0.1 with little loss. Choice can’t do that. 27B batch gives per-document probabilities and is now cheap, but it scores 0.013 lower, and I haven’t checked whether its probabilities support the same prune.
• Depth. Thirty documents fit in one request. A reranker scores a hundred or a thousand candidates in parallel calls. How choice behaves on a much longer list, where the documents may not fit or their position starts to matter, is a different test.
• Latency. Mixedbread answered in 217 ms at the median in September, timed from a laptop. Perplexity choice answered in 479 ms here, timed from a different machine on a different day. That’s not a fair comparison, and it isn’t one I’m making, but a tight latency budget is its own requirement and this run doesn’t settle it.
• Other data. These are three public English benchmarks that may be in any model’s training data, with short documents. Private, multilingual or long-document collections could go either way.
• Price. List prices move. Perplexity’s halved in a month, Mixedbread’s fell from $1.42 to an estimated $0.27 per 1,000 queries, and that new figure is an estimate from token counts, not a bill.

So the case for a dedicated reranker now rests on what it does besides ordering: calibrated per-document scores, long candidate lists and tight latency. For ordering a short list, Perplexity has caught the best of them, and the best of them has dropped its price. Whichever you pick, you now pay about a fifth of what top-quality reranking cost in September.

Limits

These are public benchmarks that may be in any model’s training data. The shortlist is 30 deep. The intervals cover query sampling and aren’t corrected for multiple comparisons. The flash batch and pair rows are too small to say anything about. Candidate-order checks covered only five SciFact queries per model and shape, too few to report. And I can’t check which weights the API served beyond the model id it returned, which matched the requested id on every successful response.

The model cards are on Hugging Face for the 27B model and flash, and pricing is in Perplexity’s API docs. The cards report Perplexity’s own Decision Index results. Those are a different test from this one, and none of their numbers appear here. Every number in this post is in the measurement snapshot.