cross-encoder-reranking
verified8993556e-a0af-496e-a619-d1bf27d46b6c
Lift RAG/search precision with retrieve-then-rerank — bi-encoder first stage, cross-encoder second stage, rerank budgets, and measuring lift.
Metadata
Skill file
# Cross-Encoder Reranking for Better Retrieval
Use when first-stage retrieval (BM25 or dense embeddings) returns the *right-ish*
candidates but the *top* results aren't the truly relevant ones — a very common RAG
symptom. Reranking with a cross-encoder reorders candidates so the most relevant
land in the top few slots the LLM actually sees.
## The two-stage pattern
Bi-encoders (your dense retrieval) encode query and document **separately** and score
by vector similarity — fast enough to run over millions of vectors, but they miss
subtle query–document interactions. A **cross-encoder** runs the query and each
candidate *together* through full cross-attention, producing a precise relevance
score for each pair — far more accurate, but O(candidates) more expensive.
So: **retrieve wide, rerank deep.**
```
query
│
▼ stage 1: bi-encoder (+ BM25 hybrid) over the full corpus (recall)
top 50–100 candidates
│
▼ stage 2: cross-encoder scores each (query, doc) pair (precision)
top 3–5 passages
│
▼ LLM context
```
Stage 1 optimizes **recall** (don't miss good docs); stage 2 optimizes **precision**
(put the good ones on top). This cascade is the standard for high-accuracy RAG/IR.
## Choosing the rerank budget (top-N to rerank)
- Rerank **50–100** candidates as the sweet spot for most RAG. More → higher ceiling
but linearly slower; cross-encoders don't batch elegantly at huge N.
- Only the final top **3–5** go into the prompt (fits context, focuses the model).
- Total added latency is manageable if stage 1 is fast; the pipelines above stay
under ~300 ms end to end at these budgets.
## Cross-encoder model choices
- Off-the-shelf sentence-transformers rerankers (e.g. `BAAI/bge-reranker-*`,
`cross-encoder/ms-marco-*`) — good general-purpose baseline.
- Hosted rerankers (e.g. Cohere Rerank, Jina Reranker): predictable latency, zero
infra, but per-call cost and data goes to the provider.
- **Self-host when** you need data residency, domain fine-tuning, or flat cost at
scale — you can fine-tune a cross-encoder on your own (query, relevant-doc,
non-relevant-doc) triples to lift domain precision further.
- 2025+ landscape shows single-vector models and ColPali-style multi-vector
rerankers; pick the one that wins on *your* eval, not the leaderboard.
## Measure the lift (or skip the feature)
Reranking is not automatically better — add it only if it measurably helps:
- Build a **labeled eval set** (queries → relevant docs).
- Compare **recall@k / nDCG@k / faithfulness** with and without reranking.
- Watch **latency**: adding a cross-encoder that costs 200 ms but only shifts
noise around top-1 might not be worth it for a streaming chat.
- Guard against over-engineering: if first-stage retrieval is already clean and
recall is high, reranking may add latency for ~0 precision gain. Measure first.
## Wiring it in (pseudo-code)
```python
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("BAAI/bge-reranker-base") # local example
candidates = first_stage_retrieve(query, top_n=100) # bi-encoder / BM25
pairs = [(query, doc.text) for doc in candidates]
scores = reranker.predict(pairs) # one score per pair
rescored = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
final = [doc for doc, _ in rescored[:5]] # top 5 to LLM
```
## Pitfalls
- **Reranking noise:** if there's little real signal in the candidates, the
cross-encoder reorders near-equal documents — cost without benefit.
- **Latency blowup:** reranking 1000+ candidates kills p99. Cap the budget.
- **Score incompatibility:** don't mix bi-encoder scores and cross-encoder scores
in one ranking without re-normalizing — use the cross-encoder's own rankings.
- **Domain gap:** a general reranker can fail on niche jargon; fine-tune or swap
models and re-measure on your eval.
- **Prompt truncation:** feed the *scored* top passages, not pre-rerank order, or
the whole effort is wasted.
## Verify
- A held-out query set shows higher recall@k / nDCG@k with reranking than without.
- p99 latency stays within your budget (rerank N measured end to end).
- The passages the LLM actually cites shift to correctly-relevant sources on a
qualitative spot-check.
- If using a hosted reranker, confirm provider/data-residency constraints are met.