When Your Reranker Is Worse Than Random

Meridian's retrieval pipeline had a strong BM25 baseline: Recall@5 of 0.987 on PubMedQA dev. Then I added a cross-encoder reranker — the stage that's supposed to add precision — and Recall@5 fell to 0.029. Random ordering of 100 candidates would score about 0.05. The reranker was not just useless; it was anti-correlated with relevance.

Measuring instead of guessing

The tempting move is to fiddle with hyperparameters. The correct move is to find where the behavior diverges between training and evaluation:

• On training pairs, the model did separate positives from negatives — the loss curves looked healthy.
• On dev queries, the gold passage fell from BM25 rank 1 to a median rank of 54 after reranking.

That gap is the whole diagnosis: with only 590 training examples, the cross-encoder had memorized its training pairs and learned nothing transferable. Healthy training metrics plus inverted eval behavior is the signature of overfitting, and the median-rank statistic made it undeniable.

The fix: never let a weak stage overrule a strong one

Two changes shipped:

1. Reciprocal-rank fusion with the base ranking. Instead of letting the reranker reorder candidates outright, its ranking is fused with BM25's. A good reranker still improves the order; a broken one can't scramble it. Recall@5 recovered to 0.983.
2. Tie-breaking by base rank rather than an arbitrary document-ID sort, so equal fusion scores preserve the stronger signal.

There was also a latency verdict waiting: reranking 100 candidates costs 421 ms P50 against BM25's 1.33 ms — roughly 320×. A stage that costs 320× and hurts quality on this corpus earns its default: disabled, documented, measured.

The takeaway

Any pipeline stage that can only help when it's good must degrade gracefully when it isn't. Fusion turns a reranker from a single point of failure into a bounded contribution — and a median-rank statistic beats an accuracy number when you need to know how something failed, not just that it did.