Retrieval systems break in specific ways. A chunk size change, an embedding model update, a reranker threshold shift — any of these can silently degrade precision on a narrow slice of queries while leaving aggregate metrics untouched.
A retrieval regression probe is the instrument that catches that slice before it reaches production.
What a Retrieval Regression Probe Is
A retrieval regression probe is a fixed query set run against your retrieval layer before every deploy. Each query has an expected set of document IDs. The probe measures precision@k — the fraction of the top-k retrieved documents that match the expected set — and fails the deploy if precision drops below a defined threshold.
It is not a general benchmark. It is not a random sample. It is a curated set of queries that have already caused production failures, run deterministically, with a binary outcome: pass or fail.
The goal is narrow: confirm that known failure modes have not regressed.
Building a Minimal Probe
Start with 10–15 queries. That is enough to cover your highest-severity failure patterns without making the probe slow or expensive to maintain.
Step 1: Source queries from real failures. Pull from production logs. Look for queries where the system retrieved a plausible-sounding document that was factually wrong for that query, or where the correct document ranked outside the top k. These are your probe candidates. Do not invent synthetic queries — they will not reflect the distribution that actually breaks your system.
Step 2: Record expected document IDs. For each query, identify the document or documents that should appear in the top-k results. Store these as a list of expected IDs. A query can have more than one acceptable answer — encode that as a set, not a single ID.
Step 3: Define a noise band. Precision@k will fluctuate slightly across deploys even when nothing meaningful changes. Set a threshold that flags real drift without triggering on noise. A common starting point: fail if precision@5 drops more than 10 percentage points below the baseline established on the last known-good deploy. Adjust based on how stable your retrieval layer is in practice.
Step 4: Automate the run. The probe should execute as a pre-deploy step in your CI pipeline. If it fails, the deploy does not proceed. No manual override without a documented exception.
How a Wider Candidate Pool Changes Probe Behavior
Many retrieval architectures use a two-stage approach: a fast lexical or dense retrieval pass that generates a candidate pool, followed by a reranker that reorders the pool before returning the top k.
Widening the candidate pool — say, from 12 to 30 candidates — is a common tuning move. The intuition is sound: give the reranker more to work with, and it has a better chance of surfacing the right document.
But wider pools introduce edge cases that a probe will catch and aggregate metrics will not.
Here is what happens:
- Recall improves for common queries. The correct document enters the candidate pool more often. Reranker precision goes up on average.
- Noise increases for ambiguous queries. With 30 candidates instead of 12, the reranker sees more documents that are topically adjacent but factually wrong for the specific query. On ambiguous queries — the ones most likely to have caused production failures — reranker precision can drop.
- The probe catches the ambiguous cases. Because your probe queries are sourced from real failures, they skew toward the ambiguous end of your query distribution. A wider pool that improves average precision@5 by 4 points may simultaneously regress precision on 3 of your 12 probe queries. The aggregate metric hides this. The probe surfaces it.
This is the core value of a regression probe: it is adversarially constructed against your system's known weaknesses, not its average behavior.
What to Do When the Probe Fails
A probe failure is not a crisis. It is the system working correctly.
When a probe fails:
- Identify which queries regressed and by how much.
- Check whether the regression is in the candidate pool (retrieval stage) or in the reranker output (ranking stage). Log both the pre-rerank and post-rerank rankings for probe queries.
- Determine whether the regression is a side effect of the intended change or an unintended consequence.
- Either fix the regression before deploying, or document the trade-off explicitly and update the probe threshold with a rationale comment.
Do not skip step 4. The probe is only useful if its thresholds reflect deliberate decisions, not accumulated tolerance for drift.
The Operational Discipline
A retrieval regression probe is not a one-time build. It requires maintenance:
- Add new queries when new failure modes appear in production.
- Retire queries when the underlying documents are removed from the corpus.
- Review the expected ID sets when the corpus is restructured.
Fifteen queries, maintained honestly, will catch more regressions than a comprehensive benchmark run once at launch.
Boring, consistent, and pre-deploy. That is the pattern that keeps retrieval systems reliable.
If you are building or auditing an AI system and want to talk through retrieval architecture and evaluation design, Start a conversation →