Full re-embedding
Regenerate document embeddings and rebuild historical indexes whenever the model changes.
High migration costPROJECT 01 · DISTRIBUTED SYSTEMS
Drift-RAG:Retrieval across changing embedding spaces.
What happens when a production RAG system upgrades its embedding model, but its historical document vectors cannot be regenerated immediately? Drift-RAG investigates whether lightweight, query-side alignment can recover retrieval quality while keeping historical indexes intact.
01 / MOTIVATION
The problem: embedding drift
A retrieval system encodes documents and queries into vector representations. When the embedding model changes, new query vectors may no longer align with document vectors produced by the older model. Searching the historical index with those incompatible vectors can severely degrade retrieval quality.
The conventional alternative is to re-embed the historical corpus using the new model. For a large, continuously growing collection, that operation can be expensive and operationally disruptive.
Query-side compensation
Learn a transformation from a small calibration set and adapt new query vectors before searching each historical index.
The approach we tested02 / SYSTEM DESIGN
A router and three autonomous shards
The live system runs across five VMs. Each historical-generation shard owns its adapter and FAISS index; the router encodes the query and fuses the ranked results.
WORKER 1 · ROUTER
Query service
Encode with BGE-M3 · Dispatch · Fuse results
Raw V3 query embedding
Ranked results returned to router for fusion
WORKER 5 · ORCHESTRATOR
Training and evaluation
Calibration · Adapter training · Index experiments · Benchmarks
Offline pipeline
Spark ingestion and cleaning, chronological corpus partitioning, embedding generation, calibration, adapter training, and FAISS index construction.
Live retrieval
FastAPI services communicate over HTTP. Each shard applies its own transformation locally, searches its index, and returns ranked results for router-side fusion.
03 / COMPENSATION METHODS
How do we align the embeddings?
Three adapter designs test whether a simple global transformation is sufficient, or whether local corrections provide additional value.
L0 · Translation
BaselineShifts each query by a global mean-difference vector. It was inexpensive, but retrieval quality remained close to zero.
L1 · Orthogonal rotation
Best overallLearns a global rotation from paired calibration embeddings using orthogonal Procrustes alignment.
L2 · Rotation + local correction
No improvementAdds cluster-specific residual corrections to L1. It did not improve retrieval quality in these experiments.
ORTHOGONAL PROCRUSTES · L1
R* = arg minR ‖XR − Y‖F2subject to RᵀR = I
Here, X and Y are paired calibration embeddings from the current and historical models. The orthogonality constraint restricts the learned transformation to a rotation or reflection, preserving vector lengths and inner products within the transformed space.
04 / EXPERIMENTAL EVALUATION
Recovering retrieval quality
The main CC-News evaluation compares unadapted queries, compensated queries, and native-model retrieval on the same historical-generation task.
V3 → V1
Oracle NDCG@10 recovered
V3 → V2
Oracle NDCG@10 recovered
| Transition | Naive | L1 compensated | Oracle | Recovery |
|---|---|---|---|---|
| V3 → V1 | 0.000 | 0.772 | 0.791 | 97.6% |
| V3 → V2 | 0.001 | 0.810 | 0.866 | 93.5% |
Recovery is calculated as compensated NDCG@10 divided by oracle NDCG@10. Values are rounded; percentages use the underlying experiment values.
Does the pattern hold on another dataset?
On NFCorpus, a BEIR benchmark with human relevance judgments, compensation also substantially improved retrieval: L1 reached approximately 0.276 NDCG@10 against an oracle of 0.359 for V3 → V1, and 0.280 against 0.400 for V3 → V2.
The recovery percentages from NFCorpus and CC-News should not be directly compared. CC-News uses title-to-body self-retrieval, while NFCorpus uses graded human relevance judgments.
WHAT DID NOT WORK?
Translation alone failed
L0 remained near zero NDCG@10 on the CC-News historical indexes. A global mean shift was insufficient for this embedding-model transition.
More complexity did not help
L2 slightly underperformed L1. Cluster-specific residual corrections did not improve on the global rotation in these experiments.
05 / ENGINEERING
Quality is only part of the system
Parallel shard requests
The initial router queried shards sequentially. Switching to parallel requests reduced mean end-to-end latency from 331 ms to a measured range of approximately 171–242 ms.
Exact versus approximate search
In the reported FAISS comparison, flat search averaged about 0.82 NDCG@10 and 1.3 ms query latency; IVF at nprobe=16 averaged about 0.74 and 0.17 ms. Approximate search traded retrieval quality for speed.
Calibration instead of migration
The reported cost baseline estimates 11–15 minutes for calibration-only re-embedding, compared with 6.8–9.3 hours for full-corpus re-embedding per transition. Full-corpus timing is a throughput-based extrapolation.
TECHNOLOGY STACK
06 / SCOPE & LIMITATIONS
What these results do—and don't—show
Drift-RAG demonstrates that query-side orthogonal alignment can recover much of the measured retrieval quality across the tested BGE model transitions. The experiments also show that the simplest effective adapter outperformed the more elaborate variants considered here.
The CC-News evaluation is a constructed self-retrieval task, not a general-purpose relevance benchmark. The project also reports an unresolved discrepancy: the two transitions rank differently under MMD distributional alignment and NDCG-based retrieval recovery. These metrics capture related but distinct properties.
The implementation returns ranked documents; it does not generate answers with an LLM. The reported results are evidence for the tested setup, not a guarantee that the same recovery will hold for unrelated embedding families, arbitrary corpora, or every production workload.
EXPLORE THE IMPLEMENTATION
Interested in the system details?
The repository contains the ingestion pipeline, adapter training, distributed retrieval services, and evaluation scripts.