SOFTWARE PROJECT
A retrieval layer built for more than one index
This is a hybrid retrieval system that pairs sparse and dense search, reranks candidates with an SBERT-based semantic model, removes near-duplicate vectors before they reach the reader, and routes a query to the right index when more than one collection is available.
- Retrieval
- Hybrid search
- Reranking
- SBERT semantic model
- Housekeeping
- Vector deduplication
- Scale
- Multi-index routing
THE QUESTION
A single index does not stay clean or fast forever
A retrieval-augmented pipeline built around one vector index tends to run into the same problems as it grows: keyword-only or embedding-only search each miss cases the other would catch, near-duplicate chunks crowd out genuinely different results, and a single flat index becomes an awkward fit once the underlying content covers more than one domain.
I built this system to address those three problems together rather than patching them one at a time.
METHOD
Retrieve, dedupe, rerank, route
- 01
Hybrid retrieval
Sparse keyword search and dense embedding search run side by side, and their candidate sets are combined so neither approach alone caps recall.
- 02
Vector deduplication
Near-duplicate chunks are collapsed before ranking, so the same passage does not occupy several of the returned slots.
- 03
Semantic reranking
An SBERT-based model reorders the surviving candidates by semantic relevance to the query, ahead of the initial retrieval score.
- 04
Index routing
A routing step decides which index, or indexes, a query should be sent to when the underlying content spans multiple collections.
SCOPE
What is public today
Not yet split into its own public repository
This project currently lives alongside other retrieval work under my GitHub account rather than as a single standalone repository. I plan to extract it once the routing layer is stable enough to publish on its own.
RESOURCES