Hybrid search for RAG: start with what the question needs

Many RAG retrieval problems get diagnosed from the wrong end. A wrong answer comes back, and the first fixes tried are a bigger model, a different chunk size or a new embedding model. Often the problem sits earlier: the retrieval method didn't match what the question needed.

A useful habit, and one that keeps coming up in the practitioner conversation around "context engineering", is to ask what kind of evidence a question needs before tuning anything. There are three common answers.

1. Exact match

"What's the fix for E-4021?" (Example.)

Error codes, part numbers, clause references, SKUs and names. The user knows the precise string. Similarity search can be surprisingly weak here: two error codes that differ by one character can look almost identical to an embedding model, while the right document may phrase everything around the code differently.

Keyword (lexical) retrieval, such as BM25, is built for this. It rewards exact term overlap and rare terms. For highly structured documents, plain text search goes a long way.

Signs you need it: identifiers in queries, and users complaining that the "obvious" document didn't come back.

2. Meaning

"Why are customers cancelling?" (Example.)

The answer is spread across tickets, survey comments and call notes, and none of them may contain the word "cancel". This is where vector search earns its place: embeddings put passages with similar meaning near each other, even when the words differ.

Signs you need it: paraphrased questions, natural-language queries over unstructured text, and content in several languages.

Watch for: questions where one small word changes the meaning, such as "not", "except" or "excluding". Similarity scores are good at topic and can miss polarity.

3. Relationships

"Which contracts does this supplier change touch?" (Example.)

No single passage contains the answer. You need to know which contracts name the supplier, which clauses depend on its terms, and which projects rely on those contracts. That's a path through connected entities, which makes it a graph question.

Signs you need it: "which … affect …", "who reports to …", "what depends on …", multi-hop questions, and answers that have to be assembled rather than found.

Hybrid is the default, not the exception

In practice, many real question sets contain all three types, often on the same day. That's why hybrid retrieval, which runs keyword and vector search in parallel and fuses the ranked results, has become a sensible default for many teams.

A common fusion method, reciprocal rank fusion, combines results by their rank position rather than their raw scores. That sidesteps the problem that keyword and vector scores aren't on the same scale.

Two cautions:

• Fusion isn't free accuracy. If one method returns a confidently wrong result, fusion can promote it. Test on your own question set.
• Reranking is a separate step. A reranker, often a small cross-encoder, reads the question and each candidate passage together and re-orders a shortlist. It's slower per item than first-stage retrieval, but it can fix ordering problems that similarity scores can't.

Add a graph when the relationship is the evidence

The graph is the step many teams add either too early or too late. Too early, and you're maintaining entity extraction for questions that keyword and vector search would have answered. Too late, and you're trying to answer multi-hop questions by stuffing more chunks into the context window and hoping.

A simple test: take 50 real questions from your users and label each one exact match, meaning or relationships. Some will be two. If relationships come up often and the answers matter, a graph is worth it. If they don't, hybrid search plus a reranker may be all you need for now.

Routing in practice

You don't need a sophisticated router to start:

1. Detect identifiers with simple rules (patterns for codes, IDs and references) and weight keyword results for those queries.
2. Run hybrid retrieval for everything else.
3. For question patterns that need relationships, expand from the retrieved entities along the graph and add the connected passages.
4. Rerank the final shortlist before it reaches the model.
5. Log which method produced each passage the model used. When an answer is wrong, you'll know where to look.

The infrastructure question

Each of these methods has traditionally lived in a different system: a text search engine, a vector database and a graph database. Routing between them is easy in a diagram and harder in production, where each system has its own write path, its own access model and its own failure modes. (We cover that cost in GraphRAG's hidden cost isn't the graph.)

Common questions

Is hybrid search always better than vector search alone? Usually it's a safer default, because it covers identifiers as well as meaning. But fusion can promote a wrong result, so measure it on your own question set rather than assuming.

Do I need a reranker? If the right passage is usually in your top 10 to 20 but not at the top, a reranker is often the cheapest improvement. If the right passage isn't in the shortlist at all, fix first-stage retrieval first.

When is a graph worth the effort? When a meaningful share of important questions need relationships, and the answers are costly to get wrong.

Where CEREBRAL fits

CEREBRAL is the vector + graph database for AI search: vectors and relationships through one API, in one backend. Vectors are compressed from the first write, and it runs self-hosted, in your cloud or on-prem.