Most retrieval projects run into the same problem at about the same time. You split up the documents, create embeddings, store the vectors, pull the top five results, and send them to the model. The demo works great. But then someone asks about invoice 4471, and the retriever gives back five paragraphs about invoicing philosophy. The results are related in meaning, but not actually helpful.
Nothing is actually broken. The first approach is just too basic for the kinds of questions people really ask. Each of the patterns below was created because someone faced a version of this problem and found a way to solve it.
If you only make one change, make it this one. Vector search understands meaning but struggles with exact matches like part numbers, error codes, surnames, or version tags such as v2.3.1. Keyword search is great for those, but fails when someone rephrases the question. The solution is to use both methods together and combine their ranked lists using Reciprocal Rank Fusion. This idea comes from a 2009 SIGIR paper by Cormack, Clarke, and Büttcher and still works well today. The key is that RRF merges the positions in the lists, not the scores. BM25 scores can be very high, while cosine similarity usually stays between 0.6 and 0.95, so mixing the scores directly is not helpful. Results that both methods rank highly move to the top. On the WANDS furniture benchmark, basic RRF scored 0.7068 NDCG, compared to 0.6983 for keyword search and 0.6953 for pure vectors. With tuning, it reached 0.7497. Elasticsearch, Weaviate, and Qdrant all support this feature.
Imagine you have eighteen research papers and a user wants to compare their methods. Top-k retrieval won’t help, because the answer is spread across multiple documents. In this approach, each document gets its own small agent with two tools: one to look up information and one to summarize the document. A coordinator sits above these agents. The coordinator doesn’t include every tool in its prompt, which helps with scaling. It finds the relevant document agents, decides which ones to use, gathers their responses, and writes the final answer. This method is powerful but costly. It requires several model calls per question, which increases latency, and debugging can be difficult if a sub-agent returns nothing.
Flat indexes lose the structure of the documents. Each chunk stands alone, not knowing it is part of a section or chapter, so a short question might compete with long passages that are not really related. Hierarchical retrieval adds another layer above the chunks by creating summaries, and searches these summaries first to decide where to look before going into details. RAPTOR, developed at Stanford and published at ICLR 2024, is a leading example. It clusters the chunks, summarizes each cluster, then clusters the summaries, repeating this process to build a tree. The results are impressive. Using RAPTOR with GPT-4 improved the top score on the QuALITY benchmark by 20 points, with RAPTOR itself reaching 62.4%, compared to 60.4% for DPR and 57.3% for BM25. The main cost is at indexing time, since you need a model to summarize every cluster, and again whenever the data changes.
Choosing how to split text into chunks is always a tough trade-off. Small chunks match exactly but lose context, while large chunks keep context but create vague embeddings that do not match well. The parent-child method avoids this trade-off. You split each document twice: first into large parent sections, then into smaller child sections inside each parent. Only the children are embedded. Retrieval matches the specific text in the children, but the model receives the parent section. Children are usually 100 to 400 characters, and parents are 500 to 2,000 characters. LangChain’s ParentDocumentRetriever manages this by storing child vectors in the vector store and parents in a document store linked by ID. This method is very efficient at query time and works best for structured documents like manuals, policy documents, or research papers.
Here’s something to consider: the user’s question is often the weakest part of the process. It might be unclear, poorly worded, or actually several questions combined, and it maps to a single point in embedding space that may not be close to the answer. Rewriting the question helps more than you might think. There are three main approaches: have the model create several rephrasings and retrieve results for each, split a complex question into real sub-questions and answer them separately, or clean up a messy query before searching. The last method is described in a paper by Ma and colleagues at EMNLP 2023, “Query Rewriting for Retrieval-Augmented Large Language Models.” Of these, breaking down the question usually gives the biggest boost in recall. None of these methods require changing your index, so they are often the cheapest to try. However, if you expand the query too much, you might start retrieving irrelevant information.
At first, this approach might sound strange. You ask the model to make up an answer, embed that answer, and use it for the search instead of the original question. The reasoning is that an answer is more similar to the passages you want to find than a question is, and the embedding model removes the made-up details but keeps the relevant structure. Gao, Ma, Lin, and Callan introduced this at ACL 2023, and it does not require training or labels. On TREC DL19, nDCG@10 improved from 44.5 with Contriever to 61.3 with HyDE; DL20 went from 42.1 to 57.9. Larger generators worked even better. The downside is that if you ask about something very obscure, the made-up answer might lead the retriever in the wrong direction. This method works best for new datasets without labeled data, which is common at the start of most projects.
Semantic similarity does not understand things like “only the current version” or “only what this user can see.” To solve this, add metadata when you bring in the data, such as source, date, author, section, document type, and permission level, and filter using this information before ranking results. This way, the retriever can enforce strict rules instead of just guessing, which is very important for systems with multiple users, versions, or dates. Self-querying goes a step further by letting a model read the user’s question and create the structured filter automatically. For example, “what did the sci-fi releases after 2020 say” becomes a real search string and a real filter. LangChain’s SelfQueryRetriever can do this with any store that supports filtering. Watch out for two problems: poor metadata when you bring in data, and filters that are so strict they return nothing.
This method deals with the same small-versus-large chunk issue as the parent-child approach, but from a different angle. Here, you index each sentence for maximum precision, since a sentence is the most specific unit of text. You also store the sentences around it as metadata. When you find a match, you replace it with the surrounding window before sending anything to the model. LlamaIndex uses SentenceWindowNodeParser and a post-processor for this, often followed by a cross-encoder reranker to filter out irrelevant matches. You will need to adjust the window size: if it’s too small, the context is lost; if it’s too large, you end up with big chunks again. This method works well for dense technical or legal documents, where the answer depends on a single sentence that needs context.
Not every question should be handled the same way, and not every question fits in the same index. A router reads the incoming query and sends it to the right place, such as the HR store, the finance store, the summary index, the detail index, or even a SQL database instead of vectors. There are three main ways to build a router: let a model choose using function calls, compare the query embedding to labeled example phrases for each route, or use simple rules. Semantic routing is very fast, taking only microseconds compared to the hundreds of milliseconds needed for inference, and sending easy questions to cheaper models saves money. The risk is that a misrouted query does not cause an error; it just gives an answer from the wrong place. Routes with unclear boundaries, like billing versus account management, often make mistakes.
It’s important to be clear: these are not just nine separate options to choose from. A mature pipeline uses most of them in order. You route the question, rewrite it, retrieve results with hybrid search on a filtered index built with parent-child or sentence window methods, rerank the results, and then generate the answer. Reranking is especially valuable. In the T2-RAGBench financial-document study, hybrid retrieval achieved Recall@5 of 0.816 compared to 0.587 for dense-only, and adding a cross-encoder gave the biggest improvement, about 17 points of MRR@3.
The order you implement these methods is more important than trying to do everything at once. Start with hybrid search and metadata filtering, since they are inexpensive and solve the most common problems. Add context expansion and a reranker if answers are technically correct but incomplete. Use query rewriting and HyDE if you need to improve recall. Save agents and summary trees for cases where questions really span multiple documents. Measure your results at every step, because failures in this process are often hard to notice.
