Retrieval-Augmented Generation Best Practices: Building Production-Ready RAG Systems

A possible reason is that teams often go after the wrong problem. The problem is not when they spend time doing fine-tuning; it's when they don't have the right, timely information. Others develop retrieval applications without sufficient consideration of those components that make them actually work: data quality, document preparation, chunking, evaluation, and monitoring. The end result is usually a system that works well during a demo but fails when it gets under load in real use and costs more and takes longer to develop.

In reality, most failures are not actually due to the model. They are due to faulty data foundations, lack of retrieval and evaluation in the development lifecycle. In most instances, improvements in these areas are more valuable than buying a bigger model or continually adjusting prompts.

This guide covers the best practices that engineering teams use when building reliable production systems using retrieval augmented generation. You will get to know how to build a quality retrieval pipeline, optimise knowledge retrieval, test performance, add guardrails, and continuously improve your application as data and business needs evolve.

Why Following Retrieval Augmented Generation Best Practices Matters

In a RAG application, the retrieval route takes precedence over the language model. Although the advance model can't deliver a correct answer when a system fails to return accurate data or when a system does not return data at all. Upgrading to a large model is typically less significant in production than an app's efficacy.

The first thing to do is to determine where the failure has taken place when answers fall short. It is not necessarily the prompt or the model that is the problem. In more common instances, it is due to a lack of knowledge or base knowledge, poor chunking, poor retrieval, or bad query routing. Solving these problems usually brings more solutions than changing models and optimization.

Such best practices also help to close the gap between a proof of concept and a production-ready system. Enterprise apps demand consistency, predictable, reliable applications. These outcomes can only be attained by system efficacy, preserving the caliber of information contained, and tracking the system over time.

In the most successful deployments, retrieval is seen as an infrastructure rather than a language model competence. Reliable business outputs can be attained at scale in a RAG architecture with a retrieval pipeline; if the information gathered is clean, it can be measured over an extended period.

RAG vs. Fine-Tuning: How to Know Which Problem You're Solving

It is more efficient to identify the issue before making an investment to maximize the benefits of the solution. Retrieval and fine-tuning serve different purposes and will not significantly enhance results if used where the other is needed, and will delay development and cost.

When a query is sent, the retrieval layer searches for relevant information and adds it to the context. It works great when the source material is dynamic, like internal documentation, product catalogs, knowledge bases, policy documents, etc. Teams do not need to retrain the underlying system in order to modify the source or index as information changes.

There is a distinct reason for fine-tuning. It employs labeled scenarios in order to improve the consistency of carrying out a certain activity, following a particular rule, or returning data in a particular format. It is not meant to be a means for updating business information or document collections frequently.

There is a lot of production systems that are accepting a combination of both. The augmented generation provides up-to-date and consented information, and lightweight fine-tuning methods like LoRA boost consistency and domain-specific actions. Once the architecture is defined, it is necessary to provide a solid pipeline that will take the architecture to production, which is discussed in the next best practices.

8 Retrieval Augmented Generation Best Practices for Production Systems

The majority of production problems come from sources outside the LM. Inaccurate responses are much more likely due to weak retrieval, poor data preparation, ineffective evaluation, and limited observability than selection of the model. The aim of this section will be to look at the best practices for the most effective parts of the RAG pipeline.

1. Build Retrieval as a Core Product, Not a Feature

Many teams consider retrieval to be an LLM add-on. It is the other way when it comes to production. Since the retrieval will decide what the model has access to, the retrieval standards will have an impact on the quality of the answers. Even the brightest LLM will not be able to produce the right answer without the proper information.

This makes retrieval a distinct system that requires its own clear purpose, measurable goals and service-level objectives (SLOs), and its own ownership. Retrieval and generation are different problems, and success should be assessed separately. Retrieval is identifying the context, and generation is generating the appropriate response in that context.

Let's start with some of the metrics we use to measure retrieval, such as Recall@K, retrieval precision, retrieval ranking quality, and the p95 retrieval latency. Assess generation independently (faithfulness, quality of responses). This separation will help us to draw clearer conclusions about whether the limited responses are from search or the language model.

• Is the knowledge base missing the information?
• Have the incorrect documents been reviewed?
• Was there any query processing that didn't retrieve the correct documents?

After retrieval issues are resolved, suggested modifications or model improvements should be taken into consideration.

Finally, avoid starting with a complex retrieval structure. Establish a secure retrieval pipeline, test it with specific metrics, and then analyze and improve it based on measurable findings. However, retrieval is more than just an LLM feature; it is part of the product's architecture, making it easier to evaluate, maintain, and scale.

2. Get Your Data Foundation Right Before Upgrading the Model

If the RAG system provides inaccurate information, the LLM will probably not correct it. Outside of a production system, the quality with which the retrieval is done would be more sensitive to chunking, indexing, embeddings, etc., than the language model itself.

One common problem is using the same chunking approach across all texts. Very small chunks are missing the context to provide complete answers; very large chunks are including multiple topics and decreasing retrieval precision. The best approach is to match the chunking strategy to the type of document rather than have a single token limit for the whole knowledge base.

There are differences between retrieval and generation as well. Incorporating smaller chunks is useful for candidate retrieval because they are able to process individual concepts, while a larger window of context can be useful for the LLM to produce a good answer. A lot of production systems create small parts, then add them to the parent piece or document before creating.

When semantic chunking, documents can be chunked according to meaning instead of the number of tokens to improve retrieval. If your knowledge base is fairly extensive, it might be useful to create a hierarchical index (summaries with links to detailed information) to increase your ability to answer general and specific questions. Putting context information in, like a summary or questions the document can answer, also improves retrieval accuracy, providing more "semantic signals" for the search system.

Embeddings are also important. In the embedding model, they need to be the same at both indexing and retrieval times to ensure that documents and queries are in the same vector space. Retrieval performance might be degraded when changing between embedding models, as the stored vectors may not match the new model.

Below is a list of suggested chunking techniques for typical documents.

Typically, there is more to be gained from using chunking, indexing, or embeddings than from improving the language model itself for getting better retrieval accuracy out of it. When considering investing in a larger LLM, please ensure that the retrieval chain consistently provides the model with the relevant context.

3. Use Hybrid Retrieval, Not Just Vector Search

It can be used to determine semantic similarity, and may not be so effective at finding exact matches, such as product names, customer IDs, error codes, or legal references. On the opposite side of the spectrum, keyword search is effective for locating keywords not semantically related to the text. Hybrid retrieval is a hybrid of the two approaches that allows for information retrieval based on meaning as well as on exact matches in the same query.

This combination results in improved retrieval quality for most enterprise applications where dense vector search will retrieve paraphrases and conceptual similarity, and keyword search (BM25) will retrieve information that is more about entities and not necessarily about concepts. Having both certainly increases the chances of ensuring that the correct documents are in the candidate set in advance of generation.

A re-ranker is another step to enhance retrieval, because it will re-rank the set of candidate documents and place the most relevant documents at the top of the context window. Ranking documents will tend to improve the quality of answers, as LLMs will focus more on the earlier context.

When selecting a vector database, remember that features affect retrieval, not just storage. Find hybrid search, efficient indexing, permission-aware retrieval with metadata filtering, and low-latency search with knowledge base expansion. The features, more than the choice of vendor itself, affect production performance.

When you combine these three features with intelligent re-ranking, you get a more powerful retrieval pipeline, giving the model more relevant context and reducing retrieval errors before starting the generation process.

4. Optimize the Query Before You Optimize the Prompt

When teams are wrong, a lot of them suggest altering the question or prompt. Optimizing query processing before retrieval can be more effective in RAG systems for producing data. If the wrong documents are retrieved, no matter how well you prompt, you won't get an accurate answer.

Query rewriting is one of the effective methods. It converts vague, incomplete, or conversational user queries into a more precise query that is more likely to match the indexed knowledge base. But a good query rewriting should enhance retrieval, not be a substitute for poor data structure or an inefficient retrieval pipeline.

All queries do not need to be retrieved in the same way. Intent-based routing enables the use of varied routing strategies depending on the user's intent. For example, a "factual" query may trigger a simple retrieval process, while a more complex query on multiple documents may trigger a longer retrieval and synthesis process. Query by intent improves query retrieval accuracy without increasing the complexity of the model.

Note that if the input is not well formulated, it is better to ask a question to clarify it rather than give an answer that is not very confident. If no good matches are found in the documents, requesting further information from the user will result in improved retrieval quality, improved reliability of answers, and reduced hallucination.

Finally, use retrieved context rather than the model's pre-trained knowledge. Clear instructions that only require the model to respond to information in retrieved documents can help ensure adherence to the context and reduce out-of-scope answers, critical in many enterprise scenarios where factual integrity is of high importance.

5. Measure What Matters From Day One

Improvement of production RAG plants is only possible with systematic evaluation. Measuring only the end result will not allow the identification of failures in retrieval, generation or in the knowledge base. Each stage should be tested separately to find out whether there are any issues that will impact users.

The first step is to determine the quality of retrieval. For top-ranked chunks, the context precision is used to assess the relevance, while for retrieved documents, the context recall is applied to assess the relevance of the whole query. A noise sensitivity measure is utilized to evaluate retrieval performance when the irrelevant documents intermingle with the relevant documents, in order to highlight the strengths and weaknesses of the ranking.

Other measures are required for generation. Faithfulness: Whether the answer is retrieved from the context or is in the model's knowledge. Answer relevancy: ensures that the answer is linked to the user's question. Completeness is used in queries that have multiple parts in which the answers which satisfy only part of the query are considered complete.

Technical metrics shouldn't be the only criteria used in evaluations. Some of the most important business KPIs that must be met to ensure measured value post deployment are task completion rate, response latency, retrieval cost, user satisfaction and resolution rate.

Automated evaluation is also very important. If prompts, embeddings, retrieval strategies, or models change, regression testing is possible with a golden set of questions in production format. Many engineering teams utilize LLM-based reviewers since traditional metrics, such as BLEU and ROUGE, emphasize lexical similarity rather than factual correctness.

Tools like Ragas, DeepEval, and TruLens provide useful ways to assess response quality, regression performance, and retrieval quality in production augmented generation systems.

Lastly, make evaluation an ongoing effort and not a deployment milestone. Evaluation suite should grow as knowledge base grows, and each production failure should be a regression test, therefore, so that they will not be repeated in future production updates.

6. Design Your RAG System for Constantly Changing Knowledge

Knowledge bases are not just a one-way street. Product documentation, policies, pricing, and other articles and internal processes are continuously updated and the retrieval pipeline should keep evolving along with them without manual intervention.

A possible approach is to organise knowledge in hierarchical knowledge bases. Information that changes rapidly can be indexed from shorter cycles and stable reference articles can be indexed in less frequency. This means that when new information is published, the index will still be retrieved without needing to create a new one.

This is because only new, changed or deleted documents are supported for automated delta indexing. However, if the embedding model, or chunking method changes then re-index everything. The retrieval accuracy may be decreased by silently mixing up embedding models or chunking methods.

Verify content prior to it getting to the index. Remove duplicate documents, verify metadata, guarantee access permissions and remove incomplete or obsolete documents. It's simpler to stop bad information from getting into the knowledge base than to fix bad retrieval later.

In cases where the data is updated often or is updated in real-time, such as stock levels, order status, account information, pricing, and other cases, access the data via API as well as indexed documents. This way, you can be certain that the answers you receive will be based on the latest information from your business, rather than just the latest indexing round.

A production RAG system should always keep its knowledge up-to-date, cross reference new knowledge and add business data (if available), which is indexed. The retrieval layer must be kept up-to-date in order to ensure the quality of the answers in the face of changes in the underlying information.

7. Make Guardrails Part of the RAG Pipeline

Guardrails should be put in place throughout the RAG pipeline and not just in the final response. Before incorrect information reaches the user, they prevent unsafe inputs to the system, unreliable outputs, and unauthorized access to data.

First attempt at input filtering. Detect attempts to inject data at an appropriate time, prevent access to untrusted or policy-burdened data sources, and only access trusted and permissions-aware data sources. This will prevent any other content from being generated that might not be relevant or approved by the account.

Verify after creating as well. Ensure answers are based on the context presented, comply with system requirements, and that they do not contain confidential or personally identifiable information (PII) or other sensitive data, such as business information. Responses which fail these checks will need to be re-generated, blocked or sent for human review.

Instead of focusing only on the final outcome, examine every stage of an agentic or multi-step workflow. If a retrieval error happens or there is a hallucinated intermediate result, the following actions would not be reliable.

For every guardrail, a metric, threshold and response should be determined. For example, if the low context conformance is failed, then the request may be regenerated; if the sensitive data is detected, it may be redacted or blocked in the response; and if the request is aborted, it won't be sent to the server. This is because a guardrail is not really a safety measure, but an operational control, if these actions are defined in advance.

8. Monitor and Improve Your RAG System Continuously

Production RAG systems must be monitored continuously since there are changes in retrieval quality, usage, and knowledge bases. Even if the language model doesn't change, retrieval accuracy slowly diminishes if it's not continually evaluated.

Monitor technical metrics such as accuracy of retrievals, rate of hallucination, response time, and the cost of infrastructure, and business metrics like job completion, user satisfaction, and resolution time. Both views are important to identify if issues are retrieval, generation or knowledge base.

Collect user input - intentional & unintentional. Retrieval failures can be signaled directly (with ratings and with thumbs down) or indirectly (with repeated queries, abandoned sessions, users rephrasing the same question). When making changes to the system, be sure to consider these signals, as the quality of the feedback may be caused by issues of latency or usability, not the answer quality.

Apply production information to improve the whole chain. That could be because they need to be chunked better, the metadata needs to be updated, there's better query routing, or it's because the documents in the source need to be updated. If the hallucination continues to occur, it is probably more likely to be a sign of context that isn't adequate or good, than a problem with the prompt or language model.

Each of the production failures should be converted to a regression test. Ensure that failed queries are added to the evaluation set and that subsequent changes to the evaluation set are compared to before deploying changes. This way, it ensures that each try will yield a better result for retrieval, and that the same problems are not encountered again.

From Best Practices to Production: Turning RAG Into a Reliable AI System

Building a production-ready RAG system is an engineering process that needs to be iterative. Do not optimize everything simultaneously, try to get a stable retrieval pipeline, benchmark it, and continuously improve it.

Identify the use case and success criteria

Start with a problem to solve in the business and metrics to measure success. Determine the accuracy, latency and business KPIs before development so that all optimizations have a clear objective.

Organize and curate knowledge base

Clean source documents, eliminate redundant and out-of-date data, maintain accurate source documents, and control access to source documents. A good knowledge base is more effective when it comes to boosting retrieval quality than adopting a bigger language model.

Choose appropriate chunking and indexing strategy

Use chunking techniques based on the type of document, not a fixed size approach to chunking. Prior to changing the model, optimise embeddings, indexing and metadata for the highest retrieval accuracy possible.

Create a hybrid retrieval pipeline

Search semantically and find the keywords, and re-ranking for the most relevant context. This provides more consistent retrieval than a vector search, especially for enterprise knowledge bases.

Get evaluation and guardrails on day one

Collect, create and calculate business indicators individually. Prevent the user from having hallucinations, breaking the policy and failing to retrieve with runtime guardrails and validation checks and automated regression testing.

Continuous monitoring, learning and iteration

Monitor production statistics, study user feedback, and transform all production failures into a regression test. Continuously improve retrieval, prompts, chunking and the knowledge base as requirements change.

The most reliable RAG systems are developed by considering retrieval as product infrastructure instead of an LLM feature. Retrieval quality, evaluation and continual improvement are woven into the engineering lifecycle, making RAG applications more accurate, scalable and easier to maintain in production.

Planning a RAG Implementation? Here's How We Can Help

Building an actual augmented generation solution is not as easy as picking a large language model or using a vector database. It relies on establishing the right retrieval design, organizing the data, developing suitable evaluation and continuously updating the system to match business requirements.

At Coding Crafts, we build custom retrieval augmented solutions that are designed for production from day one. We work together with clients to know about their data environment, select the best retrieval architecture, develop the hybrid search pipelines, define evaluation metrics, and integrate automation into existing business processes.

If you're looking to adopt a RAG or improve an existing one, Coding Crafts can help you create a powerful and trustworthy AI solution that meets your business goals, data, and future requirements.