Running RAG in Production

Tomohiro Iida · Published June 30, 2026 · Updated June 30, 2026

An internal knowledge search assistant connects your regulations, manuals, past projects, and FAQs to an LLM so it answers questions with cited sources. A demo can be running in an evening; reaching quality good enough for daily use is a different problem. This article is for engineering teams building or evaluating a retrieval-augmented generation (RAG) system over internal documents, and focuses less on concepts or how to run a proof of concept, and more on the parts that matter for going into production: retrieval quality, citations, evaluation, and security.

What RAG solves, and when to use something else

An LLM carries knowledge learned during training, but on its own it has no access to your company's regulations, past projects, or their latest versions. There are three main ways to connect internal documents to an LLM, and they behave differently.

• Fine-tuning: adjusts the model's weights through further training. Effective for fixing tone or for classification tasks. It can be used to teach knowledge too, but it lags RAG on freshness, coverage, and how reliably it can point back to a source. Reflecting new knowledge requires retraining or a separate external-knowledge mechanism.
• Stuffing documents into the prompt: pastes the whole relevant document into the prompt every time. Works for small volumes, but input length and cost grow as documents increase. Long-context models and caching can ease this, but repeatedly stuffing large volumes of documents into every prompt is not an efficient way to operate.
• RAG: retrieves only the documents relevant to the question and appends the retrieved passages to the prompt. Document updates are reflected by re-indexing, alongside versioning and permission updates, and it can return retrieval-based citations.

For a use case like internal knowledge — high volume, frequent updates, a need for citations — RAG is the practical default. Using fine-tuning alongside RAG makes sense specifically when you also want to lock down tone or output format.

The pipeline, end to end

RAG splits into two phases: an upfront "index building" phase, and a "retrieval and generation" phase that runs on every question.

Index building (upfront, batch processing):

• Document ingestion
• Chunking
• Embedding into vectors
• Storage in a vector store

Retrieval and generation (on every question, in real time):

• User's question
• Question is embedded into a vector
• Vector store is searched (retrieval)
• Relevant chunks are reordered (reranking)
• Question and chunks are assembled into a prompt
• The LLM generates an answer with citations (generation)

Quality compounds across every stage, so a weak link anywhere breaks the whole chain. If the retrieval stage fails to surface the right chunk, the LLM might still happen to answer correctly from its own training knowledge, but that is not a cited, evidence-based answer. When an answer looks wrong, check retrieval before touching the generation prompt — it narrows the problem down faster.

Getting chunking right

Documents are not embedded whole; they are split into units sized for retrieval, and this split has a large effect on retrieval quality.

• Size: too large and a single chunk mixes multiple topics, hurting retrieval precision. Too small and it loses context. Start around a few hundred tokens and tune from there against your actual documents.
• Overlap: overlap neighboring chunks slightly so context does not break at a boundary. Too much overlap increases duplicate hits — around 10 to 20 percent is a reasonable starting point.
• Splitting by heading: regulations and manuals have a chapter-and-section structure. Splitting along that structure keeps meaning intact better than cutting mechanically by character count.
• Metadata: attach source document name, section number, last-updated date, department, and access scope to every chunk. This is an easily overlooked investment that pays off for citations, filtering, and freshness management all at once.

Fixed-length mechanical splitting is good enough for some cases, but for structured documents like internal regulations, splitting along the document's own structure works better.

Improving retrieval quality

Vector search alone (searching by semantic closeness) misses things. A practical setup combines:

• Hybrid search: combine vector search with keyword search such as BM25. Vector search handles paraphrasing well but tends to miss exact-string matches like product codes or internal jargon, which keyword search covers. Combine both scores to rank results. For small-scale, exact-match-heavy use cases, keyword or database search alone may be enough.
• Reranking: pull a larger set of candidates from the first-pass search, then use a dedicated reranking model to reorder them by relevance to the question. This corrects the rough ordering from the first pass and improves the quality of the chunks that finally reach the LLM.
• Metadata filtering: narrow candidates before search — "only the latest HR version," "only regulations still in force." This also ties directly into freshness and access control.

You don't need everything from day one. A practical order is to start with a vector-plus-keyword hybrid as the baseline, then add reranking once evaluation shows a gap.

Generation: citations, hallucination, and "I don't know"

Retrieved chunks get appended to the prompt for the LLM to answer from, and this is where production quality diverges from a prototype.

• Always return citations: have the answer list the source it drew on (document name, section). Users can check the primary source and catch errors. For internal knowledge search or checking a regulation, an answer with no citation cannot be verified and is not trustworthy enough.
• Constrain answers to the retrieved evidence: state explicitly in the system prompt that the model should answer only from the provided documents and not guess at anything not covered. LLMs tend to fill gaps from their own knowledge, so this needs an explicit constraint.
• Allow "I don't know": when no relevant chunk is found, or what's found is insufficient, have the system return "no matching document found" rather than force an answer. Being honest about a miss is more trustworthy in production than a confident wrong answer. It also helps to stop before generation entirely when the retrieval score falls below a threshold.

How well a model follows instructions varies by model and configuration. Even with a model known for strong instruction-following, such as Anthropic's Claude, it is safer not to rely on the prompt alone for this constraint — combine it with retrieval-side controls such as thresholds and filters.

Example system prompt: "You are an internal knowledge search assistant. Answer only from the reference documents below. If the reference documents do not cover the question, say no matching document was found. Do not fill gaps by guessing. Always list the document name and section number you relied on at the end of your answer."

Running evals

Shipping something because "it seems to work" means accuracy can silently degrade with every model update, document addition, or index change. RAG is a system that assumes ongoing evaluation.

• Retrieval hit rate: build a set of representative questions and measure how often the correct chunk appears near the top. This is where you isolate retrieval-stage quality.
• Answer-to-evidence consistency: check that the answer does not contradict the retrieved chunks. A mismatch between citation and answer is a sign of hallucination.
• Regression testing: rerun the evaluation set whenever you change chunk size, update the model, or add documents, to check nothing has degraded. An evaluation set, once built, becomes a lasting asset.

An evaluation set can start small. Even a few dozen commonly asked questions from the field is enough to talk about improvement in numbers rather than impressions.

Common failure patterns

• Chunks too large or too small: topics get mixed together, or context gets cut off. Use evaluation to find the right size.
• Stale documents slipping through: an expired older version turns up in search and becomes the basis for a wrong answer. Filter it out using last-updated metadata and an active-version flag.
• Cross-searching without regard to permissions: searching all documents indiscriminately can leak information the user is not authorized to see.
• Answers with no citation: there is no way to check the basis for the answer or catch an error. Citations are a requirement, not an optional feature.

Security and access control

Internal knowledge is a concentration of confidential information, and it deserves the same design weight as any technology decision.

• Managing the path to an external LLM: with a SaaS-hosted LLM, the question text and retrieved chunks get sent to an external service (an on-premises, VPC, or local-model setup does not send data externally). Check each provider's data-handling policy — whether it trains on your data, and how long it retains it — and where needed, choose a plan that disables training on your data, or a setup that stays entirely within infrastructure you control.
• Applying access permissions inside RAG: the original document's view permissions must always be enforced at the retrieval stage. Give every chunk access-scope metadata and filter by the user's permissions at search time. Restricting what's displayed after the fact, in the application layer, can be too late — by then the LLM has already seen information the user was not authorized to view. Enforce permissions at retrieval, before anything reaches the LLM. Treat this as a hard rule.
• Audit logging: record who asked what, and which chunks were passed in as evidence. Useful for investigating a potential leak and for ongoing quality improvement.

When not to use RAG

RAG is not a universal answer. Consider other approaches when:

• The answer lives in structured data, not documents: an aggregate like "total sales last month" is a database query, not a search problem. Having the LLM call SQL or a function suits this better than RAG.
• Documents change often and instantly: a value like a real-time inventory count changes faster than an index can keep up. Fetch it from an API at query time instead.
• The main goal is fixing output tone or format: if the goal is formatting the output rather than adding knowledge, fine-tuning or a template fits better.
• The decision requires a high degree of rigor: don't hand a legal or medical final decision to RAG. Treat it strictly as a guide pointing back to primary sources.

RAG's strength is pulling the relevant passage, with evidence, out of a large body of internal documents. Confirming early on whether your problem actually fits that shape is the fastest route to something that works in production.

Frequently asked questions

Should I choose RAG or fine-tuning?

RAG, if you want to add or update knowledge; fine-tuning, if you want to lock down the tone or format of the output. Internal knowledge search falls into the first case. The two aren't mutually exclusive — using fine-tuning for tone and RAG for knowledge together also works well.

What should I use for a vector database?

There are several options: a dedicated vector database, a vector extension on an existing database, or the vector features of a search engine. Choose based on document volume, existing infrastructure, and operational capacity. For smaller scale, starting by extending existing infrastructure and considering a dedicated product once scale grows is a reasonable path. Rather than committing to a specific product upfront, decide based on evaluation.

An answer is wrong — where should I look first?

Check the retrieval stage first. Confirm whether the correct chunk appears near the top of the search results; if not, revisit chunking design, hybrid search, and reranking. If the correct chunk is near the top but the answer is still wrong, adjust the generation-stage prompt and constraints.

Can hallucination be eliminated completely?

Not to zero. Showing citations, constraining answers to the retrieved evidence, allowing "I don't know," and ongoing evaluation together lower the rate and make it easier for users to notice when it does happen — that is the realistic goal.

How do I apply access permissions?

Give each chunk access-scope metadata and filter at retrieval time based on the user's permissions. Controlling what's displayed after generation is too late in principle, since information could already have reached the LLM by then — filtering before retrieval is the rule.