What is RAG? AI that answers from your documents

Making a language model search your own documents before it writes an answer: the parts of the system, where it goes wrong, testing with a fixed question set and personal data points.

· 12 min readAI

Short answer

RAG means a language model searches the company’s own documents before it answers and uses the passages it finds as context. Its parts are documents, chunking, embeddings, search, an answer step and citations. Stale documents, wrong chunks and unsourced answers are the usual failures; test with a fixed question set.

A new starter asks the internal assistant, “How does annual leave carry over?” The assistant answers with full confidence. The answer comes from an HR document that was withdrawn last year; the current one sits in the same folder, but the assistant never finds it. The company in this article is fictional. This is where the question “what is RAG” earns its place. If an assistant answers from your documents, the answer depends first on which document it finds, then on how it uses what it found.

We covered the steps of a customer-facing chat box in how to set up an AI assistant. This article looks at the layer underneath it: how answers are built from documents.

What RAG adds to a language model

RAG stands for retrieval-augmented generation: the model builds its answer with the help of a search. By the glossary definition, the model first finds the relevant passages in the business’s own documents and takes them in as context before it writes.

The academic work that carries this name is the 2020 paper by Lewis and colleagues. The authors study models that connect a pre-trained language model to an index of documents through search. In the abstract they report that these models produce more specific, diverse and factual language than a comparison model that relies only on its own parameters. The same abstract says that providing provenance for a model’s decisions and updating its knowledge are open research problems for pre-trained models.

For a business the consequence is this: the model is not retrained on your documents. The documents are searched on every question. So when you update a document and refresh the index, the assistant can use the new information, and the answer can show which file it came from. RAG only produces answers; for systems that do things, such as opening an order or starting a refund, see what an AI agent is.

The six parts of a RAG system

The system works at two different times. When documents are added or changed, a preparation step runs: the documents are split into chunks, an embedding is computed for each chunk, and the result is written to an index. An embedding is a representation that turns the meaning of a text into a list of numbers; texts with similar meaning get similar numbers. When each question arrives, the index is searched, the answer is written from the chunks found, and a source is placed beside it.

How a RAG system flows: documents prepared into an index, a search on every question, an answer with its source, a hand-over to a person when no chunk is found Preparation (on changes) Documents Split into chunks Embeddings computed Index vector store Each question Question Search Top chunks Answer step Answer and source No chunk: no guessing, hand over to a person How a RAG system flows: documents prepared into an index, a search on every question, an answer with its source, a hand-over to a person when no chunk is found Preparation (on changes) Documents Split into chunks Embeddings computed Index vector store Each question Question Search Top chunks Answer step Answer and source No chunk: no guessing, hand over to a person
Figure: Documents are split and written to an index in advance; every question searches the index, the answer is built from the chunks found with its source, and with no chunk the conversation goes to a person.

OpenAI’s retrieval guide explains that a file added to a vector store is automatically chunked, embedded and indexed. According to the same guide, a search returns the relevant chunks, their similarity scores and the file of origin. Semantic search can also find results that share few or no keywords with the question but say the same thing.

The table below sets out the six parts and the place where each one breaks most often.

Part What it does Where it breaks most often
Documents The only source of the answer An old and a current version sitting side by side
Chunking Splits a document into searchable sections The chunk loses which product or period it belongs to
Embeddings Turn the meaning of each chunk into numbers Weak where an exact match is needed, such as codes and clause numbers
Search Brings back the chunks closest to the question The right chunk does not come, a wrong one does
Answer step Gives the model the question and chunks and has it write The model steps outside the chunks and guesses
Citations Put the file and section beside the answer There is no source, or it does not support the answer

Setting it up in six steps

  1. Collect the questions first. Write down real questions from the inbox, support records and what colleagues ask. This list sets the scope and later becomes the first version of the test set.
  2. Record each document’s owner and date. Every document needs a person responsible and a last-updated date. Remove contradicting or outdated versions before they reach the index.
  3. Split the chunks with their context. Cut at headings and clause boundaries, and add the file name, section heading and version date to each chunk. Anthropic’s contextual retrieval article shows with an example that search gets harder when a chunk does not say which company or period it is about, and suggests adding explanatory context to the chunk.
  4. Search in two ways. The same article says embeddings can miss crucial exact matches, and describes combining embeddings with keyword search (BM25). OpenAI’s guide also has a score threshold: raising it limits the results to more relevant chunks but may leave out some useful ones. If your tool has such a threshold, choose its value with your test set.
  5. Write the rules into the answer step. Tell the model to rely only on the chunks it is given. Tell it to say “I could not find this in the documents” when they are not enough, and to put the file and section beside every answer. The hand-over step in the assistant article covers how to pass a conversation to a person when nothing is found.
  6. Limit access by user. If everyone can search every document, a pay letter and a product manual should not sit in the same index. The details are in the personal data section below.

Where it goes wrong

Stale documents. The index only knows the state it saw during preparation. If a document changes and the index is not refreshed, the assistant gives the old answer and cites a source for it. OWASP’s LLM08 entry lists contradicting data from several sources, and corrupted data entering the knowledge base on purpose or by accident, as separate risks. We suggest putting a version date on every document and setting a rule that refreshes the index when a document changes.

The wrong chunk. Search can bring back a section that looks like the question but does not contain the answer. The model writes an answer from what it has, and because it reads fluently the mistake goes unnoticed. The way to catch it is to check the chunks the search returned as well as the answer.

Answers without a source. If the answer has no file and section, nobody can check it quickly. If a source is written, check that it really supports the answer; a file name on its own does not show the answer is right.

Testing with a fixed question set

A RAG system is not tried once and left alone. Behaviour changes as documents and settings change. The remedy is to keep a question set that does not change and run the same set after every change. For each question, mark two separate things: was the right chunk retrieved, and does the answer match that chunk. If the chunk is wrong, the problem is in search; if the chunk is right and the answer wrong, the problem is in the answer step.

Put a few examples of each kind in the list. The table below is a starting draft.

Question type Expected result What it catches
A question answered in one document Correct answer and correct source Basic function
A question whose answer spans two sections Both sections are retrieved Chunk boundary problems
A question with no answer in the documents “I could not find it” and a hand-over Invented answers
A topic with an old and a new version Only the current document The old document staying in the index
The same question in different words The same answer Weak search
A question from a user without permission No answer given A breached access limit

When you change a setting, rerun the whole set; one fix can break another answer. If you keep the results with dates, you can see whether the system is improving over time.

When documents contain personal data

Contracts, personnel files and customer correspondence carry personal data. Every document that enters the index means that data is processed in a new place. The Personal Data Protection Authority’s guide on generative AI brings out these points.

  • Data minimisation. The guide says the personal data used should be limited to what is necessary. Do not index documents the assistant does not need in order to answer its questions.
  • Retention. The guide recommends regularly reviewing which personal data is kept and for how long, and developing retention and destruction policies. We suggest also deleting a removed document from the index and the logs.
  • Transfers abroad. If you work with a provider based abroad, the transfer must comply with Article 9 of Law No. 6698 and the related regulation. Read in the contract which country the provider processes data in and whether documents are used for training.
  • Privacy notice. The guide also refers to the privacy notice duty in Article 10 of the Law. If the data of people in your documents is also processed for the assistant, check with your lawyer whether your current privacy notice covers it.

On security, OWASP’s LLM08 entry lists these among its measures: permission-aware access control and detailed logs of retrieval activity. When user groups share one index, one user’s question can bring back a chunk from another person’s document. So label each chunk with the groups allowed to see it and filter the search by that label. For the framework, see also the KVKK glossary entry; for a legal assessment, ask your own lawyer.

Common mistakes

  • Building the index once and not refreshing it when documents change.
  • Checking the answer and never looking at the chunks the search returned.
  • Asking for citations without checking that the source supports the answer.
  • Keeping no test set and judging every settings change by feel.
  • Leaving the access limit until after launch.
  • Throwing a whole folder into the index without removing drafts, copies and files with personal data.

Common questions

Is RAG the same as training the model on our own data?

No. In RAG the documents are not built into the model; they are searched on every question and handed over as context while the answer is written. That is why updating is easy. How a provider stores the text you send, and whether it uses it for training, depends on the contract terms.

Does RAG prevent wrong answers completely?

No. Search can bring back the wrong chunk, a document can be out of date, and the model can step outside the chunk. RAG ties the answer to documents and makes it checkable; accuracy still comes from the quality of the documents and the search. That is why a fixed question set and human review are needed.

Where should we start?

Start with one topic area, for example only returns and delivery documents. In a small area it is easy to build the test set, see the errors and fix the documents. Tie any widening to the test results. For other jobs that suit automation, see business process automation examples.

One thing you can do today

Write the real questions you have received in recent weeks on one page. Beside each, note the name of the document that holds the answer and its last-updated date. Questions you cannot point to a document for, or whose documents are old, are the first things to clean up before building any RAG. The rest of the list becomes the first draft of the fixed test set.

If you would like to build this together, our automation and AI service page explains how we set up an assistant that answers from your documents.

Sources

This article is general information, not legal, financial or technical advice. Platforms, their rules and the law change over time, so some details may have changed since 15 November 2026 or contain small errors. Before you make a decision for your business, check the current official sources or write to us.

Tell us about
your project.

Fill in the form or write to us directly, and we will agree on a day to talk.

Your first meeting is with Deniz Sezer, founder of Detmore.

First meeting free · Reply within one working day · Written proposal in 2 to 3 days · NDA ready

What do you need? pick as many as you like

Name, email and phone are required.

Contact

We reply within one working day.