RAG just splits, embeds, and picks the nearest. So you don't need RAG for everything
RAG is the usual answer when you want AI to answer from internal documents. Underneath, it just splits documents, turns them into vectors, and passes the nearest pieces to the AI. Knowing this shows what it does well, what it doesn't, and when to skip it.

On this page
- Key points
- What RAG does: four steps
- 1. Split (chunking)
- 2. Embed (embeddings)
- 3. Pick the nearest (retrieval)
- 4. Pass and answer (generation)
- What RAG is bad at, and why
- When not to use RAG, and what to use instead
- If your material is small, give it all
- For numbers and lists, ask the database
- For fixed-form questions, ordinary search or an FAQ is enough
- For actions, give the AI tools
- When RAG is still the right choice
- Summary: five questions for choosing an approach
You want AI to read your internal documents and answer questions about them. Bring this up, and the reply is almost always "Then you need RAG." Sound familiar?
RAG (Retrieval-Augmented Generation) sounds impressive, but what it does is simple. It splits documents into small pieces, turns each piece into a list of numbers (a vector), picks a few pieces closest to the question, and passes them to the AI. That is all.
Because it is simple, its strengths and weaknesses are clear. This article walks through how RAG works using a fictional set of company policies, then covers when RAG is a poor fit and what to use instead.
Key points
- RAG is not a technology that makes AI smarter. It is a search mechanism that chooses which material to give the AI. The quality of the answer depends on the quality of the chunks it picks.
- Because it picks "nearby chunks," it struggles with questions that count items, compare across the whole, or need the latest figures.
- If your material is small, give all of it to the AI. If you need numbers, query the database. RAG is one option among several.
What RAG does: four steps
As an example, take a 300-person company that wants AI to answer questions about its work rules, expense policy, and travel policy (80 pages of PDF in total).
1. Split (chunking)
First, the documents are cut into pieces (chunks) of a few hundred characters each. 80 pages become a few hundred chunks.
There is no single right way to split. You can cut by character count, by heading, or with some overlap between pieces, and this choice changes answer quality a great deal. If the heading "Article 12: Travel expenses" and the line below it, "except for executives," end up in different chunks, a later step may pick only one of them.
2. Embed (embeddings)
Next, a list of numbers called an embedding is created for each chunk. Pass text to an embedding model and you get back hundreds to thousands of numbers. The model is built so that texts with similar meaning get similar numbers.
For example, "How much is the daily travel allowance?" and "The daily allowance for domestic trips is 2,000 yen" use different words but end up with close numbers. This is the difference from traditional full-text search, which finds nothing unless the keywords match.
The resulting numbers are stored in a vector database together with the text of each chunk.
3. Pick the nearest (retrieval)
When a user asks a question, the question is turned into numbers with the same model. Then, from the hundreds of stored chunks, the top few (say, five) with numbers closest to the question are selected.
Closeness is measured by how much two lists of numbers point in the same direction.
It is not picking by understanding meaning. It is picking mechanically by numeric closeness.
4. Pass and answer (generation)
Finally, the five selected chunks and the question are passed to the AI together with an instruction like "Answer based on this material." The AI sees only these five chunks. It has not read all 80 pages of policy.
What RAG is bad at, and why
The AI sees only "a few chunks close to the question." From this single fact, you can predict which questions it handles poorly.
| Example question | Why it is hard |
|---|---|
| "List every procedure in our policies that needs a manager's approval" | Even if the relevant passages are scattered across dozens of places, only the top few are passed |
| "Why is the hotel limit different in the expense policy and the travel policy?" | Comparing two documents requires the relevant passage from both to be selected |
| "How many expense claims were filed last month?" | Counts and totals live in business system data, not in documents |
| "What is this policy trying to say overall?" | A summary needs the whole document, but only part of it is passed |
| "What are the exceptions to Article 12?" | Numbers like "Article 12" or model numbers are hard to catch by closeness of meaning |
On the other hand, RAG is good at questions whose answer is written in one or two places in the material. Typical internal FAQs, such as "How much is the daily travel allowance?" or "How many days of parental leave can I take?", fall into this group.
There is one more common misunderstanding. RAG does not stop AI from making things up. If an irrelevant chunk is picked, the AI will build a plausible answer from it. That is why it is standard practice to show which chunks the answer was based on, so a person can check.
When not to use RAG, and what to use instead
If your material is small, give it all
Today's AI can read a large amount of text at once. In an article introducing an improved RAG technique, Anthropic writes:
If your knowledge base is smaller than 200,000 tokens (about 500 pages of material), you can just include the entire knowledge base in the prompt that you give the model, with no need for RAG or similar methods.
— Anthropic, "Introducing Contextual Retrieval" (September 2024)
The 80 pages of policy in our example can be passed whole, without splitting. Because the AI reads everything, it can also handle "list them all," "compare," and "summarize." The cost of sending the same material repeatedly can be kept down with prompt caching.
For numbers and lists, ask the database
The answer to "How many expense claims were filed last month?" is in the expense system's database, not in a document. Here it is more reliable to have the AI build an SQL query or API call and ask the database directly. Searching monthly reports with RAG risks picking up figures from an older month.
For fixed-form questions, ordinary search or an FAQ is enough
For things that only need an exact string match, such as model numbers, article numbers, or employee IDs, classic keyword search is more accurate. If 80% of inquiries are the same 20 questions, a well-organized FAQ page may be cheaper and faster.
For actions, give the AI tools
If you want the AI to carry out business operations, such as reserving stock or making a booking, give it those operations as tools instead of documents. That is a different topic from RAG, and belongs to mechanisms like MCP.
When RAG is still the right choice
- You have thousands of pages or more and cannot pass them all
- Material is added often, and you do not want to rebuild everything each time
- Most questions are of the "find what is written somewhere" type
- Different departments may see different documents, and you want to filter at the search stage
Even then, accuracy improves with better splitting, combining keyword search, and reordering the selected chunks (reranking). In the same Anthropic article, adding context from the whole document to each chunk before embedding, combined with keyword search, cut retrieval failures by 49%, and adding reranking cut them by 67%. In other words, most of the improvements are on the search side, not the AI side.
Summary: five questions for choosing an approach
Before having AI answer questions about your company's data, check these five points from top to bottom, and the approach will follow naturally.
| What to check | If | Choose |
|---|---|---|
| Is the answer in documents or in business system data? | It is in data: counts, amounts, lists | Query the database or API |
| Is the volume small enough to pass in full? | A few hundred pages or less | Try passing everything first |
| Do questions find one place, or count and compare? | They count, compare, or summarize the whole | Pass everything, or prepare aggregates separately |
| Are many searches by number or model? | Yes | Add keyword search |
| Can people check the basis of each answer? | No | Show the source chunks with the answer |
If none of these apply, and you have a large amount of material with mostly "find what is written somewhere" questions, that is when RAG comes in.
RAG is a useful tool, but only one tool. Once you know it simply splits, embeds, and picks the nearest, you can choose the method that fits your data and your questions.
At Vitalify Asia, we build generative AI into business systems and services. We have also developed approaches like the ones in this article for having AI answer questions from internal data. If you are thinking about using your documents and data with AI, feel free to contact us.
Since 2008, VFA has been involved in more than 1,000 development projects. Many of them are not public, so please feel free to ask us for details.
Vitalify Asia development portfolio
Each service's specifications are updated continuously. This article reflects information as of November 20, 2026.
Vitalify Asia Co., Ltd. https://vitalify.asia/
Struggling to turn ideas into reality? With a proven track record of over 1,000 clients, our agile and flexible team will accelerate your business growth.


