Every organisation has the same quiet problem: the answers people need are somewhere in a PDF, a shared drive, an old email thread or a policy document nobody can find. New employees ask colleagues. Experienced staff waste time searching. Customers wait while support teams look things up.
Public AI chatbots don't help much here, because they don't know your internal documents — and when they don't know, they sometimes invent plausible-sounding answers. Retrieval-augmented generation, or RAG, is the technique that fixes this. It is how most businesses build a reliable private ChatGPT on their own data.
RAG Explained: What It Is
RAG combines two steps. First, retrieval: when someone asks a question, the system searches your approved documents and finds the most relevant passages. Second, generation: an AI language model writes an answer using only those passages, and shows which documents it used.
Think of it as an open-book exam. Instead of asking the model to remember everything, you hand it the right pages and ask it to answer from them. That makes answers specific to your business, up to date and checkable.
Why Not Just Fine-Tune a Model?
Fine-tuning means training a model further on your data. It is useful for teaching style or specialised formats, but it is a poor way to store facts. Fine-tuned models can still misremember details, are expensive to update when documents change, and can't easily cite sources or enforce who may see what. RAG keeps knowledge in your document store, so updating an answer is as simple as updating the document — and access control stays in your hands.
How a RAG System Works, Step by Step
1. Collect and prepare documents
Gather the sources the assistant may use: policy PDFs, SOPs, product manuals, wiki pages, help-centre articles, spreadsheets and database records. Clean them up — remove duplicates and outdated versions, and extract text properly from scans and tables. Garbage in still means garbage out.
2. Split into chunks
Documents are divided into smaller passages, called chunks, that each cover one idea — a policy clause, a product specification, a procedure step. Chunking well matters: too small and context is lost; too large and retrieval becomes imprecise. Good chunking follows the document's own structure, such as headings and sections.
3. Create embeddings and index them
Each chunk is converted into an embedding — a list of numbers that captures its meaning — and stored in a vector database along with metadata such as source, date, department and access level. Many systems also keep a keyword index, because exact terms like product codes and policy numbers matter.
4. Retrieve relevant passages
When a user asks a question, it is converted into an embedding and matched against the index, often combined with keyword search (hybrid search). A reranking step then orders the candidates by relevance, and permission filters remove anything the user is not allowed to see.
5. Generate a grounded answer
The top passages are sent to the language model with instructions: answer only from these sources, cite them, and say clearly if the answer isn't there. The user sees the answer with links to the original documents.
What Makes RAG Good or Bad
Building a basic RAG demo takes an afternoon. Building one people trust takes care in a few areas.
- Retrieval quality: most wrong answers come from retrieving the wrong passages, not from the model. Hybrid search and reranking help a lot.
- Fresh content: documents change, so re-index on a schedule or when files are updated, and prefer the latest version.
- Permissions: an HR assistant must not reveal salary data to everyone. Filter results by the user's existing access rights.
- Honest uncertainty: the assistant should say "I couldn't find that in our documents" rather than guess.
- Citations: showing sources lets people verify answers and builds trust quickly.
- Tables and numbers: spreadsheets and tables need special handling so figures aren't garbled.
Business Use Cases for RAG
- HR and policy assistant: leave rules, reimbursements and procedures answered from official documents.
- Customer support knowledge base: agents or customers get answers from manuals and help articles.
- Sales enablement: product specifications, pricing rules and case studies on demand.
- Compliance and audit: search across circulars, SOPs and regulations with exact references.
- Engineering and operations: troubleshooting guides and maintenance manuals for technicians.
- Onboarding: new employees ask questions instead of reading hundreds of pages.
Keeping Your Data Private
A private ChatGPT for business is only private if the architecture is. Key questions to settle before building:
- Where is the document index stored, and who controls it?
- Which AI model provider processes questions, and is it configured not to retain or train on your data?
- Does data need to stay in India or on your own servers for regulatory reasons? Open-source models can run on infrastructure you control.
- How are personal data and DPDP Act obligations handled for documents that contain them?
How to Measure a RAG Assistant
Before launch, build an evaluation set of 50–100 real questions with known correct answers and sources. Measure whether the right passages were retrieved, whether answers are correct and faithful to the sources, and how often the assistant appropriately says it doesn't know. After launch, track user feedback, unanswered questions and citation clicks. This is part of the LLM evaluation and observability work we include in every custom AI build.
Common Mistakes to Avoid
- Indexing everything at once, including outdated and duplicate documents.
- Ignoring permissions until after launch.
- Judging quality from a few demo questions instead of a real evaluation set.
- Hiding sources, which makes it impossible for users to verify answers.
- Treating launch as the end instead of the start of continuous improvement.
RAG and AI Agents
RAG is also the knowledge layer for AI agents. An agent that processes a claim or answers a customer can retrieve the relevant policy before acting, which keeps its decisions grounded. Many projects start with a RAG assistant and later add agentic actions once the knowledge base is trusted.
Getting Started
- Pick one department and one clear audience — for example, HR questions from employees.
- Collect the 50–200 documents that answer most of their questions.
- Gather real questions people ask today to build your evaluation set.
- Build, evaluate, fix retrieval issues, then launch to a pilot group.
- Expand sources and audiences once quality is proven.
How Long Does It Take, and What Does It Cost?
A focused RAG assistant for one department, with a few hundred well-organised documents, can usually be built and evaluated in a few weeks. Timelines grow with the number and messiness of sources, the need for strict permissions, integrations with tools like Slack, Microsoft Teams or your intranet, and how much evaluation and tuning is required.
Running costs have three parts: hosting the document index and application, the AI model usage per question (which depends on model choice and how much context is sent), and maintenance as documents and needs change. Sensible design — retrieving only the most relevant passages, choosing an appropriately sized model and caching common questions — keeps per-question costs low. We give a clear estimate after reviewing your sources and expected usage.
One cost that is easy to overlook is content ownership. Someone in each department should be responsible for keeping source documents current and retiring outdated ones. Assistants stay accurate when the documents behind them are well maintained, and that ongoing effort is usually small compared with the time the assistant saves.
Want your team to chat with your company documents securely? Explore our RAG development services or book a free consultation.