RAG chatbots: how they work, what they cost and when a business needs one

What is a RAG chatbot?

A RAG chatbot is an AI assistant that looks up the answer in your own content before it replies. RAG stands for retrieval-augmented generation, a method described in a 2020 research paper from Facebook AI Research. The name says what happens: retrieve the relevant passages, then generate an answer from them.

In practice it works in three steps. Your documents are split into short passages and indexed for search by meaning, not just keywords. When a customer asks something, the system finds the few passages that match. The language model (from OpenAI or Anthropic, for example) then writes the answer using only those passages, and can show which page it came from.

RAG vs a ChatGPT wrapper vs fine-tuning

A plain ChatGPT wrapper sends the customer's question to a model with a short instruction like "you are the assistant for Acme Plumbing". The model answers from its general training, so it doesn't know your prices, hours or return policy, and it will fill the gaps with plausible guesses. That's fine for small talk and risky for anything a customer will act on.

Fine-tuning retrains a model on your examples so it learns a style or a task format. It's good at teaching tone and structure, but it's a poor way to teach facts: updating a price means retraining, and the model still can't point to a source. RAG keeps the facts outside the model, so you edit a document and the next answer uses the new version.

For most business chatbots, RAG is the right base. Fine-tuning is occasionally added on top for very specific formats, and many projects never need it.

What data can a RAG chatbot use?

Almost any text your team already uses to answer customers. Common sources are help center articles, FAQ pages, PDFs (manuals, brochures, price lists), policies on returns, shipping or cancellations, and internal procedures your staff follow.

Structured data works too, but differently. A product catalog, stock levels or order status change every day, so instead of indexing them as documents we usually connect the chatbot to the live system (Shopify, your CRM or a database) and let it look them up at question time. That combination of documents plus live lookups is what turns a chatbot into an AI agent.

The quality of the answers depends on the quality of the sources. If two PDFs give different refund rules, the chatbot will be confused in the same way a new employee would. Part of every project is cleaning up and choosing the approved sources.

How accurate is a RAG chatbot?

More accurate than a plain model, but not perfect. RAG cuts down invented answers because the model is told to use only the retrieved passages. It can still pick the wrong passage, miss a detail spread across two documents, or phrase something loosely.

Good builds add guardrails. The chatbot cites the page or document behind each answer so customers and staff can check it. It says "I don't know" when the sources don't cover the question, instead of guessing. It hands the conversation to a person on WhatsApp, email or live chat for complaints, refunds, medical or legal questions, or when the customer asks for one. And every conversation is logged, so someone reviews real questions and fills the gaps in the sources.

How much does a RAG chatbot cost?

There are two kinds of cost: building it and running it. With us, a RAG chatbot for your website or WhatsApp starts at USD 2,000 to build, including source preparation, the assistant, handoff rules and testing. An AI agent that also reads and acts on your CRM, ERP or database starts at USD 6,000.

Running costs come from two places. The model API is billed per token (roughly, per word processed) by providers like OpenAI and Anthropic, so it grows with the number and length of conversations. The search index needs storage, either a managed vector database or PostgreSQL with the pgvector extension, which often runs on infrastructure you already pay for. Check each provider's current pricing page before budgeting, because rates change often.

Our chatbots also carry a monthly plan for hosting, model usage and improvements based on real conversations. The table below shows the current starting prices.

When a business doesn't need a RAG chatbot

If customers ask the same ten questions, a well-written FAQ page and a clear contact button will do the job for free. If your help content fits in a couple of pages, a simpler chatbot with that text in its instructions can work without a retrieval layer. And if most conversations need a human decision (custom quotes, complaints, sensitive cases), a shared inbox with good templates beats any bot.

A RAG chatbot pays off when you have a lot of content, questions arrive outside business hours, and the same answers are buried across documents your team keeps searching. If you're not sure which side you're on, our AI consulting service starts with exactly that assessment.