RAG (retrieval-augmented generation) is a chatbot architecture that retrieves relevant content from your own knowledge base and provides it to the language model as context before it generates a response — instead of relying only on the model's general training knowledge.
Why RAG exists
A language model's training data has a cutoff date and doesn't include your specific business content — your pricing, your policies, your product details. Without RAG, a chatbot either answers generically or, worse, generates a plausible-sounding but incorrect answer. RAG solves this by giving the model your actual content as context at the moment it generates a response.
The RAG pipeline, step by step
1. Index your content → break docs into chunks, generate embeddings, store them
2. User asks a question → embed the question the same way
3. Retrieve → find the most similar stored chunks (vector similarity search)
4. Augment → insert those chunks into the model's context/prompt
5. Generate → model answers using the retrieved chunks as grounding
1. Indexing
Your source content (FAQ, docs, policies) is split into chunks — typically a few paragraphs each — and each chunk is converted into an embedding (a numeric representation of its meaning) and stored, usually in a vector database.
2–3. Retrieval
When a user asks a question, it's embedded the same way, and the system finds the stored chunks whose embeddings are most similar (typically via cosine similarity) — these are the chunks most likely to be relevant to the question.
4–5. Augmented generation
The retrieved chunks are inserted into the prompt sent to the model, along with an instruction like "answer only using the provided context." The model generates a response grounded in that specific content, not just its general training.
Why this matters for accuracy
Without retrieval, asking a chatbot "what's your refund policy?" relies entirely on the model either knowing nothing (and saying so, ideally) or guessing based on generic patterns from training data. With RAG, the actual text of your refund policy is provided as context, so the model can quote or summarize it directly — dramatically reducing fabricated answers on business-specific questions.
Common RAG design decisions
| Decision | Trade-off |
|---|---|
| Chunk size | Smaller chunks retrieve more precisely but may lose surrounding context; larger chunks keep context but retrieve less precisely |
| Number of chunks retrieved | More chunks give the model more to work with but increase cost and can dilute relevance |
| Chunking strategy | Splitting by heading/section usually preserves meaning better than fixed-length splitting |
| Freshness | Content changes require re-indexing — stale indexes cause stale (or wrong) answers |
A minimal RAG implementation sketch
async function answerQuestion(question: string, knowledgeBase: Chunk[]) {
const questionEmbedding = await embed(question);
const relevantChunks = knowledgeBase
.map((chunk) => ({ chunk, score: cosineSimilarity(questionEmbedding, chunk.embedding) }))
.sort((a, b) => b.score - a.score)
.slice(0, 4)
.map((r) => r.chunk.text);
const prompt = `Answer only using the context below. If the answer isn't here, say you don't know.
Context:
${relevantChunks.join("\n\n")}
Question: ${question}`;
return callModel(prompt);
}
This is the same underlying pattern used in How to Build an AI Chatbot, described there at the implementation level.
Limitations of RAG
- Retrieval quality caps answer quality — if the relevant chunk isn't retrieved, the model can't use it, no matter how good the model is.
- Doesn't eliminate fabrication entirely — a model can still misinterpret retrieved context; explicit "say you don't know" instructions and lower temperature settings help but aren't a complete guarantee.
- Requires maintenance — your knowledge base needs to stay current, or the chatbot will confidently answer from outdated information.