RAG: Retrieval-Augmented Generation
Combining LLMs with external knowledge
Skills taught: llms.rag
What is RAG?
RAG grounds LLM responses in real, up-to-date documents. Instead of relying on training data, the model retrieves relevant chunks from your document store and answers from them — dramatically reducing hallucination for domain-specific queries.
- RAG Pipeline
- Index phase: Chunk documents → embed chunks → store in vector DB. Query phase: Embed user question → retrieve top-k similar chunks → inject into prompt → LLM answers from context.
- Chunk
- A fixed-size or semantically-bounded piece of a document (typically 300-1000 tokens). Documents are split into chunks before embedding so retrieval returns focused, relevant snippets rather than entire documents.
Phase 1: Indexing Documents
# pip install anthropic openai chromadb tiktoken
import chromadb
from openai import OpenAI
import tiktoken
client = OpenAI()
vector_client = chromadb.Client()
collection = vector_client.create_collection("company_docs")
def chunk_text(text: str, max_tokens: int = 500) -> list[str]:
"""Split text into chunks of max_tokens."""
enc = tiktoken.encoding_for_model("gpt-4")
words = text.split()
chunks, current, current_tokens = [], [], 0
for word in words:
word_tokens = len(enc.encode(word + " "))
if current_tokens + word_tokens > max_tokens:
chunks.append(" ".join(current))
current, current_tokens = [word], word_tokens
else:
current.append(word)
current_tokens += word_tokens
if current:
chunks.append(" ".join(current))
return chunks
def embed(texts: list[str]) -> list[list[float]]:
response = client.embeddings.create(
input=texts, model="text-embedding-3-small"
)
return [d.embedding for d in response.data]
# Index documents
docs = [
"Our refund policy: Customers can return items within 30 days for a full refund.",
"Shipping: Standard delivery takes 5-7 business days. Express is 2 business days.",
"Warranty: All electronics carry a 1-year manufacturer warranty."
]
chunked = [(chunk, f"doc_{i}_{j}")
for i, doc in enumerate(docs)
for j, chunk in enumerate(chunk_text(doc))]
texts, ids = zip(*chunked)
collection.add(
documents=list(texts),
embeddings=embed(list(texts)),
ids=list(ids)
)
print(f"Indexed {len(ids)} chunks")
Phase 2: Querying with RAG
import anthropic
from openai import OpenAI
client = OpenAI()
anthro = anthropic.Anthropic()
def rag_query(question: str, top_k: int = 3) -> str:
# 1. Embed the question
q_embedding = client.embeddings.create(
input=question, model="text-embedding-3-small"
).data[0].embedding
# 2. Retrieve top-k most similar chunks
results = collection.query(
query_embeddings=[q_embedding],
n_results=top_k
)
retrieved_chunks = results["documents"][0]
# 3. Build context-augmented prompt
context = "\n\n".join(f"[{i+1}] {chunk}" for i, chunk in enumerate(retrieved_chunks))
# 4. Generate grounded answer
response = anthro.messages.create(
model="claude-haiku-5-5",
max_tokens=512,
system="""You are a helpful assistant. Answer questions using ONLY the provided context.
If the context does not contain the answer, say 'I don't have information about that.'
Do not use your training knowledge.""",
messages=[{"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"}]
)
return response.content[0].text
print(rag_query("What is the return policy?"))
# Returns answer grounded in the indexed documents
RAG vs Fine-Tuning
RAG vs Fine-Tuning for Domain Knowledge
- RAG: Retrieves relevant docs at query time and injects into prompt
- Fine-Tuning: Trains the model weights on domain-specific data
Tip: Always instruct the LLM to use only the provided context and to say 'I don't know' if the answer isn't there. Without this, models will fall back to training knowledge and hallucinate.
Warning: Chunk size matters enormously. Too small (100 tokens): context is fragmented and lacks meaning. Too large (2000 tokens): the retrieved chunk buries the relevant information. Sweet spot: 300-800 tokens with 10-20% overlap.
Frequently Asked Questions
What is a chunk?
A fixed-size piece of a document (e.g., 500 tokens). Documents are split into chunks before embedding. Retrieval returns focused snippets rather than whole documents. Use 10-20% overlap between adjacent chunks to avoid splitting context mid-sentence.
What embedding model should I use?
For English: text-embedding-3-small (OpenAI) is fast and cheap (1536 dims). For multilingual: multilingual-e5-large or Cohere embed v3. For code: code-search-* variants. Always use the same model for indexing and querying.
How do I evaluate RAG quality?
Two key metrics: (1) Retrieval recall — are the right chunks retrieved? Measure by checking if the gold-standard answer chunks are in top-k results. (2) Faithfulness — is the LLM answer supported by retrieved chunks? Frameworks like RAGAS automate this.