A large language model only knows what was in its training data. Ask it about your internal handbook, last week’s release notes, or a 400-page contract and it will guess. Retrieval-Augmented Generation (RAG) fixes this. You fetch the relevant passages at query time and hand them to the model as context. This guide builds a complete RAG pipeline in about 80 lines of Python, with no framework magic hiding the moving parts.
Quick Takeaways
- RAG = retrieve, then generate. Embed your documents once. At query time, embed the question, find the nearest chunks, and pass them to the LLM.
- Chunking decides quality. This build uses 200-word chunks with a 40-word overlap, which keeps each chunk under the embedding model’s 512-token limit.
- You don’t need a vector database to start. With normalized embeddings, cosine similarity is a single NumPy dot product. That is fast enough for tens of thousands of chunks.
- Grounding rules matter as much as retrieval. A strict system prompt (“answer only from context, cite sources, say ‘I don’t know'”) is your main defense against hallucination.
| Component | Choice in this guide | Why |
|---|---|---|
| Embedding model | BAAI/bge-small-en-v1.5 |
384 dimensions, 512-token input, runs on CPU, free |
| Vector store | NumPy array + JSON | Zero infrastructure |
| Similarity | Cosine (dot product on unit vectors) | Standard for text embeddings |
| Generator | claude-sonnet-5-5 via Anthropic API |
Strong instruction following for grounded answers |
| Retrieval depth | top-k = 4 | Balances recall against context noise |
How a RAG Pipeline Works
A RAG system has two phases.
Indexing (offline, run once per document change):
- Load raw documents.
- Split them into chunks.
- Convert each chunk to an embedding vector.
- Store the vectors next to the original text.
Querying (online, run per question):
- Embed the user’s question with the same model.
- Score every stored vector against the question vector.
- Take the top-k chunks.
- Build a prompt with those chunks and ask the LLM to answer from them only.
Everything else in RAG engineering (rerankers, hybrid search, query rewriting) is an upgrade to one of these steps.
Step 1: Set Up the Project
Create a folder with a docs/ subfolder. Drop in a few .txt or .md files. Your own notes, a product manual, or exported wiki pages all work.
mkdir rag-scratch && cd rag-scratch
mkdir docs
python -m venv .venv && source .venv/bin/activate
pip install sentence-transformers numpy anthropic
export ANTHROPIC_API_KEY="your-key-here"
Anthropic does not offer its own embedding model, so this build uses an open-source one locally. That keeps document text on your machine during indexing. Only the retrieved chunks and the question go to the API.
Step 2: Load and Chunk Documents
Embedding models work best on focused passages. A whole 20-page file blurs into one vector. A single sentence loses context. Chunks of 150 to 300 words are a practical middle ground.
Overlap prevents a fact from being cut in half at a chunk boundary. With 40 words of overlap, the end of one chunk repeats at the start of the next.
# rag.py
import json
from pathlib import Path
import numpy as np
from sentence_transformers import SentenceTransformer
import anthropic
EMBED_MODEL = "BAAI/bge-small-en-v1.5"
QUERY_PREFIX = "Represent this sentence for searching relevant passages: "
CHUNK_WORDS = 200
OVERLAP_WORDS = 40
TOP_K = 4
LLM = "claude-sonnet-5-5"
def load_docs(folder):
docs = []
for p in Path(folder).rglob("*"):
if p.suffix.lower() in {".txt", ".md"}:
docs.append({"source": p.name, "text": p.read_text(encoding="utf-8")})
return docs
def chunk(text, size=CHUNK_WORDS, overlap=OVERLAP_WORDS):
words = text.split()
step = size - overlap
return [
" ".join(words[i:i + size])
for i in range(0, max(len(words) - overlap, 1), step)
]
Word counts approximate tokens at roughly 1 word ≈ 1.3 tokens. A 200-word chunk lands near 260 tokens, well under the 512-token cap.
Upgrade path: Split on headings or paragraphs first, then merge to the target size. Structure-aware chunking beats fixed windows on manuals and wikis.
Step 3: Create Embeddings
An embedding model maps text to a point in vector space. Similar meaning lands close together. bge-small-en-v1.5 outputs 384-dimensional vectors.
Two settings matter here:
normalize_embeddings=Truescales each vector to length 1. Cosine similarity then equals a plain dot product.- The query prefix. BGE models were trained so that questions get an instruction prefix while documents do not. Skipping it measurably hurts retrieval.
def build_index(folder="docs"):
model = SentenceTransformer(EMBED_MODEL)
records = []
for doc in load_docs(folder):
for i, c in enumerate(chunk(doc["text"])):
records.append({"source": doc["source"], "chunk_id": i, "text": c})
vectors = model.encode(
[r["text"] for r in records],
normalize_embeddings=True,
batch_size=32,
show_progress_bar=True,
)
np.save("index.npy", vectors.astype("float32"))
Path("chunks.json").write_text(json.dumps(records), encoding="utf-8")
print(f"Indexed {len(records)} chunks, shape {vectors.shape}")
For 10,000 chunks, the matrix holds 10,000 × 384 float32 values, about 15 MB. It fits in memory with room to spare.
Embedding Model Options
| Model | Dimensions | Hosting | Cost | Best for |
|---|---|---|---|---|
bge-small-en-v1.5 |
384 | Local, open source | Free | Prototypes, private data, CPU |
all-MiniLM-L6-v2 |
384 | Local, open source | Free | Fastest baseline; truncates near 256 tokens |
OpenAI text-embedding-3-small |
1,536 | API, closed | ~$0.02 / 1M tokens | Easy hosted option |
OpenAI text-embedding-3-large |
3,072 | API, closed | ~$0.13 / 1M tokens | Higher accuracy, bigger storage |
Check each provider’s pricing page before budgeting. List prices change.
Step 4: Retrieve the Best Chunks
Retrieval is one matrix-vector product. Each stored vector gets a similarity score against the query vector. Sort descending and slice the top k.
def retrieve(query, model, vectors, records, k=TOP_K):
q = model.encode([QUERY_PREFIX + query], normalize_embeddings=True)[0]
scores = vectors @ q # cosine similarity
top = np.argsort(-scores)[:k]
return [{**records[i], "score": float(scores[i])} for i in top]
Choosing k. Too low and the answer chunk may be missing. Too high and irrelevant text dilutes the prompt. Start at 4, then measure (see the evaluation section).
Score thresholds. Cosine scores for BGE models on relevant matches often sit above roughly 0.5, but the range varies by corpus. Print scores on your own data before choosing a cutoff.
Step 5: Generate a Grounded Answer
Now hand the retrieved chunks to the LLM. The system prompt does the heavy lifting. It sets three rules: use only the context, cite chunk numbers, and admit when the answer is absent.
SYSTEM = """You answer questions using only the provided context.
Rules:
1. If the context does not contain the answer, reply exactly: "I don't know based on the provided documents."
2. Cite the supporting chunks inline as [1], [2], matching the context numbers.
3. Do not use outside knowledge. Keep answers under 200 words."""
def answer(query, hits):
context = "\n\n".join(
f"[{n}] (source: {h['source']})\n{h['text']}"
for n, h in enumerate(hits, 1)
)
client = anthropic.Anthropic()
msg = client.messages.create(
model=LLM,
max_tokens=1024,
temperature=0.2,
system=SYSTEM,
messages=[{
"role": "user",
"content": f"<context>\n{context}\n</context>\n\nQuestion: {query}",
}],
)
return msg.content[0].text
Settings explained:
temperature=0.2keeps output close to the source text. Factual Q&A rarely benefits from high randomness.max_tokens=1024caps response length and cost.- XML-style
<context>tags separate retrieved material from the question. Claude models handle tagged structure well.
If the model you select rejects a sampling parameter, drop temperature and rely on the system prompt.
Step 6: Wire It Together
def main():
import sys
if len(sys.argv) > 1 and sys.argv[1] == "index":
build_index()
return
model = SentenceTransformer(EMBED_MODEL)
vectors = np.load("index.npy")
records = json.loads(Path("chunks.json").read_text(encoding="utf-8"))
while True:
q = input("\nAsk (blank to quit): ").strip()
if not q:
break
hits = retrieve(q, model, vectors, records)
print("\nRetrieved:", [(h["source"], round(h["score"], 2)) for h in hits])
print("\n" + answer(q, hits))
if __name__ == "__main__":
main()
Run it:
python rag.py index # build the index once
python rag.py # start asking questions
The retrieved-sources printout is deliberate. When an answer looks wrong, check retrieval first. Most RAG failures are retrieval failures, not generation failures.
Step 7: Measure Retrieval Quality
Do not tune by gut feel. Build a small evaluation set of 20 to 50 questions, each paired with the source file that contains the answer. Then compute hit rate@k: the share of questions where the correct source appears in the top k.
def hit_rate(eval_set, model, vectors, records, k=4):
hits = 0
for question, expected_source in eval_set:
results = retrieve(question, model, vectors, records, k)
if any(r["source"] == expected_source for r in results):
hits += 1
return hits / len(eval_set)
eval_set = [
("What is the refund window?", "policy.md"),
("How do I rotate API keys?", "security.md"),
]
Change one variable at a time (chunk size, overlap, k, embedding model) and rerun. A move from 200-word to 120-word chunks might lift hit rate by several points on fact-heavy docs and hurt it on narrative ones. Only your data can tell you.
Common Failure Modes and Fixes
| Symptom | Likely cause | Fix |
|---|---|---|
| Right file, wrong passage | Chunks too large | Drop to 100–150 words |
| Answer split across chunks | No overlap | Raise overlap to 20% |
| Keyword queries miss (IDs, error codes) | Dense vectors miss exact tokens | Add BM25 hybrid search |
| Confident wrong answers | Weak grounding prompt | Enforce “I don’t know” rule, lower temperature |
| Good chunks, ignored by model | Too many chunks, noise | Lower k, add a reranker |
| Slow queries at 1M+ chunks | Brute-force scan | Move to FAISS, HNSW, or a vector DB |
Real-World Use Cases
Internal support assistant. Index HR policies, IT runbooks, and onboarding docs. Staff ask “How do I request a laptop replacement?” and get an answer with citations to the exact policy file. The “I don’t know” rule routes gaps to a human instead of inventing a procedure.
Contract and compliance lookup. Chunk contracts by clause. Store the clause number in the record metadata. Answers cite clause IDs, so a reviewer verifies each claim in seconds.
Developer documentation bot. Index API docs and changelogs. Add a version field to each record and filter by version before scoring. This stops the bot from mixing v1 and v2 behavior.
Advanced prompt template for stricter grounding:
Answer the question using only the context below.
First, list the context numbers that contain relevant facts.
Then write the answer, citing each claim with [n].
If no chunk is relevant, output: I don't know based on the provided documents.
Forcing the model to name its evidence first reduces unsupported claims.
Scaling Beyond This Build
Brute-force NumPy search stays comfortable up to roughly 50,000 to 100,000 chunks on a laptop. Past that, or when you need metadata filters and live updates, upgrade in this order:
- Hybrid search. Combine dense scores with BM25 keyword scores. This fixes exact-term misses.
- Reranking. Retrieve top 20, then rerank with a cross-encoder and keep the top 4.
- Approximate nearest neighbor. Use FAISS or HNSW indexes for sub-linear search.
- Managed vector store. Use pgvector, Qdrant, Weaviate, or Pinecone for persistence, filtering, and concurrent writes.
FAQ
What is a RAG app in simple terms?
A RAG app searches your documents for passages relevant to a question, then gives those passages to an LLM so it can answer from them. The model reads your data at query time instead of relying on memory.
Do I need a vector database to build RAG?
No. For small and mid-sized corpora, a NumPy array of normalized embeddings and a dot product handle retrieval. Add a vector database when you need scale, metadata filtering, or frequent updates.
What is the best chunk size for RAG?
Start with 150 to 300 words and 10–20% overlap. Then test against a labeled question set. Fact-dense documents favor smaller chunks. Narrative documents favor larger ones.
How do I stop a RAG app from hallucinating?
Use a strict system prompt that limits answers to the retrieved context and defines a fixed “I don’t know” reply. Add inline citations, keep temperature low, and check retrieved chunks first when an answer looks wrong.



