Build a RAG App From Scratch: Documents, Embeddings, and Answers

A large language model only knows what was in its training data. Ask it about your internal handbook, last week’s release notes, or a 400-page contract and it will guess. Retrieval-Augmented Generation (RAG) fixes this. You fetch the relevant passages at query time and hand them to the model as context. This guide builds a complete RAG pipeline in about 80 lines of Python, with no framework magic hiding the moving parts.

Quick Takeaways

  • RAG = retrieve, then generate. Embed your documents once. At query time, embed the question, find the nearest chunks, and pass them to the LLM.
  • Chunking decides quality. This build uses 200-word chunks with a 40-word overlap, which keeps each chunk under the embedding model’s 512-token limit.
  • You don’t need a vector database to start. With normalized embeddings, cosine similarity is a single NumPy dot product. That is fast enough for tens of thousands of chunks.
  • Grounding rules matter as much as retrieval. A strict system prompt (“answer only from context, cite sources, say ‘I don’t know'”) is your main defense against hallucination.
Component Choice in this guide Why
Embedding model BAAI/bge-small-en-v1.5 384 dimensions, 512-token input, runs on CPU, free
Vector store NumPy array + JSON Zero infrastructure
Similarity Cosine (dot product on unit vectors) Standard for text embeddings
Generator claude-sonnet-5-5 via Anthropic API Strong instruction following for grounded answers
Retrieval depth top-k = 4 Balances recall against context noise

How a RAG Pipeline Works

A RAG system has two phases.

Indexing (offline, run once per document change):

  1. Load raw documents.
  2. Split them into chunks.
  3. Convert each chunk to an embedding vector.
  4. Store the vectors next to the original text.

Querying (online, run per question):

  1. Embed the user’s question with the same model.
  2. Score every stored vector against the question vector.
  3. Take the top-k chunks.
  4. Build a prompt with those chunks and ask the LLM to answer from them only.

Everything else in RAG engineering (rerankers, hybrid search, query rewriting) is an upgrade to one of these steps.

Step 1: Set Up the Project

Create a folder with a docs/ subfolder. Drop in a few .txt or .md files. Your own notes, a product manual, or exported wiki pages all work.

mkdir rag-scratch && cd rag-scratch
mkdir docs
python -m venv .venv && source .venv/bin/activate
pip install sentence-transformers numpy anthropic
export ANTHROPIC_API_KEY="your-key-here"

Anthropic does not offer its own embedding model, so this build uses an open-source one locally. That keeps document text on your machine during indexing. Only the retrieved chunks and the question go to the API.

Step 2: Load and Chunk Documents

Embedding models work best on focused passages. A whole 20-page file blurs into one vector. A single sentence loses context. Chunks of 150 to 300 words are a practical middle ground.

Overlap prevents a fact from being cut in half at a chunk boundary. With 40 words of overlap, the end of one chunk repeats at the start of the next.

# rag.py
import json
from pathlib import Path

import numpy as np
from sentence_transformers import SentenceTransformer
import anthropic

EMBED_MODEL = "BAAI/bge-small-en-v1.5"
QUERY_PREFIX = "Represent this sentence for searching relevant passages: "
CHUNK_WORDS = 200
OVERLAP_WORDS = 40
TOP_K = 4
LLM = "claude-sonnet-5-5"


def load_docs(folder):
    docs = []
    for p in Path(folder).rglob("*"):
        if p.suffix.lower() in {".txt", ".md"}:
            docs.append({"source": p.name, "text": p.read_text(encoding="utf-8")})
    return docs


def chunk(text, size=CHUNK_WORDS, overlap=OVERLAP_WORDS):
    words = text.split()
    step = size - overlap
    return [
        " ".join(words[i:i + size])
        for i in range(0, max(len(words) - overlap, 1), step)
    ]

Word counts approximate tokens at roughly 1 word ≈ 1.3 tokens. A 200-word chunk lands near 260 tokens, well under the 512-token cap.

Upgrade path: Split on headings or paragraphs first, then merge to the target size. Structure-aware chunking beats fixed windows on manuals and wikis.

Step 3: Create Embeddings

An embedding model maps text to a point in vector space. Similar meaning lands close together. bge-small-en-v1.5 outputs 384-dimensional vectors.

Two settings matter here:

  • normalize_embeddings=True scales each vector to length 1. Cosine similarity then equals a plain dot product.
  • The query prefix. BGE models were trained so that questions get an instruction prefix while documents do not. Skipping it measurably hurts retrieval.
def build_index(folder="docs"):
    model = SentenceTransformer(EMBED_MODEL)
    records = []
    for doc in load_docs(folder):
        for i, c in enumerate(chunk(doc["text"])):
            records.append({"source": doc["source"], "chunk_id": i, "text": c})

    vectors = model.encode(
        [r["text"] for r in records],
        normalize_embeddings=True,
        batch_size=32,
        show_progress_bar=True,
    )
    np.save("index.npy", vectors.astype("float32"))
    Path("chunks.json").write_text(json.dumps(records), encoding="utf-8")
    print(f"Indexed {len(records)} chunks, shape {vectors.shape}")

For 10,000 chunks, the matrix holds 10,000 × 384 float32 values, about 15 MB. It fits in memory with room to spare.

Embedding Model Options

Model Dimensions Hosting Cost Best for
bge-small-en-v1.5 384 Local, open source Free Prototypes, private data, CPU
all-MiniLM-L6-v2 384 Local, open source Free Fastest baseline; truncates near 256 tokens
OpenAI text-embedding-3-small 1,536 API, closed ~$0.02 / 1M tokens Easy hosted option
OpenAI text-embedding-3-large 3,072 API, closed ~$0.13 / 1M tokens Higher accuracy, bigger storage

Check each provider’s pricing page before budgeting. List prices change.

Step 4: Retrieve the Best Chunks

Retrieval is one matrix-vector product. Each stored vector gets a similarity score against the query vector. Sort descending and slice the top k.

def retrieve(query, model, vectors, records, k=TOP_K):
    q = model.encode([QUERY_PREFIX + query], normalize_embeddings=True)[0]
    scores = vectors @ q                      # cosine similarity
    top = np.argsort(-scores)[:k]
    return [{**records[i], "score": float(scores[i])} for i in top]

Choosing k. Too low and the answer chunk may be missing. Too high and irrelevant text dilutes the prompt. Start at 4, then measure (see the evaluation section).

Score thresholds. Cosine scores for BGE models on relevant matches often sit above roughly 0.5, but the range varies by corpus. Print scores on your own data before choosing a cutoff.

Step 5: Generate a Grounded Answer

Now hand the retrieved chunks to the LLM. The system prompt does the heavy lifting. It sets three rules: use only the context, cite chunk numbers, and admit when the answer is absent.

SYSTEM = """You answer questions using only the provided context.
Rules:
1. If the context does not contain the answer, reply exactly: "I don't know based on the provided documents."
2. Cite the supporting chunks inline as [1], [2], matching the context numbers.
3. Do not use outside knowledge. Keep answers under 200 words."""


def answer(query, hits):
    context = "\n\n".join(
        f"[{n}] (source: {h['source']})\n{h['text']}"
        for n, h in enumerate(hits, 1)
    )
    client = anthropic.Anthropic()
    msg = client.messages.create(
        model=LLM,
        max_tokens=1024,
        temperature=0.2,
        system=SYSTEM,
        messages=[{
            "role": "user",
            "content": f"<context>\n{context}\n</context>\n\nQuestion: {query}",
        }],
    )
    return msg.content[0].text

Settings explained:

  • temperature=0.2 keeps output close to the source text. Factual Q&A rarely benefits from high randomness.
  • max_tokens=1024 caps response length and cost.
  • XML-style <context> tags separate retrieved material from the question. Claude models handle tagged structure well.

If the model you select rejects a sampling parameter, drop temperature and rely on the system prompt.

Step 6: Wire It Together

def main():
    import sys
    if len(sys.argv) > 1 and sys.argv[1] == "index":
        build_index()
        return

    model = SentenceTransformer(EMBED_MODEL)
    vectors = np.load("index.npy")
    records = json.loads(Path("chunks.json").read_text(encoding="utf-8"))

    while True:
        q = input("\nAsk (blank to quit): ").strip()
        if not q:
            break
        hits = retrieve(q, model, vectors, records)
        print("\nRetrieved:", [(h["source"], round(h["score"], 2)) for h in hits])
        print("\n" + answer(q, hits))


if __name__ == "__main__":
    main()

Run it:

python rag.py index     # build the index once
python rag.py           # start asking questions

The retrieved-sources printout is deliberate. When an answer looks wrong, check retrieval first. Most RAG failures are retrieval failures, not generation failures.

Step 7: Measure Retrieval Quality

Do not tune by gut feel. Build a small evaluation set of 20 to 50 questions, each paired with the source file that contains the answer. Then compute hit rate@k: the share of questions where the correct source appears in the top k.

def hit_rate(eval_set, model, vectors, records, k=4):
    hits = 0
    for question, expected_source in eval_set:
        results = retrieve(question, model, vectors, records, k)
        if any(r["source"] == expected_source for r in results):
            hits += 1
    return hits / len(eval_set)

eval_set = [
    ("What is the refund window?", "policy.md"),
    ("How do I rotate API keys?", "security.md"),
]

Change one variable at a time (chunk size, overlap, k, embedding model) and rerun. A move from 200-word to 120-word chunks might lift hit rate by several points on fact-heavy docs and hurt it on narrative ones. Only your data can tell you.

Common Failure Modes and Fixes

Symptom Likely cause Fix
Right file, wrong passage Chunks too large Drop to 100–150 words
Answer split across chunks No overlap Raise overlap to 20%
Keyword queries miss (IDs, error codes) Dense vectors miss exact tokens Add BM25 hybrid search
Confident wrong answers Weak grounding prompt Enforce “I don’t know” rule, lower temperature
Good chunks, ignored by model Too many chunks, noise Lower k, add a reranker
Slow queries at 1M+ chunks Brute-force scan Move to FAISS, HNSW, or a vector DB

Real-World Use Cases

Internal support assistant. Index HR policies, IT runbooks, and onboarding docs. Staff ask “How do I request a laptop replacement?” and get an answer with citations to the exact policy file. The “I don’t know” rule routes gaps to a human instead of inventing a procedure.

Contract and compliance lookup. Chunk contracts by clause. Store the clause number in the record metadata. Answers cite clause IDs, so a reviewer verifies each claim in seconds.

Developer documentation bot. Index API docs and changelogs. Add a version field to each record and filter by version before scoring. This stops the bot from mixing v1 and v2 behavior.

Advanced prompt template for stricter grounding:

Answer the question using only the context below.
First, list the context numbers that contain relevant facts.
Then write the answer, citing each claim with [n].
If no chunk is relevant, output: I don't know based on the provided documents.

Forcing the model to name its evidence first reduces unsupported claims.

Scaling Beyond This Build

Brute-force NumPy search stays comfortable up to roughly 50,000 to 100,000 chunks on a laptop. Past that, or when you need metadata filters and live updates, upgrade in this order:

  1. Hybrid search. Combine dense scores with BM25 keyword scores. This fixes exact-term misses.
  2. Reranking. Retrieve top 20, then rerank with a cross-encoder and keep the top 4.
  3. Approximate nearest neighbor. Use FAISS or HNSW indexes for sub-linear search.
  4. Managed vector store. Use pgvector, Qdrant, Weaviate, or Pinecone for persistence, filtering, and concurrent writes.

FAQ

What is a RAG app in simple terms?

A RAG app searches your documents for passages relevant to a question, then gives those passages to an LLM so it can answer from them. The model reads your data at query time instead of relying on memory.

Do I need a vector database to build RAG?

No. For small and mid-sized corpora, a NumPy array of normalized embeddings and a dot product handle retrieval. Add a vector database when you need scale, metadata filtering, or frequent updates.

What is the best chunk size for RAG?

Start with 150 to 300 words and 10–20% overlap. Then test against a labeled question set. Fact-dense documents favor smaller chunks. Narrative documents favor larger ones.

How do I stop a RAG app from hallucinating?

Use a strict system prompt that limits answers to the retrieved context and defines a fixed “I don’t know” reply. Add inline citations, keep temperature low, and check retrieved chunks first when an answer looks wrong.

Hot this week

SHAP vs LIME: Explain Any Model’s Predictions

Compare SHAP vs LIME for explaining ML predictions. See code, speed tradeoffs, and a decision guide. Pick the right explainer for your model today.

Predict House Prices With Linear Regression: A Complete Project

Build a house price predictor with linear regression in Python. Clean data, train, and evaluate with RMSE and R². Start the full project now.

AI and Copyright: The Lawsuits, the Rulings, and What They Mean for Creators

Track every major AI copyright lawsuit in 2026: Anthropic's $1.5B deal, NYT v. OpenAI, Ross, Midjourney. See what it means for your work. Read now.

Build Your First Spam Email Classifier

Build a spam email classifier in Python with TF-IDF and Naive Bayes. Copy the code, hit 98%+ accuracy, and ship your first ML model today.

Best AI Music and Voice Generators in 2026

Compare Suno v6, ElevenLabs Music, Udio, Eleven v4, Cartesia and Fish Audio on price, rights and latency. Pick your tool and start creating today.

Topics

SHAP vs LIME: Explain Any Model’s Predictions

Compare SHAP vs LIME for explaining ML predictions. See code, speed tradeoffs, and a decision guide. Pick the right explainer for your model today.

Predict House Prices With Linear Regression: A Complete Project

Build a house price predictor with linear regression in Python. Clean data, train, and evaluate with RMSE and R². Start the full project now.

AI and Copyright: The Lawsuits, the Rulings, and What They Mean for Creators

Track every major AI copyright lawsuit in 2026: Anthropic's $1.5B deal, NYT v. OpenAI, Ross, Midjourney. See what it means for your work. Read now.

Build Your First Spam Email Classifier

Build a spam email classifier in Python with TF-IDF and Naive Bayes. Copy the code, hit 98%+ accuracy, and ship your first ML model today.

Best AI Music and Voice Generators in 2026

Compare Suno v6, ElevenLabs Music, Udio, Eleven v4, Cartesia and Fish Audio on price, rights and latency. Pick your tool and start creating today.

Train/Test Splits Done Right: Best Practices and Common Mistakes

Avoid data leakage with proper train/test splits. Compare random, stratified, group and time-series methods with Python code. Fix your pipeline today.

AI Engineer vs ML Engineer vs Data Scientist: What’s the Difference?

AI engineer vs ML engineer vs data scientist: compare skills, tools, and roles side by side. Find your best-fit career path and start learning today.

Chain-of-Thought, Few-Shot, and Role Prompts: When Each Technique Helps

Chain-of-thought, few-shot, and role prompts each fix different failures. Learn when each works, when it backfires, and copy the templates.

Related Articles

Popular Categories