RAG vs Fine-Tuning vs Long Context: Which Should You Use?

Most teams pick an LLM customization strategy by copying whatever their last project used. That is how you end up fine-tuning a model to memorize a pricing page that changes every week, or stuffing 800K tokens into a prompt to answer a question that needs three paragraphs. Each method solves a different problem: RAG fixes missing knowledge, fine-tuning fixes missing behavior, and long context fixes missing working memory.

Quick Takeaways

• Use RAG when knowledge is large, private, or changes often, and you need citations.
• Use fine-tuning when you need consistent style, format, or domain reasoning, not new facts.
• Use long context when the corpus fits in the window and you need cross-document reasoning with minimal infrastructure.
• Combine them. Production systems usually pair a lightly fine-tuned model with retrieval, or use prompt caching with long context.

What Each Approach Actually Does

RAG (Retrieval-Augmented Generation)

RAG leaves model weights untouched. At query time, you embed the question, retrieve the top-k relevant chunks from a vector store, and insert them into the prompt. The model answers from that evidence.

Typical production settings:

• Chunk size: 300-800 tokens with 10-15% overlap
• Embedding dimensions: 384 to 3072, depending on the model
• Top-k: 4-8 chunks, often re-ranked from a candidate set of 20-50
• Retrieval: hybrid search (dense vectors plus BM25) outperforms either alone on most enterprise corpora
• Temperature: 0.0-0.3 for grounded answers

RAG’s core advantage is separation of concerns. Your knowledge lives in a database you can update, audit, and permission-filter per user. Its core weakness is retrieval quality: if the right chunk never reaches the prompt, the model cannot use it.

Fine-Tuning

Fine-tuning changes the model’s weights using your examples. Modern practice favors parameter-efficient fine-tuning (PEFT), especially LoRA and QLoRA, which train small adapter matrices instead of the full network. A typical LoRA setup uses rank r=8-64, alpha of roughly 2x rank, and a learning rate near 2e-4. QLoRA loads the base model in 4-bit (NF4) quantization, which makes 7B-13B models trainable on a single 24 GB GPU.

Fine-tuning excels at:

• Locking in an output schema (strict JSON, SQL dialects, clinical note formats)
• Matching brand voice
• Distilling a large model’s behavior into a smaller, cheaper one
• Teaching domain-specific reasoning patterns or tool-calling habits

It performs poorly at injecting facts. Models trained on a knowledge base tend to recall it unreliably and cannot cite sources. Facts also go stale the moment the data changes.

Long Context

Long-context models accept hundreds of thousands of tokens, and some accept over a million. You skip the retrieval layer and place the documents directly in the prompt. This is the fastest path to a working prototype, and it handles questions that span many sections, such as “which clauses across these 40 contracts conflict?”

The trade-offs are real:

• Cost scales with input tokens on every call unless you use prompt caching
• Latency grows with prompt length, particularly time-to-first-token
• Recall degrades in the middle of very long prompts on many models (“lost in the middle”), and multi-fact retrieval across huge contexts is harder than single-needle tests suggest
• No per-user access control unless you assemble prompts per user

Head-to-Head: Where Each Method Wins

The Decision Framework

Ask these five questions in order.

1. Does the corpus fit in the context window with room to spare? If it is under roughly 100K tokens and rarely changes, start with long context. Add caching if you query it repeatedly.
2. Does the knowledge change weekly or faster, or need per-user permissions? Choose RAG.
3. Do you need citations or audit trails? Choose RAG, or long context with quoted-passage extraction.
4. Is the failure mode wrong format, tone, or reasoning style rather than wrong facts? Choose fine-tuning.
5. Is volume high enough that prompt tokens dominate your bill? Consider fine-tuning a smaller model to replace a long instruction prompt, or RAG to shrink the context.

If you answer “yes” to questions from two different categories, you need a hybrid.

Cost and Latency: A Back-of-Envelope Model

Pricing changes often, so plug in current numbers from your provider. The structure of the math stays stable.

Consider 100K tokens of context at 10,000 queries per day. Without caching, you pay for 1 billion input tokens daily. With prompt caching, repeated prefix tokens are billed at a steep discount, so the same workload often costs a fraction of that. RAG sending 4K tokens per query would use 40 million. Long context wins on engineering time; RAG wins on unit economics at scale.

Implementation Examples

RAG with Chroma and the Anthropic API

# pip install chromadb anthropic
import chromadb
from anthropic import Anthropic

client = Anthropic()  # reads ANTHROPIC_API_KEY
db = chromadb.Client()
col = db.create_collection("docs")  # default embedder: all-MiniLM-L6-v2 (384 dims)

# Index: pre-chunk your documents to ~500 tokens
chunks = ["Refunds are processed within 5 business days...",
          "Enterprise plans include SSO and audit logs..."]
col.add(documents=chunks, ids=[f"c{i}" for i in range(len(chunks))])

def answer(question: str) -> str:
    hits = col.query(query_texts=[question], n_results=4)  # top-k = 4
    context = "\n\n".join(hits["documents"][0])
    msg = client.messages.create(
        model="claude-sonnet-5-5",
        max_tokens=600,
        temperature=0.2,  # low for grounded answers
        system="Answer only from the context. If missing, say you don't know.",
        messages=[{"role": "user",
                   "content": f"<context>\n{context}\n</context>\n\nQuestion: {question}"}],
    )
    return msg.content[0].text

print(answer("How long do refunds take?"))

Long Context with Prompt Caching

from anthropic import Anthropic
client = Anthropic()

big_doc = open("annual_report.txt").read()  # e.g., ~120K tokens

resp = client.messages.create(
    model="claude-sonnet-5-5",
    max_tokens=800,
    system=[
        {"type": "text", "text": "You are a financial analyst. Quote sources."},
        {"type": "text", "text": big_doc,
         "cache_control": {"type": "ephemeral"}},  # cache the static prefix
    ],
    messages=[{"role": "user", "content": "Summarize segment margin changes."}],
)
print(resp.content[0].text)

Put static content first and the variable question last. Cache hits require an identical prefix.

QLoRA Fine-Tuning Skeleton

# pip install transformers peft bitsandbytes trl
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model
import torch

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_compute_dtype=torch.bfloat16)
model = AutoModelForCausalLM.from_pretrained("your-base-model",
                                             quantization_config=bnb)

lora = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
                  target_modules=["q_proj", "v_proj"], task_type="CAUSAL_LM")
model = get_peft_model(model, lora)
model.print_trainable_parameters()  # typically <1% of weights
# Train with trl.SFTTrainer on 500-5,000 high-quality examples, lr ~2e-4

Quality beats quantity. A few hundred clean, consistent examples often outperform tens of thousands of noisy ones.

Real-World Workflows

Enterprise Support Assistant (RAG + Guardrails)

A SaaS company indexes its docs, release notes, and resolved tickets. The pipeline uses hybrid retrieval, a cross-encoder re-ranker, and a system prompt that forces “answer from context or escalate.” Each response returns source links. Hallucinations drop because the model can refuse when retrieval comes back empty. Access control happens at the retrieval layer by filtering on tenant ID.

Structured Extraction at Scale (Fine-Tuning)

A logistics firm extracts 40 fields from invoices into strict JSON. A prompted frontier model works but needs a 3,000-token instruction block and still drifts on edge cases. They fine-tune a small model on 2,000 labeled invoices. The prompt shrinks to under 200 tokens, schema adherence improves, and per-call cost falls sharply.

Contract Review (Long Context)

A legal team loads a 250-page master agreement plus amendments into a cached prompt and asks cross-reference questions. No chunking means no lost cross-clause dependencies. They still verify answers against quoted passages.

The Hybrid Stack

Many mature systems layer all three:

1. Fine-tune a mid-size model for tone, tool-calling, and output schema.
2. RAG supplies current, permissioned facts.
3. Long context holds the retrieved set (20-50 chunks) plus conversation history, with caching on the static system prompt.

Prompt Template for Grounded RAG Answers

You are a support specialist. Use ONLY the documents in <context>.
Rules:
1. Cite the source ID in brackets after each claim, e.g. [doc-12].
2. If the context lacks the answer, reply: "I don't have that information."
3. Never use outside knowledge for product details.

<context>
{retrieved_chunks_with_ids}
</context>

Question: {user_question}

Common Mistakes

• Fine-tuning for facts. You get confident, outdated, uncitable answers.
• Skipping evaluation. Build a test set of 50-200 real questions before choosing. Measure answer accuracy, retrieval recall@k, latency, and cost.
• Oversized chunks. Chunks above ~1,000 tokens dilute embeddings and waste context.
• Ignoring the baseline. A strong prompt with a few examples sometimes beats all three approaches. Test it first.
• Treating the window size as a guarantee. Advertised context length is a maximum, not a promise of uniform recall.

FAQ

Is RAG better than fine-tuning?

Neither is better overall. RAG is better for injecting current or private knowledge with citations. Fine-tuning is better for changing how a model behaves, such as its format, tone, or reasoning style. Use RAG for facts and fine-tuning for behavior.

Will long context replace RAG?

Not for large or changing corpora. Context windows keep growing, but per-query cost, latency, access control, and recall limits still favor RAG once data exceeds what you can economically send on every request. Long context replaces RAG mainly for small, stable document sets.

Can I combine RAG and fine-tuning?

Yes, and it is a common production pattern. Fine-tune the model to follow your output format and use retrieved documents correctly, then use RAG to supply up-to-date facts at query time.

How much data do I need to fine-tune an LLM?

For LoRA or QLoRA on style or format tasks, 500-5,000 high-quality examples is a typical range. Complex reasoning tasks may need more. Data quality and consistency matter more than volume.