Context Engineering Explained: Going Beyond Prompt Engineering

A great prompt cannot rescue a model that sees the wrong information. Most production LLM failures trace back to what sits inside the context window, not to how the instruction was phrased. Context engineering is the practice of designing, selecting, compressing, and ordering everything a model reads at inference time: instructions, retrieved documents, tool outputs, conversation history, and memory. This guide breaks down the architecture, shows working code, and gives you a framework you can apply to agents and RAG systems today.

Quick Takeaways

• Context engineering manages the entire input payload to an LLM. Prompt engineering is one component of it.
• Model quality degrades as irrelevant tokens accumulate, a pattern often called context rot. A bigger window does not fix it.
• Four core operations drive every system: write, select, compress, isolate.
• Agents benefit most. Long tool-calling loops generate tokens faster than any window can hold.

What Is Context Engineering?

An LLM is a function. It takes a sequence of tokens and predicts the next one. It has no memory beyond that sequence. Everything the model “knows” about your task at inference time lives in the context window.

Context engineering treats that window as a scarce resource. You decide what enters it, in what order, in what format, and when it gets removed.

The context payload typically contains six layers:

1. System instructions: role, rules, output format.
2. User input: the current request.
3. Short-term memory: recent conversation turns.
4. Long-term memory: facts persisted across sessions.
5. Retrieved knowledge: documents from a vector store or search index.
6. Tool definitions and outputs: function schemas and their results.

Prompt engineering touches layer one and part of layer two. Context engineering owns all six.

Why Prompt Engineering Alone Fails

Three problems appear once you move from demos to production.

1. Context Rot

Models do not use long inputs uniformly. Research on long-context behavior (for example, the “Lost in the Middle” study) shows retrieval accuracy drops when key facts sit in the middle of a long input rather than at the start or end. Adding tokens can reduce accuracy.

2. Context Poisoning and Distraction

A single hallucinated fact written into memory gets reread on every later turn. The model treats its own earlier error as ground truth. Irrelevant documents also pull the model’s attention away from the relevant ones.

3. Context Clash

Two retrieved documents may contradict each other. Without a resolution rule, the model picks one arbitrarily.

A better prompt cannot fix any of these. Better context assembly can.

The Four Pillars: Write, Select, Compress, Isolate

Write: Persist Context Outside the Window

Store information externally so the window stays lean. Common patterns:

• Scratchpads: the agent writes intermediate notes to a file or state object.
• Memory stores: durable facts saved per user, retrieved when relevant.

Select: Pull In Only What Matters

Selection is where RAG lives. Retrieve by semantic similarity, then rerank.

Typical specifications for a production retrieval stack:

Retrieve wide, rerank narrow, and pass only the best chunks to the model.

Compress: Shrink Without Losing Signal

Two techniques dominate:

• Summarization: replace old conversation turns with a short summary once history exceeds a threshold (for example, 70% of the window).
• Trimming: drop stale tool outputs, duplicate chunks, or low-score passages.

Isolate: Split Context Across Components

Give each sub-task its own clean window. In multi-agent systems, a research sub-agent explores with a large context, then returns a 1,000-2,000 token distilled result to the lead agent. The lead never sees the noise.

Implementation: A Minimal Context Builder in Python

The following example assembles context with a token budget, ordered layers, and automatic history compression. It uses the Anthropic Python SDK. Replace the model string with the current model available to your account.

pip install anthropic tiktoken

import anthropic
import tiktoken

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from env
enc = tiktoken.get_encoding("cl100k_base")  # approximate token counter

MAX_CONTEXT_TOKENS = 8000      # your self-imposed budget
SUMMARY_TRIGGER = 0.7          # compress history at 70% of budget

def count(text: str) -> int:
    return len(enc.encode(text))

def summarize(history: list[dict]) -> str:
    """Compress old turns into a short summary."""
    transcript = "\n".join(f"{m['role']}: {m['content']}" for m in history)
    resp = client.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=400,
        temperature=0.2,
        messages=[{
            "role": "user",
            "content": f"Summarize key facts and decisions in under 200 words:\n{transcript}"
        }],
    )
    return resp.content[0].text

def build_context(system: str, history: list[dict],
                  retrieved: list[str], query: str):
    # 1. Compress history if over the trigger threshold
    hist_tokens = sum(count(m["content"]) for m in history)
    if hist_tokens > MAX_CONTEXT_TOKENS * SUMMARY_TRIGGER:
        summary = summarize(history[:-4])          # keep last 4 turns verbatim
        history = [{"role": "user",
                    "content": f"[Earlier conversation summary]\n{summary}"},
                   {"role": "assistant", "content": "Understood."}] + history[-4:]

    # 2. Place retrieved docs near the query (end of context = higher recall)
    docs = "\n\n".join(
        f"<doc id='{i}'>\n{d}\n</doc>" for i, d in enumerate(retrieved)
    )
    final_user = f"<documents>\n{docs}\n</documents>\n\nQuestion: {query}"

    return system, history + [{"role": "user", "content": final_user}]

system_prompt = "Answer only from the provided documents. Cite doc ids. Say 'not found' if absent."
system, messages = build_context(system_prompt, [], ["Chunk A...", "Chunk B..."], "What is X?")

response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=1000,
    temperature=0.0,   # deterministic for factual QA
    system=system,
    messages=messages,
)
print(response.content[0].text)

Annotations:

• temperature=0.0 suits factual, grounded answers. Raise it to 0.7-1.0 for creative tasks.
• XML-style tags (<doc>) give the model clear boundaries between sources.
• Documents go before the question. Long-context guidance consistently shows queries placed at the end improve response quality.
• The last four turns stay verbatim, so recent nuance is not lost.

Context Engineering for Agents

Agents loop: reason, call a tool, read the result, repeat. Each loop appends tokens. A 30-step task can easily pass 100K tokens of mostly stale tool output.

Practical controls:

• Tool result clearing: once an agent has acted on a result, replace the raw output with a one-line note.
• Tool selection: exposing 50+ tools degrades selection accuracy. Retrieve the 5-10 most relevant tool schemas per step.
• Just-in-time retrieval: store file paths or IDs in context, not file contents. Load the content only when needed.
• Structured note-taking: have the agent write progress to a NOTES.md file and reread it after a context reset.

Techniques Compared

Prompt Caching: The Cost Lever

Static content (system prompt, tool schemas, reference docs) should sit at the start of the context so providers can cache the prefix. Cached input tokens are billed at a fraction of the normal rate, and latency drops on repeated calls.

Rules of thumb:

1. Put stable content first, volatile content last.
2. Never insert timestamps or random IDs into the cached prefix.
3. Check your provider’s current pricing and minimum cacheable length before designing around it.

Real-World Workflows

Customer Support Agent

Problem: The agent hallucinates refund policies.

Context design:

1. System layer: tone, escalation rules.
2. Retrieved layer: top 4 policy chunks, reranked, filtered by customer region.
3. Memory layer: the customer’s last three tickets, summarized.
4. Instruction: “If policy text does not cover the case, escalate.”

Result: Answers stay grounded in retrievable text, and the escalation rule gives the model a safe exit.

Coding Assistant

Problem: The agent loses track of the project after 40+ tool calls.

Context design:

• Keep a PLAN.md the agent updates after each milestone.
• Load files just-in-time instead of pasting the repo.
• Clear tool outputs older than 10 steps.
• Compact the conversation near the window limit, preserving decisions, open bugs, and file paths.

Enterprise RAG for Compliance

Problem: Contradictory document versions produce conflicting answers.

Context design:

• Attach metadata (version, effective_date) to each chunk.
• Filter to the latest version before retrieval.
• Add a rule: “If sources conflict, prefer the most recent effective date and flag the conflict.”

A Context Audit Checklist

Run this before blaming the model.

1. Is the needed fact actually in the window? Log the full payload.
2. Is it buried? Move critical facts to the start or end.
3. Is there contradictory content? Deduplicate and add a conflict rule.
4. Is there dead weight? Remove stale turns and unused tool schemas.
5. Is the format clear? Use delimiters and consistent structure.
6. Is the budget enforced? Set hard token caps per layer.

Measuring Context Quality

You cannot improve what you do not measure. Track:

Frameworks such as RAGAS and LangSmith evaluate several of these automatically. Build a fixed test set of 50-200 queries and rerun it after every context change.

FAQ

What is the difference between prompt engineering and context engineering?

Prompt engineering focuses on writing effective instructions. Context engineering manages everything the model sees: instructions, retrieved data, memory, tool outputs, and history. Prompt engineering is a subset of context engineering.

Does a larger context window make context engineering unnecessary?

No. Larger windows raise capacity but not attention quality. Models still lose accuracy on irrelevant or poorly placed information, and longer inputs raise cost and latency. Curation matters at every window size.

Is RAG the same as context engineering?

No. RAG is one technique within context engineering, covering the “select” step. Context engineering also includes memory, compression, tool management, and isolation.

How do I start with context engineering?

Log the full payload sent to your model for 20 real queries. Identify irrelevant, duplicated, or missing information, then add token budgets, reranking, and history summarization one at a time. Measure after each change.