How AI Agents Work: Planning, Tools, Memory, and Loops

An AI agent is a language model wrapped in a control loop. The model reasons, picks an action, observes the result, and repeats until the goal is met or a budget runs out. A chatbot answers once. An agent works through a task over many steps, and every step feeds the next. This guide breaks the architecture into four components, shows a working implementation, and covers the failure modes that break agents in production.

Quick Takeaways

• An agent = LLM + planner + tools + memory + loop. Remove any one and you have a chatbot or a script.
• The loop is the product. Quality depends on stopping conditions, error handling, and context management more than on model choice.
• Context is the scarce resource. Every tool result competes for the context window, so memory design decides how long an agent stays coherent.
• Start with a single agent and a few tools. Add multi-agent orchestration only when one agent measurably fails.

What Makes Something an Agent

Autonomy is a spectrum. At one end sits a workflow: a fixed chain of LLM calls where your code decides the path. At the other sits an agent: the model decides the path at runtime.

Use a workflow when you can enumerate the steps. Use an agent when you cannot.

The Agent Loop

Every agent reduces to one pattern:

1. Perceive: assemble context (goal, history, tool results).
2. Reason: the model decides what to do next.
3. Act: the model emits a tool call or a final answer.
4. Observe: your code runs the tool and appends the result.
5. Repeat until a stop condition fires.

The loop has three stop conditions you must define explicitly:

• Success: the model returns a final answer with no tool call.
• Budget: a hard cap on steps (commonly 10–25), tokens, or wall-clock time.
• Failure: repeated identical tool calls, unrecoverable errors, or a human-escalation trigger.

Skip the budget and one confused agent can burn thousands of dollars overnight.

Planning: How Agents Decide What to Do

ReAct (Reason + Act)

ReAct interleaves a short reasoning trace with each action. The model writes a thought, calls a tool, reads the observation, and writes the next thought. It is the default pattern in most frameworks because it adapts after every result.

Strength: flexible, easy to debug by reading traces.
Weakness: myopic. It optimizes the next step, not the whole route.

Plan-and-Execute

A planner call drafts a multi-step plan up front. An executor then works through it, with optional re-planning when a step fails.

Strength: better coherence on long tasks, and the plan can run on a stronger model while execution runs on a cheaper one.
Weakness: stale plans. If step 2 invalidates step 5, a rigid executor keeps marching.

Reflection and Self-Critique

After acting, the agent critiques its own output and retries. This raises quality on code and writing tasks but multiplies token cost, often 2–3x. Cap reflection rounds at 2–3.

Tree Search and Branching

Methods like Tree of Thoughts explore several candidate plans and score them. They help on puzzles and constrained reasoning but are rarely worth the cost for everyday tool-using agents.

Tools: How Agents Act

A tool is a function the model can request by name. The model never executes anything. It emits structured arguments (JSON), your code runs the function, and you return the result.

A good tool definition has three parts: a clear name, a precise description (the model reads this to decide when to call it), and a typed input schema.

Tool Design Rules

• Write descriptions for the model, not for humans. State what the tool does, when to use it, and what it returns.
• Keep the toolset small. Accuracy of tool selection drops as tools multiply. Aim for under 15–20 tools per agent.
• Return concise results. Truncate or paginate. A 50,000-token tool result wrecks the context window.
• Make errors informative. Return what went wrong and how to fix it, so the model can self-correct.
• Mark destructive actions. Gate deletes, payments, and sends behind human approval.

Model Context Protocol (MCP)

MCP is an open standard for exposing tools and data sources to models through a common interface. Instead of writing custom integrations per model, you run an MCP server once and any compatible client can use it. It is the fastest way to share tools across agents and applications.

Memory: How Agents Stay Coherent

Memory splits into layers with different cost and persistence.

Managing the Context Window

Long tasks overflow the window. Three techniques handle it:

1. Summarization: compress old turns into a running summary once history passes a threshold (for example 70% of the window).
2. Context editing: drop stale tool results while keeping the decisions they informed.
3. External notes: have the agent write progress to a file or scratchpad and re-read it, which survives context resets.

Retrieval Memory

For semantic recall, embed text with a model producing vectors (commonly 768–3072 dimensions), store them in a vector database, and retrieve the top-k (typically 3–8) by cosine similarity. Combine with keyword search (hybrid retrieval) to catch exact terms that embeddings blur.

Build a Minimal Agent in Python

This example uses the Anthropic Messages API with tool use. It implements the full loop with a step cap.

pip install anthropic
export ANTHROPIC_API_KEY="your-key"

import anthropic

client = anthropic.Anthropic()
MODEL = "claude-sonnet-4-5"  # swap in the current model string from the docs
MAX_STEPS = 10

# 1. Define tools: name, description, JSON schema
tools = [{
    "name": "calculator",
    "description": "Evaluate a basic arithmetic expression. Use for any math.",
    "input_schema": {
        "type": "object",
        "properties": {"expression": {"type": "string"}},
        "required": ["expression"],
    },
}]

# 2. Implement the tool (your code executes it, not the model)
def run_tool(name, args):
    if name == "calculator":
        try:
            # Restrict characters; never eval untrusted input in production
            allowed = set("0123456789+-*/(). ")
            if not set(args["expression"]) <= allowed:
                return "Error: invalid characters."
            return str(eval(args["expression"]))
        except Exception as e:
            return f"Error: {e}"
    return "Error: unknown tool."

# 3. The agent loop
def run_agent(goal):
    messages = [{"role": "user", "content": goal}]

    for step in range(MAX_STEPS):               # budget stop condition
        response = client.messages.create(
            model=MODEL,
            max_tokens=1024,
            tools=tools,
            messages=messages,
        )
        messages.append({"role": "assistant", "content": response.content})

        # Success stop condition: no tool requested
        if response.stop_reason != "tool_use":
            return "".join(b.text for b in response.content if b.type == "text")

        # Act + observe: run every requested tool, return results
        results = []
        for block in response.content:
            if block.type == "tool_use":
                output = run_tool(block.name, block.input)
                results.append({
                    "type": "tool_result",
                    "tool_use_id": block.id,
                    "content": output,
                })
        messages.append({"role": "user", "content": results})

    return "Stopped: step budget exhausted."

print(run_agent("What is (1847 * 23) + 4096?"))

What to notice: the messages list is the working memory. The for loop is the agent loop. stop_reason is the success signal. MAX_STEPS is the safety net. Every production framework is an elaboration of these 30 lines.

Framework Comparison

Prototype with the raw loop first. Adopt a framework when you need persistence, branching, or human-in-the-loop checkpoints.

Single-Agent vs. Multi-Agent Systems

Multi-agent designs pay off in two cases: parallel research across independent sources, and context isolation, where a subagent does messy exploration and returns only a short summary. For everything else, a single well-tooled agent wins.

Real-World Use Cases and Workflows

Customer Support Triage

An agent reads a ticket, calls tools to look up the order, checks policy documents via retrieval, drafts a reply, and escalates to a human when confidence is low or a refund exceeds a threshold. Key design choice: human approval gates on money-moving actions.

Coding Agents

The agent reads a codebase, edits files, runs tests, and iterates on failures. The test suite acts as a ground-truth feedback signal, which is why coding is among the most reliable agent domains.

Research and Analysis

An orchestrator splits a question into subtopics, dispatches subagents to search in parallel, then synthesizes their summaries. Citations from tool results keep claims verifiable.

Prompt Template: A Reliable Agent System Prompt

You are a [role] agent. Goal: [specific outcome].

Rules:
- Use tools to verify facts. Do not guess.
- Think step by step before each tool call.
- If a tool fails, read the error and try a different approach.
- Stop and ask the user before any irreversible action.
- When done, give a final answer with no tool call.

Constraints: maximum [N] steps. Final answer under [M] words.

Why Agents Fail (and How to Fix It)

Compounding error math matters. If each step succeeds 95% of the time, a 10-step task succeeds only about 60% of the time (0.95¹⁰). Shorter paths and verification steps raise reliability faster than a bigger model.

Evaluation

Measure task success rate, steps per task, cost per task, and latency. Build a fixed test set of 20–50 realistic tasks and rerun it on every prompt or tool change. Without evals, you are guessing.

FAQ

What is the difference between an AI agent and a chatbot?

A chatbot generates one response per prompt. An AI agent runs a loop: it plans, calls tools, observes results, and continues until a goal is complete. Agents take actions in external systems; chatbots only produce text.

What are the main components of an AI agent?

Four: a planner (the LLM reasoning about next steps), tools (functions it can call), memory (context window plus external storage), and a loop that orchestrates steps and enforces stop conditions.

How does an AI agent remember things?

Short-term memory is the context window, holding the conversation and tool results. Long-term memory uses external storage such as vector databases, files, or logs, retrieved when relevant. Summarization keeps long tasks within context limits.

Do I need a framework to build an AI agent?

No. A working agent is a loop around an LLM API with tool calling, as shown above in about 30 lines. Frameworks add persistence, branching, and observability once your needs grow.