Reasoning Models Explained: How AI “Thinks” Before It Answers

A standard LLM predicts the next token and commits to it. A reasoning model spends extra compute generating an internal chain of intermediate steps first, then answers. That single change lifts accuracy on math, code, and multi-step logic by wide margins, and it changes how you prompt, budget, and deploy these systems.

Quick Takeaways

• Reasoning models generate hidden “thinking tokens” before the final answer, trading latency and cost for accuracy.
• They are trained with reinforcement learning (RL) on verifiable tasks, not just next-token imitation of human text.
• You control depth with parameters such as reasoning effort (OpenAI) or budget_tokens (Anthropic).
• Use them for math, code, planning, and audits. Skip them for simple retrieval, chat, and formatting tasks.

What Is a Reasoning Model?

A reasoning model is an LLM trained to produce a long, structured chain of thought (CoT) before committing to a final response. The chain is generated as ordinary tokens. The model uses them as a scratchpad to decompose the problem, try approaches, check results, and backtrack.

Standard models have a fixed compute budget per token. Each token costs roughly the same forward pass whether the question is “What is 2+2?” or “Prove this theorem.” Reasoning models break that constraint. They spend more tokens, and therefore more compute, on harder problems. This is called test-time compute scaling, or inference-time scaling.

The idea has a simple precursor. Researchers found that adding “let’s think step by step” to a prompt improved accuracy on arithmetic and logic tasks. Reasoning models internalize that behavior through training instead of relying on the user to ask.

How Reasoning Models Work Under the Hood

Step 1: Pretraining on a Base Model

Every reasoning model starts as a normal Transformer-based LLM. The base model learns language, code, and world knowledge through next-token prediction over trillions of tokens. This stage sets the ceiling on what the model knows.

Step 2: Reinforcement Learning on Verifiable Rewards

The key step is RL. Developers give the model problems with checkable answers: math with known solutions, code with unit tests, logic puzzles with deterministic outcomes. The model samples many reasoning attempts. Correct final answers earn reward, wrong ones do not.

Over thousands of training steps, behaviors emerge that nobody hand-coded:

• Self-verification: re-checking intermediate results.
• Backtracking: abandoning a failing approach and starting over.
• Decomposition: splitting a problem into sub-goals.
• Longer chains on harder problems: the model learns that more thinking pays off.

DeepSeek-R1 documented this clearly in its open research. Its R1-Zero variant developed longer reasoning and self-correction from RL signals alone, with no supervised reasoning examples.

Step 3: Inference with Thinking Tokens

At runtime, the model emits a reasoning trace, then a final answer. Providers handle the trace differently:

• OpenAI o-series: the raw trace is hidden. You get a summary at most, but you are billed for the reasoning tokens.
• Anthropic extended thinking: thinking blocks are returned in the API response. You set a token budget.
• DeepSeek-R1 and open-weight models: the full trace is visible, which helps with debugging and distillation.

Step 4: Sampling and Search Strategies

Some systems add techniques on top of the single chain:

• Majority voting (self-consistency): sample N chains, pick the most common answer.
• Best-of-N with a verifier: a reward model scores each candidate.
• Tree search: explore branching reasoning paths and prune weak ones.

These raise accuracy further but multiply cost by roughly N.

Reasoning vs. Chain-of-Thought Prompting

CoT prompting still helps on non-reasoning models. On native reasoning models, long hand-written “think step by step” scaffolds add little and can sometimes hurt.

Major Reasoning Model Families

Pricing and exact model names change often. Check each provider’s pricing page before budgeting, and remember that thinking tokens are billed as output tokens on the major APIs.

Implementation: Calling Reasoning Models via API

OpenAI (Responses API)

from openai import OpenAI

client = OpenAI()

response = client.responses.create(
    model="o3",
    reasoning={"effort": "medium"},   # low | medium | high
    input="A train leaves at 2:40 PM going 72 mph. Another leaves the same "
          "station at 3:10 PM going 90 mph. When does the second catch up?"
)

print(response.output_text)
print(response.usage.output_tokens_details.reasoning_tokens)  # hidden thinking cost

Annotations: effort sets how long the model thinks. The reasoning_tokens field shows how much of your bill went to invisible reasoning.

Anthropic (Extended Thinking)

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-5-5",
    max_tokens=16000,                       # must exceed budget_tokens
    thinking={"type": "enabled", "budget_tokens": 8000},
    messages=[{
        "role": "user",
        "content": "Find the bug in this recursive function and explain why it fails."
    }]
)

for block in response.content:
    if block.type == "thinking":
        print("THINKING:", block.thinking)
    elif block.type == "text":
        print("ANSWER:", block.text)

Annotations: budget_tokens caps the thinking length. Set max_tokens higher than the budget. Loop over content blocks to separate the reasoning trace from the answer.

Local Open-Weight Model (Ollama)

ollama run deepseek-r1:14b

Distilled R1 variants come in sizes from roughly 1.5B to 70B parameters. Quantized builds (such as Q4) run on consumer GPUs. Expect the visible trace to wrap inside <think> tags.

Prompt Engineering for Reasoning Models

Reasoning models reward a different style than classic LLMs.

1. State the goal and constraints, not the method. Let the model choose its approach.
2. Skip manual CoT scaffolds. “Think step by step” is redundant.
3. Provide complete context up front. Mid-reasoning clarification is not possible.
4. Define the output format and success criteria. Say what a correct answer looks like.
5. Keep few-shot examples minimal. Heavy examples can anchor the model into worse paths.
6. Tune effort to difficulty. Start at low or medium. Escalate only when accuracy demands it.

Template: Structured Problem Prompt

Goal: Determine whether the attached SQL migration can cause data loss.
Context: PostgreSQL 15, table `orders` has 40M rows, no downtime allowed.
Constraints: Must be reversible. Must not lock the table for more than 2 seconds.
Output: 1) Verdict (safe/unsafe), 2) Risky statements quoted, 3) Safer rewrite.

When to Use (and Avoid) Reasoning Models

Real-World Workflows

Workflow 1: Enterprise Code Review

A team routes pull requests through a two-stage pipeline. A fast standard model triages style issues. A reasoning model at high effort reviews only changes touching auth, payments, or concurrency. This keeps reasoning spend limited to the roughly 10% of diffs where deep analysis matters.

Workflow 2: Reducing Hallucinations in RAG

Retrieval-augmented generation can still produce unsupported claims. A common pattern:

1. Retrieve top-k passages (for example, k=8, 512-token chunks).
2. Ask a reasoning model to answer and list the passage supporting each claim.
3. Reject or regenerate any claim without a supporting citation.

The reasoning step makes the model check its draft against evidence before answering. This reduces, but does not eliminate, unsupported statements.

Workflow 3: Data Analysis Audit

Feed a reasoning model a query plus its result table and ask it to find logical errors in joins, filters, or aggregations. It reasons through row counts and edge cases that a quick pass would miss.

Limits and Trade-Offs

• Cost: Thinking tokens are billed. A short question can consume thousands of reasoning tokens.
• Latency: Responses can take many seconds to minutes at high effort.
• Overthinking: Models sometimes spend long chains on trivial questions.
• Faithfulness gap: The visible trace does not always reflect the true factors behind an answer. Treat it as a useful signal, not a guaranteed explanation.
• Not a fact database: Better reasoning does not add missing knowledge. Pair it with retrieval or tools for current facts.

Cost Control Checklist

• Cap max_tokens and thinking budgets.
• Route by difficulty: standard model first, escalate on failure.
• Log reasoning-token usage per request.
• Cache stable prompt prefixes where the provider supports it.
• Benchmark on your own data. Public leaderboards rarely match your workload.

Frequently Asked Questions

Do reasoning models actually “think” like humans?

No. They generate intermediate tokens that function as a scratchpad, trained through reinforcement learning to reach correct answers. The behavior resembles deliberate reasoning, but the mechanism is still next-token prediction over a longer, learned trace.

Are reasoning models better than standard LLMs at everything?

No. They outperform on math, coding, and multi-step logic. On summarization, translation, and casual conversation, standard models are usually faster, cheaper, and equally accurate.

Why are reasoning models more expensive?

You pay for thinking tokens in addition to the visible answer. Providers bill these as output tokens, and a single hard query can generate thousands of them before the first word of the answer appears.

Can I run a reasoning model locally?

Yes. Open-weight models such as DeepSeek-R1 distills and Qwen reasoning variants run locally through tools like Ollama or llama.cpp. Smaller quantized versions (for example, 7B to 14B at Q4) fit on consumer GPUs, with lower accuracy than the full-size models.