Most prompt failures come from using the wrong technique, not from a weak model. Chain-of-thought fixes reasoning errors. Few-shot examples fix format and pattern errors. Role prompts fix tone and framing. Each one can also backfire, especially on reasoning models.
Quick Takeaways
- Chain-of-thought (CoT) helps standard models on multi-step math, logic, and code. It adds little on reasoning models that already think internally.
- Few-shot prompting is the most reliable way to lock in output format, labels, and style. Use 3-5 examples on standard models and test whether zero-shot does as well on reasoning models.
- Role prompts shape voice, vocabulary, and audience. They are a style control, not an accuracy booster.
- Diagnose first. Wrong answer: try CoT. Wrong format: add examples. Wrong tone: set a role.
| Technique | Fixes | Token overhead | Best for | Risk |
|---|---|---|---|---|
| Zero-shot CoT | Reasoning slips | +100-1,000 output tokens | Math, logic, planning | Wasted tokens on reasoning models |
| Few-shot | Format, labels, style drift | +200-1,500 input tokens | Classification, extraction, JSON | Bias toward example content |
| Few-shot CoT | Hard reasoning with a fixed pattern | Highest | Weaker or smaller models | Can degrade reasoning models |
| Role prompt | Tone, depth, audience fit | +10-50 input tokens | Customer-facing text, domain voice | Confident but wrong answers |
What Each Technique Actually Does
Each technique changes a different part of what the model sees. Pick by the failure you observe.
- Chain-of-thought changes how much computation the model spends before it answers. It forces intermediate tokens that act as working memory.
- Few-shot changes the pattern the model completes. The examples define the input-output mapping.
- Role prompts change the distribution of likely wording. They bias vocabulary, register, and assumed audience.
None of these adds knowledge. A prompt cannot make a model know something it does not.
Chain-of-Thought: Spend Tokens on Reasoning
CoT asks the model to write out intermediate steps before the final answer. Wei et al. (2022) introduced the few-shot version with worked examples. Kojima et al. (2022) showed that appending a phrase like “Let’s think step by step” works zero-shot.
When CoT helps
Use it on standard (non-reasoning) models when the task needs more than one logical hop:
- Multi-step arithmetic and word problems
- Constraint-based planning (scheduling, routing)
- Code debugging that needs tracing
- Policy or contract analysis with several conditions
When CoT backfires
Reasoning models such as OpenAI’s o-series, Claude with extended thinking, and DeepSeek R1 run a chain of thought internally. OpenAI’s guidance says explicit “think step by step” prompting is unnecessary for these models and that asking them to reason more may hurt performance.
Research on in-context learning adds a second warning. One study found that CoT can underperform direct answering on pattern-based tasks. Long-CoT models used roughly 40× more inference tokens for comparable or worse results.
CoT template for standard models
Solve the problem below.
<problem>
{problem}
</problem>
First, write your reasoning inside <thinking> tags. Check each step.
Then give only the final answer inside <answer> tags.
Separating reasoning from the answer lets you parse the output and drop the scratchpad before showing it to users.
Native extended thinking (Anthropic API)
For reasoning-capable models, turn on built-in thinking instead of prompting for it:
import anthropic
client = anthropic.Anthropic()
response = client.messages.create(
model="claude-sonnet-5-5",
max_tokens=8000, # must exceed budget_tokens
thinking={"type": "enabled", "budget_tokens": 4000},
messages=[{
"role": "user",
"content": "A train leaves at 9:40 and travels 212 km at 83 km/h. "
"When does it arrive?"
}],
)
for block in response.content:
if block.type == "text":
print(block.text)
Set budget_tokens based on task difficulty. Start near 2,000-4,000 and raise it only if accuracy improves. Leave temperature at its default when thinking is enabled.
Few-Shot Prompting: Show the Pattern
Few-shot prompting places input-output examples in the prompt. Anthropic’s documentation calls examples one of the most reliable ways to steer output format, tone, and structure.
When few-shot helps
- Classification with custom labels (ticket routing, sentiment with your own scale)
- Extraction into strict JSON schemas
- Style transfer where rules are hard to describe but easy to show
- Edge-case handling where one example beats a paragraph of rules
How to build good examples
- Use 3-5 examples to start. Add more only after testing.
- Cover edge cases, not just easy cases.
- Vary the examples so the model does not copy surface features.
- Balance labels. If four of five examples say “positive,” the model leans positive.
- Wrap each example in XML tags so it is not confused with the real input.
Few-shot template
import anthropic
client = anthropic.Anthropic()
prompt = """Classify each support ticket as billing, bug, or feature_request.
Respond with the label only.
<examples>
<example>
<ticket>I was charged twice for March.</ticket>
<label>billing</label>
</example>
<example>
<ticket>The export button throws a 500 error.</ticket>
<label>bug</label>
</example>
<example>
<ticket>Could you add dark mode to the dashboard?</ticket>
<label>feature_request</label>
</example>
<example>
<ticket>My invoice shows the wrong company name and the PDF won't open.</ticket>
<label>billing</label>
</example>
</examples>
<ticket>{ticket}</ticket>"""
response = client.messages.create(
model="claude-sonnet-5-5",
max_tokens=10,
temperature=0, # deterministic labels
messages=[{"role": "user",
"content": prompt.format(ticket="App crashes when I upload a PNG.")}],
)
print(response.content[0].text) # bug
The fourth example is deliberate. It teaches the model how to resolve a ticket that mixes two categories.
When few-shot backfires
Several studies report that few-shot CoT can hurt reasoning-model performance even with high-quality demonstrations. Another found that for strong recent models, traditional CoT exemplars mainly align output format rather than improve reasoning.
The practical rule: for reasoning models, start zero-shot. Add examples only for format, and only if an ablation shows they help.
Role Prompts: Control Voice, Not Accuracy
A role prompt assigns an identity (“You are a senior tax accountant”). Anthropic’s docs describe it as a way to improve focus and tone through the system parameter. That holds for style and framing.
The accuracy claim is weaker. A large study tested 162 personas on 2,410 factual questions across four model families. It found that adding personas does not improve performance versus a no-persona control. A later Wharton study on expert personas found they did not reliably improve accuracy, and some low-knowledge personas made results worse.
A confident expert voice can also make a wrong answer sound more trustworthy. That is a risk in legal, medical, and financial products.
Where role prompts earn their place
- Customer-facing tone (warm, concise, formal)
- Audience framing (“Explain to a finance director with no ML background”)
- Output depth (a quick triage note vs a full memo)
- Consistent persona across a multi-turn product
Role prompt template
system = (
"You are a support specialist for a B2B analytics product. "
"Write for IT admins. Be concise, state the fix first, "
"and never guess about billing; escalate instead."
)
response = client.messages.create(
model="claude-sonnet-5-5",
max_tokens=500,
system=system, # role and constraints live here
messages=[{"role": "user", "content": "Our SSO login loops. Help."}],
)
Put the role and standing constraints in the system parameter. Put task-specific content in the user turn.
Choosing a Technique: Decision Table
| Symptom | First fix | Second fix | Avoid |
|---|---|---|---|
| Wrong math or logic (standard model) | CoT with <thinking> tags |
Few-shot CoT | Role prompt alone |
| Wrong math or logic (reasoning model) | Clear, direct problem statement | Raise thinking budget | “Think step by step” |
| Inconsistent output format | 3-5 few-shot examples | Prefill or JSON schema | Longer rule lists |
| Wrong labels on edge cases | Add edge-case examples | Clarify label definitions | More generic examples |
| Tone too stiff or too casual | Role + audience in system prompt | One style example | Long persona backstories |
| Factual errors | Retrieval (RAG) and citations | Verification step | Expert persona |
Real-World Workflows
Workflow 1: Ticket triage (few-shot + light role)
A SaaS team routes 5,000 tickets a day. A zero-shot prompt mislabels tickets that mention both billing and bugs. Adding four examples, including one mixed case, fixes the boundary. A one-line role sets the output to “label only.” Setting temperature 0 keeps the labels stable across runs.
Workflow 2: Financial variance analysis (CoT)
An analyst asks a standard model to explain a budget gap across 12 line items. A direct prompt skips arithmetic steps. A <thinking> scratchpad forces the model to compute each variance first. The parsed <answer> block feeds a report.
Workflow 3: Reasoning model migration (strip, then test)
A team moves a CoT-heavy prompt from a standard model to a reasoning model. Accuracy drops. They remove the “think step by step” lines and the five reasoning exemplars, then run a zero-shot ablation against their eval set. Accuracy recovers. They re-add one format example only.
Workflow 4: Stacked prompt for a customer assistant
Combine techniques by layer:
- System prompt: role, audience, hard constraints
- User turn: 2-3 examples for the output format
- Task: the actual input, wrapped in XML tags
- Model setting: extended thinking only for escalation cases
Test each layer separately. If you cannot show a layer helps on your eval set, remove it.
Testing Protocol
Do not trust intuition. Run this loop:
- Build an eval set of 50-200 real inputs with known-good outputs.
- Run a bare zero-shot baseline.
- Add one technique at a time.
- Score accuracy, format validity, and cost per 1,000 requests.
- Keep only the techniques that beat the baseline.
| Metric | How to measure |
|---|---|
| Accuracy | Exact match or rubric-graded score |
| Format validity | JSON parse rate, schema pass rate |
| Cost | Input + output tokens × price per 1M tokens |
| Latency | p50 and p95 response time |
FAQ
Is chain-of-thought prompting still worth using?
Yes, on standard models and for multi-step tasks. It adds little on reasoning models, which already generate internal reasoning. OpenAI’s guidance says prompting those models to think step by step is unnecessary. Test zero-shot first.
How many few-shot examples should I use?
Start with 3-5. Cover edge cases and balance the labels. For reasoning models, test zero-shot first, because examples can reduce performance. Add examples back only for format control.
Does role prompting make an LLM more accurate?
Usually not. Large studies found personas do not reliably improve factual accuracy, and some hurt it. Role prompts do help with tone, depth, and audience. Use them for style, and use retrieval or verification for accuracy.
Can I combine chain-of-thought, few-shot, and role prompts?
Yes. Put the role in the system prompt, examples in the user turn, and reasoning instructions only where the task needs them. Add one layer at a time and measure each against a zero-shot baseline.




