How Do Large Language Models Work? A Plain-English Explanation

A large language model (LLM) does one thing: it predicts the next token in a sequence. Everything else, from writing code to summarizing legal contracts, emerges from doing that single task at massive scale with a neural network trained on trillions of words. This guide walks through the full pipeline, from raw text to a chat response, with the specs, formulas, and code you need to understand each stage.

Quick Takeaways

• LLMs predict tokens, not meaning. They convert text into numbers, run it through a Transformer network, and output a probability distribution over the next token.
• Self-attention is the core mechanism. It lets every token weigh every other token in the context window, typically 8K to 1M+ tokens in modern models.
• Three training stages build a useful assistant: pretraining on raw text, supervised fine-tuning (SFT), and alignment through RLHF or similar methods.
• Parameters are the learned weights. Frontier models range from roughly 7B (small open models) to hundreds of billions or more (large closed models).

What Is a Large Language Model?

An LLM is a neural network with billions of adjustable numbers, called parameters, trained to model the statistical structure of human language. Given the text “The capital of France is,” the model assigns a high probability to ” Paris” and low probabilities to everything else.

Three properties define the category:

• Large: Parameter counts run from 1B to well over 1T in some architectures.
• Language: Training data is text, plus code, and increasingly images, audio, and video.
• Model: It is a function that maps an input sequence to a probability distribution.

Modern LLMs are almost all built on the Transformer architecture, introduced in the 2017 paper “Attention Is All You Need.”

The LLM Pipeline at a Glance

Step 1: Tokenization

Models cannot read letters. They read tokens, which are chunks of text that are often whole words, word fragments, or punctuation.

Most LLMs use Byte-Pair Encoding (BPE) or a close variant. BPE starts with individual bytes and repeatedly merges the most frequent adjacent pairs until it reaches a target vocabulary size, typically 32,000 to 256,000 entries.

A rough rule for English: 1 token ≈ 4 characters ≈ 0.75 words. So 1,000 tokens is about 750 words.

# pip install tiktoken
import tiktoken

enc = tiktoken.get_encoding("cl100k_base")  # BPE vocabulary of ~100K tokens
text = "Large language models predict tokens."
ids = enc.encode(text)

print(ids)                           # list of integer token IDs
print([enc.decode([i]) for i in ids])  # shows how the text was split
print(len(ids), "tokens")

Tokenization explains several well-known quirks. Models struggle to count letters in a word because they never see individual letters, only token chunks. Rare words and non-English text often split into more tokens, which raises cost and shrinks the usable context.

Step 2: Embeddings

Each token ID is looked up in an embedding matrix of shape [vocab_size, d_model]. The result is a dense vector that encodes the token’s learned meaning.

Typical d_model values:

• Small models (about 7B parameters): 4,096
• Mid-size models (about 70B): 8,192
• Very large models: 12,288 or higher

Similar words end up near each other in this space. Because Transformers process tokens in parallel, they also need positional information. Modern models use Rotary Position Embeddings (RoPE), which encode token order by rotating query and key vectors.

Step 3: The Transformer and Self-Attention

This is where the model builds context. A Transformer is a stack of identical blocks, commonly 32 to 120+ layers. Each block has two parts:

1. Multi-head self-attention
2. A feed-forward network (FFN)

How Self-Attention Works

For every token, the model creates three vectors using learned weight matrices:

• Query (Q): what this token is looking for
• Key (K): what this token offers
• Value (V): the information this token carries

The attention formula is:

Attention(Q, K, V) = softmax(QKᵀ / √d_k) · V

In plain terms: compare each token’s query against every key, convert the scores into weights with softmax, then blend the value vectors using those weights. The √d_k scaling keeps scores stable as dimensions grow.

In decoder-only LLMs, a causal mask blocks each token from seeing future tokens. This is what makes next-token prediction a valid training task.

Multi-Head Attention

Instead of one attention operation, the model runs 8 to 128 heads in parallel. Each head learns to track different relationships: syntax, coreference, long-range dependencies, and so on. Outputs are concatenated and projected back to d_model.

The Feed-Forward Network

After attention, each token passes through a two-layer network with a nonlinearity such as SwiGLU. The hidden layer is usually about 4x the size of d_model. Researchers believe much of the model’s stored factual knowledge lives in these layers.

Efficiency Tricks in Modern Models

Step 4: From Logits to Text (Decoding)

The final layer produces a logit for every token in the vocabulary. Softmax turns these into probabilities. The model then picks one token, appends it to the input, and repeats. This loop is called autoregressive generation.

Sampling Controls You Can Tune

A practical starting point: temperature 0.0 to 0.3 for code, extraction, and factual tasks. Use 0.7 to 1.0 for brainstorming and creative writing.

Here is a minimal call using the Anthropic API:

# pip install anthropic
import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from environment

response = client.messages.create(
    model="claude-sonnet-4-5",   # replace with a current model string
    max_tokens=500,              # output cap
    temperature=0.2,             # low randomness for factual answers
    messages=[
        {"role": "user", "content": "Explain self-attention in two sentences."}
    ],
)

print(response.content[0].text)
print(response.usage)  # input and output token counts

How LLMs Are Trained

Training happens in stages. Each one changes what the model can do.

Stage 1: Pretraining

The model reads trillions of tokens from web pages, books, code, and papers. For each position, it predicts the next token, and cross-entropy loss measures how wrong the prediction was. Backpropagation and an optimizer such as AdamW then nudge billions of parameters to reduce that loss.

Key facts:

• Training data volume: often 1 to 15+ trillion tokens
• Hardware: thousands of GPUs or TPUs running for weeks or months
• Precision: BF16 or FP8 to save memory and speed up math
• Result: a base model that completes text well but does not reliably follow instructions

Stage 2: Supervised Fine-Tuning (SFT)

Humans write thousands to millions of high-quality prompt-and-response pairs. Training on these teaches the model the format of helpful conversation.

Stage 3: Alignment

The model is tuned to be helpful, honest, and safe. Common methods:

Scaling Laws

Research on scaling laws shows that loss falls predictably as you increase parameters, data, and compute. The influential Chinchilla result suggested that, for a fixed compute budget, parameters and training tokens should grow together. In practice, many teams now train smaller models on far more data than Chinchilla-optimal, because a smaller model is cheaper to serve.

Parameters, Context Windows, and Memory

An LLM has no persistent memory between calls. The context window is its only working memory. Anything the model “remembers” in a chat is simply the earlier conversation re-sent as input on every turn.

Memory Math for Local Deployment

Weight memory is roughly parameters × bytes per parameter:

These figures cover weights only. Add memory for the KV cache and activations.

Closed vs. Open Models

Check each provider’s pricing page for current per-token rates, since they change often.

Why LLMs Hallucinate

An LLM optimizes for plausible continuations, not verified truth. When it lacks a fact, it still produces fluent text that fits the pattern. Hallucinations rise when:

• The topic is rare in training data
• The prompt asks for specifics such as citations, statistics, or quotes
• The context is long and noisy
• Temperature is high

How Teams Reduce Hallucinations

Retrieval-Augmented Generation (RAG) is the most common fix. The system retrieves relevant documents at query time and places them in the prompt, so the model answers from supplied evidence.

# Minimal RAG skeleton (conceptual)
# pip install anthropic sentence-transformers numpy
import numpy as np
from sentence_transformers import SentenceTransformer
import anthropic

embedder = SentenceTransformer("all-MiniLM-L6-v2")  # 384-dim embeddings
docs = [
    "Our refund window is 30 days from delivery.",
    "Shipping to the EU takes 5 to 7 business days.",
    "Support is available Monday to Friday, 9am to 5pm UTC.",
]
doc_vecs = embedder.encode(docs, normalize_embeddings=True)

def retrieve(query, k=2):
    q = embedder.encode([query], normalize_embeddings=True)[0]
    scores = doc_vecs @ q                 # cosine similarity (vectors normalized)
    return [docs[i] for i in np.argsort(-scores)[:k]]

query = "How long do I have to return an item?"
context = "\n".join(retrieve(query))

client = anthropic.Anthropic()
reply = client.messages.create(
    model="claude-sonnet-4-5",            # replace with a current model string
    max_tokens=300,
    temperature=0,
    system="Answer only from the provided context. If the answer is missing, say you don't know.",
    messages=[{"role": "user", "content": f"Context:\n{context}\n\nQuestion: {query}"}],
)
print(reply.content[0].text)

Other mitigations include lower temperature, citation requirements, output validation, and human review for high-stakes use.

Practical Workflows and Real-World Use Cases

Prompt Engineering Template

A reliable structure for production prompts:

ROLE: You are a [specific expert role].
TASK: [One clear objective].
CONTEXT: [Background data, documents, constraints].
FORMAT: [Exact output schema, such as JSON keys or bullet count].
RULES: [What to avoid, tone, length limits].
EXAMPLES: [1 to 3 input-output pairs].

Adding few-shot examples often improves format consistency more than adding extra instructions. For reasoning-heavy tasks, ask the model to work through the problem step by step before giving a final answer.

Where Enterprises Deploy LLMs

Limits Worth Knowing

• Knowledge cutoff: The model only knows data up to its training date unless given tools or retrieval.
• No true understanding claims: Models show strong reasoning on many tasks but fail unpredictably on others.
• Context degradation: Accuracy can drop when key facts sit deep inside very long prompts.
• Prompt sensitivity: Small wording changes can shift outputs.
• Bias: Models inherit patterns from training data.

Frequently Asked Questions

How do large language models learn?

They learn by predicting the next token across trillions of text examples. After each prediction, the training algorithm measures the error and adjusts billions of parameters through backpropagation. Later stages, SFT and alignment, teach the model to follow instructions and behave safely.

Do LLMs actually understand language?

LLMs build rich internal representations that capture grammar, facts, and reasoning patterns, and they perform well on many comprehension tasks. Whether this counts as “understanding” in a human sense is debated. What is certain: they work by statistical pattern learning, not by looking up stored sentences.

What is the difference between an LLM and ChatGPT or Claude?

An LLM is the underlying neural network. Products like ChatGPT and Claude are applications built on LLMs, adding a chat interface, system prompts, safety layers, tools, and memory features.

Why do LLMs make things up?

Because they generate the most probable next token, not a verified fact. When knowledge is missing, the model still produces fluent text. RAG, lower temperature, and source-grounded prompts reduce this behavior.