Pretraining, Fine-Tuning, and Post-Training: How an LLM Is Made

A large language model does not arrive as a chatbot. It starts as a next-token predictor trained on trillions of tokens, then passes through supervised tuning and preference optimization before it follows instructions, refuses unsafe requests, or writes working code. Each stage uses different data, different objectives, and very different compute budgets. Understanding the pipeline tells you why base models ramble, why instruct models behave, and which stage to touch when you need to change model behavior.

Quick Takeaways

• Pretraining teaches language, facts, and reasoning patterns through next-token prediction on 1T to 30T+ tokens. It consumes roughly 90%+ of total compute.
• Fine-tuning (supervised, or SFT) adapts the model to instruction following or a domain using thousands to millions of curated examples.
• Post-training is the umbrella for SFT plus preference optimization (RLHF, DPO) and reinforcement learning with verifiable rewards. It shapes helpfulness, safety, and reasoning.
• Most teams should not pretrain. Start with prompting, then RAG, then LoRA fine-tuning, in that order.

Figures are order-of-magnitude ranges that vary by lab and model.

Stage 1: Pretraining

Pretraining produces the base model. The model reads text and predicts the next token. Repeated across trillions of tokens, this simple objective forces it to learn grammar, world knowledge, code syntax, and basic reasoning.

The Objective

Pretraining minimizes cross-entropy loss over the next token:

L = −Σ log P(x_t | x_1, …, x_{t−1})

The model assigns a probability to every token in its vocabulary (typically 32K to 256K tokens) at each position. Lower loss means better predictions.

Data Pipeline

Raw data does not go straight into training. A typical pipeline runs:

1. Collection. Web crawls, books, code repositories, scientific papers, and licensed corpora.
2. Filtering. Language identification, quality classifiers, and heuristic rules remove boilerplate and spam.
3. Deduplication. Exact and fuzzy dedup (for example MinHash) cut memorization and wasted compute.
4. Safety and privacy scrubbing. Remove PII and toxic or illegal content.
5. Tokenization. Byte-pair encoding (BPE) or a similar scheme converts text to integer IDs.
6. Mixing. Weight domains such as web, code, and math. Code and math often get upsampled because they improve reasoning.

Data quality now matters as much as quantity. Many recent gains come from better filtering and synthetic data, not larger raw crawls.

Architecture and Scaling

Nearly all frontier LLMs use a decoder-only Transformer. Key specs to know:

• Parameters: from 1B (edge models) to hundreds of billions, or more in sparse systems.
• Mixture of Experts (MoE): only a subset of parameters activates per token, cutting inference cost per token at a given total size.
• Context window: pretrained at 4K–8K tokens, then extended to 128K–1M+ in a late stage using positional-encoding tricks such as RoPE scaling.
• Precision: BF16 is common, with FP8 increasingly used for training speed.

Scaling laws guide the budget. The Chinchilla result suggested roughly 20 tokens per parameter for compute-optimal training. Modern open models often train far beyond that ratio, because a smaller model trained longer is cheaper to serve.

The Base Model Problem

A base model completes text. Ask it “What is the capital of France?” and it may continue with more quiz questions, because that pattern appears in its training data. It is knowledgeable but not an assistant. That gap is what the next stages close.

Stage 2: Fine-Tuning

Fine-tuning continues training on a much smaller, curated dataset. The most common form is supervised fine-tuning (SFT), also called instruction tuning.

How SFT Works

Each example is a prompt paired with an ideal response. The loss is the same cross-entropy, but computed only on the response tokens. The prompt is masked.

Typical SFT settings:

Quality beats volume here. A few thousand carefully written examples can shift style and format dramatically.

Full Fine-Tuning vs. Parameter-Efficient Methods

LoRA freezes the base weights and learns small low-rank matrices W = W₀ + BA, where the rank r is typically 8–64. QLoRA quantizes the frozen base to 4-bit (NF4), letting you tune a multi-billion-parameter model on one consumer or workstation GPU.

Copy-Paste Example: LoRA SFT with Hugging Face

pip install transformers peft trl datasets bitsandbytes accelerate

from datasets import load_dataset
from peft import LoraConfig
from trl import SFTTrainer, SFTConfig

# Any chat-formatted dataset with a "messages" column
dataset = load_dataset("trl-lib/Capybara", split="train")

peft_config = LoraConfig(
    r=16,                      # LoRA rank
    lora_alpha=32,             # scaling factor
    lora_dropout=0.05,
    target_modules="all-linear",
    task_type="CAUSAL_LM",
)

args = SFTConfig(
    output_dir="sft-lora-out",
    num_train_epochs=2,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,   # effective batch size = 32
    learning_rate=2e-4,
    bf16=True,
    max_length=2048,
    logging_steps=10,
)

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-1.5B",       # swap for any base model you can run
    args=args,
    train_dataset=dataset,
    peft_config=peft_config,
)
trainer.train()
trainer.save_model()

Pick a base model small enough for your GPU. Verify the dataset schema matches your TRL version before a long run.

When Fine-Tuning Is the Wrong Tool

Fine-tuning changes behavior and style. It is a poor way to inject fast-changing facts. Use RAG for knowledge that updates, and fine-tune for tone, format, tool-use patterns, or domain-specific reasoning.

Stage 3: Post-Training

Post-training is the full set of steps after pretraining that turn a base model into a product. It includes SFT, but the defining additions are preference optimization and reinforcement learning.

Preference Optimization: RLHF and DPO

Demonstrations only show one good answer. Preferences teach the model which of several answers is better.

RLHF (Reinforcement Learning from Human Feedback)

1. Sample multiple responses per prompt.
2. Humans (or AI judges) rank them.
3. Train a reward model on those rankings.
4. Optimize the policy against the reward model with PPO or a similar algorithm, with a KL penalty to keep it close to the SFT model.

DPO (Direct Preference Optimization)
DPO skips the reward model. It trains directly on pairs of (chosen, rejected) responses with a classification-style loss. It is simpler, cheaper, and more stable, so many open-model recipes use it.

Reinforcement Learning with Verifiable Rewards

For math, code, and logic, correctness can be checked automatically. A unit test passes or a final answer matches. The model samples many reasoning attempts, and answers that verify get reinforced. This approach underpins modern reasoning models that produce long chains of thought before answering. It scales because it needs no human labeling for each sample.

Safety and Alignment

Post-training is also where refusal behavior, honesty, and harm avoidance are shaped. Teams combine SFT on refusal examples, preference data, red-teaming, and automated evaluations. Alignment is never “finished.” Models are re-evaluated and patched continuously.

Stage Comparison at a Glance

Practical Workflows and Real-World Use Cases

Workflow 1: Customer Support Assistant

1. Start with a strong instruct model and a system prompt.
2. Add RAG over your help center to ground answers.
3. If tone or ticket-routing format is inconsistent, collect 2K–10K reviewed transcripts.
4. Run LoRA SFT on them.
5. Add DPO using pairs where agents preferred one reply over another.

Workflow 2: Domain Specialist (Legal, Medical, Finance)

1. Continue pretraining (“domain-adaptive pretraining”) only if you hold billions of tokens of specialized text and the base model clearly lacks the vocabulary.
2. Otherwise, SFT on curated expert Q&A.
3. Evaluate against a held-out benchmark written by domain experts.
4. Keep a human review step for high-stakes output.

Workflow 3: Cheaper Inference with Distillation

Use a large “teacher” model to generate high-quality responses, then SFT a small “student” on them. This transfers behavior at a fraction of the serving cost.

Choosing the Right Intervention

Common Mistakes

• Training on low-quality data. Noisy SFT examples teach noisy behavior.
• Over-training. More than 3 epochs of SFT often causes overfitting and catastrophic forgetting.
• Skipping evaluation. Hold out a test set and track both task metrics and general capability regressions.
• Using fine-tuning to add facts. The model may memorize inconsistently. Use retrieval.
• Ignoring the chat template. Mismatched special tokens quietly degrade results.

FAQ

What is the difference between pretraining and fine-tuning?

Pretraining trains a model from scratch on trillions of tokens to predict the next word, giving it broad knowledge. Fine-tuning continues training on a small curated dataset to teach a specific behavior, such as following instructions or writing in a given style.

Is post-training the same as fine-tuning?

Not exactly. Post-training is the broader phase after pretraining. It includes supervised fine-tuning, but also preference optimization (RLHF, DPO) and reinforcement learning. Fine-tuning is one component of post-training.

How much data do I need to fine-tune an LLM?

For style or format changes, 1,000 to 10,000 high-quality examples often suffice. Broader instruction tuning uses tens of thousands to millions. Data quality matters more than quantity.

Should I fine-tune or use RAG?

Use RAG when the model needs current or proprietary facts. Fine-tune when you need consistent behavior, tone, or structure. Many production systems combine both.