Reinforcement Learning Explained: Agents, Rewards, and Policies

An RL agent learns by acting, failing, and adjusting. No labeled dataset tells it the right answer. It receives a number after each action and learns to make that number grow over time. This loop powers game-playing systems, robotic controllers, and the RLHF stage that shapes modern LLMs. This guide covers the core vocabulary, the math behind it, the major algorithm families, a working Python implementation, and where RL runs in production.

Quick Takeaways

• Reinforcement learning (RL) trains an agent to choose actions in an environment to maximize cumulative reward.
• A policy (π) maps states to actions. Learning a good policy is the entire goal.
• Value-based methods (Q-learning, DQN) estimate returns. Policy-gradient methods (REINFORCE, PPO) optimize the policy directly.
• RL underpins RLHF and verifiable-reward training for LLMs, and it drives robotics, recommendation, and control systems.

What Is Reinforcement Learning?

Reinforcement learning is a machine learning paradigm where an agent learns a decision-making strategy through trial and error. The agent observes a situation, acts, and receives feedback in the form of a reward. Its objective is to maximize the expected cumulative reward over time, not just the next reward.

Two properties separate RL from supervised learning:

1. Delayed feedback. A chess move may only prove good or bad 40 moves later. This is the credit assignment problem.
2. Data depends on the agent. The agent’s own actions determine what it sees next, so the data distribution shifts as it learns.

The Core Components

Agent and Environment

The agent is the learner and decision-maker. The environment is everything it interacts with. At each timestep t, the loop runs as follows:

1. The agent observes state sₜ.
2. It selects action aₜ.
3. The environment returns reward rₜ₊₁ and next state sₜ₊₁.

State, Action, and Reward

Policy

The policy π(a|s) defines the agent’s behavior. It comes in two forms:

• Deterministic: a = π(s). The same state always yields the same action.
• Stochastic: π(a|s) is a probability distribution. Sampling from it drives exploration.

Value Functions

A value function estimates how good a state or action is under a policy.

• State value: V^π(s) = expected return starting from s, following π.
• Action value: Q^π(s,a) = expected return after taking a in s, then following π.

Return and Discounting

The return Gₜ sums future rewards, discounted by γ (gamma):

G_t = r_{t+1} + γ·r_{t+2} + γ²·r_{t+3} + ...

A γ of 0.99 makes the agent far-sighted. A γ of 0.9 makes it favor near-term rewards. Values below 1 also keep infinite-horizon sums finite.

The Math: Markov Decision Processes

Most RL problems are formalized as a Markov Decision Process (MDP), defined by the tuple (S, A, P, R, γ):

• S: set of states
• A: set of actions
• P(s’|s,a): transition probabilities
• R(s,a): reward function
• γ: discount factor

The Markov property says the future depends only on the current state, not the full history. The Bellman equation expresses the recursive structure of value:

Q*(s,a) = E[ r + γ · max_a' Q*(s', a') ]

Nearly every value-based algorithm is a method for solving this equation iteratively.

Exploration vs. Exploitation

The agent must balance two goals. Exploitation uses the best-known action. Exploration tries unknown actions that might be better.

Major Algorithm Families

Value-Based Methods

Q-learning is the foundational off-policy algorithm. It updates a table of Q-values:

Q(s,a) ← Q(s,a) + α · [ r + γ · max_a' Q(s',a') − Q(s,a) ]

Here α is the learning rate, typically 0.1 for tabular problems. Deep Q-Networks (DQN) replace the table with a neural network. They add two stabilizers: an experience replay buffer (commonly 100k-1M transitions) and a target network synced every few thousand steps.

Policy-Gradient Methods

These methods adjust policy parameters θ directly by gradient ascent on expected return. REINFORCE is the simplest form. It has high variance, so practitioners add a learned baseline (the critic) to form actor-critic methods.

Modern Workhorses

PPO is the default choice for most projects. It is stable, simple to tune, and the algorithm most often used in RLHF pipelines.

Model-Free vs. Model-Based

Model-free methods learn directly from experience. Model-based methods learn a model of the environment’s dynamics and plan with it. Model-based approaches are far more sample-efficient but suffer when the learned model is inaccurate.

Hands-On: Q-Learning in Python

This example trains an agent on Gymnasium’s FrozenLake-v1, a 4×4 grid with a hidden hole hazard.

pip install gymnasium numpy

import gymnasium as gym
import numpy as np

env = gym.make("FrozenLake-v1", is_slippery=False)

n_states = env.observation_space.n    # 16
n_actions = env.action_space.n        # 4
Q = np.zeros((n_states, n_actions))   # Q-table

alpha, gamma = 0.1, 0.99              # learning rate, discount
epsilon, eps_min, eps_decay = 1.0, 0.05, 0.999
episodes = 5000

for ep in range(episodes):
    state, _ = env.reset()
    done = False
    while not done:
        # ε-greedy action selection
        if np.random.rand() < epsilon:
            action = env.action_space.sample()
        else:
            action = np.argmax(Q[state])

        next_state, reward, terminated, truncated, _ = env.step(action)
        done = terminated or truncated

        # Bellman update
        best_next = np.max(Q[next_state])
        Q[state, action] += alpha * (reward + gamma * best_next * (not terminated) - Q[state, action])
        state = next_state

    epsilon = max(eps_min, epsilon * eps_decay)

print("Learned greedy policy:", np.argmax(Q, axis=1).reshape(4, 4))

Annotations:

• (not terminated) zeroes the future term at terminal states.
• Decaying ε shifts the agent from exploration to exploitation.
• Set is_slippery=True for a stochastic environment. Expect to raise episodes to 20,000+.

Scaling Up with Stable-Baselines3

For deep RL, use a tested library rather than writing PPO from scratch:

from stable_baselines3 import PPO

model = PPO(
    "MlpPolicy", "CartPole-v1",
    learning_rate=3e-4,
    n_steps=2048,
    batch_size=64,
    gamma=0.99,
    clip_range=0.2,
    verbose=1,
)
model.learn(total_timesteps=100_000)
model.save("ppo_cartpole")

Reward Design: Where Projects Succeed or Fail

The agent optimizes exactly what you reward, not what you meant. This is reward hacking: a boat-racing agent famously looped to collect bonus targets instead of finishing the race.

Real-World Use Cases

RLHF and LLM Post-Training

RLHF trains language models in three steps:

1. Supervised fine-tuning (SFT) on demonstration data.
2. Reward model training on human preference comparisons.
3. Policy optimization (usually PPO) against the reward model, with a KL-divergence penalty keeping the model close to the SFT version.

Newer methods skip or simplify parts of this pipeline. DPO optimizes preferences directly without a separate reward model. RLVR (reinforcement learning from verifiable rewards) uses automatic checks, such as unit tests or math answer verification, as the reward signal. The RLVR approach has become central to training reasoning models.

Other Production Domains

Common Pitfalls and Practical Tips

• Seed everything. RL results vary heavily between runs. Report averages over 5+ seeds.
• Start simple. Validate your pipeline on CartPole before moving to complex environments.
• Log episode returns, not just loss. Loss curves in RL are often uninformative.
• Normalize observations and rewards to stabilize training.
• Use simulators first. Real-world exploration can be unsafe and expensive.
• Tune γ, learning rate, and entropy coefficient first. These three move results most.

When Not to Use RL

RL needs many interactions and a well-defined reward. If you have labeled data, use supervised learning. If you have logged expert behavior, consider imitation learning or offline RL. RL fits best when decisions are sequential, outcomes are delayed, and simulation is cheap.

Frequently Asked Questions

What is the difference between reinforcement learning and supervised learning?

Supervised learning trains on labeled input-output pairs and minimizes prediction error. Reinforcement learning has no labels. The agent learns from scalar rewards produced by its own actions, often delayed, and must explore to discover good behavior.

What is a policy in reinforcement learning?

A policy is the agent’s decision rule. It maps each state to an action (deterministic) or to a probability distribution over actions (stochastic). The goal of RL is to find the policy that maximizes expected cumulative reward.

Which RL algorithm should a beginner start with?

Start with tabular Q-learning to understand the Bellman update, then move to PPO via Stable-Baselines3. PPO handles discrete and continuous actions, is stable across tasks, and is widely used in practice.

How does RL relate to ChatGPT-style language models?

RL fine-tunes language models after pretraining. In RLHF, a reward model trained on human preferences scores outputs, and an algorithm like PPO updates the model to produce higher-scoring responses. Newer pipelines also use verifiable rewards for math and code.