A/B Testing Fundamentals: Design, Run, and Analyze Your First Experiment

You changed a button color and conversions rose 4%. Was it the button, or random noise? A/B testing answers that question with statistical rigor. It splits users into randomized groups, applies one change to one group, and measures whether the observed difference exceeds what chance alone would produce.

This guide walks through the full workflow: framing a testable hypothesis, calculating sample size, randomizing users, running the test, and analyzing results in Python with statsmodels and SciPy.

Quick Takeaways

  • A/B testing is a randomized controlled experiment: control (A) vs. variant (B), one primary metric, one pre-defined decision rule.
  • Calculate sample size before launch. Power analysis using the baseline rate, MDE, α, and power (1−β) prevents underpowered tests.
  • Never peek and stop early. Repeated significance checks inflate the false positive rate far beyond your nominal α = 0.05.
  • Report effect size and confidence intervals, not just the p-value.
Concept Typical Default Purpose
Significance level (α) 0.05 Max tolerated false positive rate
Power (1−β) 0.80 Probability of detecting a true effect
MDE Business-defined Smallest lift worth detecting
Randomization unit User ID Prevents cross-contamination

What Is A/B Testing?

A/B testing compares two versions of a single element to determine which performs better on a target metric. Users are randomly assigned to control (A) or treatment (B). Randomization balances both observed and unobserved confounders across groups, so any systematic difference in the outcome is attributable to the change.

Statistically, A/B testing is a two-sample hypothesis test. The most common forms:

  • Two-proportion z-test for binary outcomes (converted vs. not).
  • Welch’s t-test for continuous outcomes (revenue per user, session duration).
  • Mann-Whitney U test for skewed, non-normal distributions.

Step 1: Design the Experiment

Define a Testable Hypothesis

A strong hypothesis states the change, the expected direction, and the metric.

  • Weak: “A new checkout page will be better.”
  • Strong: “Replacing the 3-step checkout with a single-page checkout will increase the checkout completion rate from 12% to at least 13.5%.”

Formalize it:

  • H₀ (null): p_B = p_A (no difference in conversion rate).
  • H₁ (alternative): p_B ≠ p_A (two-sided) or p_B > p_A (one-sided).

Use two-sided tests by default. They protect against shipping a variant that actually hurts performance.

Choose Metrics

Metric Type Role Example
Primary (OEC) Decides the test Checkout completion rate
Secondary Explains the “why” Time to complete, cart abandonment
Guardrail Detects harm Page load time, refund rate, error rate

Pick one primary metric. Testing many metrics without correction inflates false positives (the multiple comparisons problem).

Select the Randomization Unit

Randomize at the level at which the metric is measured. For a user-level conversion metric, randomize by user ID, not by session or page view. Randomizing by session lets one user see both variants, which violates independence assumptions and contaminates results.

Use deterministic hashing so users always see the same variant:

import hashlib

def assign_variant(user_id: str, experiment_name: str, split: float = 0.5) -> str:
    """Deterministically assign a user to 'control' or 'treatment'."""
    # Salt with the experiment name so assignments are independent across experiments
    key = f"{experiment_name}:{user_id}".encode("utf-8")
    # Hash to a stable integer, then map to [0, 1)
    bucket = int(hashlib.md5(key).hexdigest(), 16) % 10_000 / 10_000
    return "control" if bucket < split else "treatment"

print(assign_variant("user_48213", "single_page_checkout_v1"))

Step 2: Calculate Sample Size

Running a test without a sample size calculation is the most common beginner mistake. An underpowered test misses real effects and produces unstable estimates.

Four inputs determine sample size:

Input Symbol Effect on Required Sample
Baseline conversion rate p₁ Rates nearer 50% need more users
Minimum detectable effect MDE Smaller MDE → much larger sample
Significance level α Lower α → larger sample
Power 1−β Higher power → larger sample
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

baseline = 0.12          # Current conversion rate
target = 0.135           # Smallest lift worth detecting (12% -> 13.5%)
alpha = 0.05             # Two-sided false positive rate
power = 0.80             # 80% chance of detecting a true effect

# Cohen's h: standardized effect size for two proportions
effect_size = proportion_effectsize(target, baseline)

analysis = NormalIndPower()
n_per_group = analysis.solve_power(
    effect_size=effect_size,
    alpha=alpha,
    power=power,
    ratio=1.0,           # Equal group sizes
    alternative="two-sided",
)

print(f"Required sample size per group: {int(n_per_group):,}")

This yields roughly 8,500 users per group. Divide by daily eligible traffic to estimate duration. Round up to full weeks to capture day-of-week seasonality.

Step 3: Run the Experiment

Follow these operational rules.

  1. Run for full weekly cycles. User behavior differs on weekdays and weekends.
  2. Do not change the experiment mid-flight. Altering traffic splits or variants invalidates the analysis.
  3. Do not peek and stop early. Checking p-values daily and stopping at the first p < 0.05 can raise the real false positive rate to 20-30%. If you need early stopping, use sequential testing or always-valid p-values.
  4. Validate randomization with an SRM check. A Sample Ratio Mismatch signals a bug in assignment or logging.
from scipy.stats import chisquare

observed = [8612, 8391]            # Users actually assigned: control, treatment
expected = [sum(observed) / 2] * 2 # Expected 50/50 split

stat, p_value = chisquare(f_obs=observed, f_exp=expected)
print(f"SRM p-value: {p_value:.4f}")

# A very low p-value (commonly < 0.001) indicates an SRM: do not trust the results
if p_value < 0.001:
    print("WARNING: Sample Ratio Mismatch detected.")

Step 4: Analyze the Results

Two-Proportion Z-Test

import numpy as np
from statsmodels.stats.proportion import proportions_ztest, proportion_confint

# Observed data
conversions = np.array([1034, 1148])   # [control, treatment]
users = np.array([8612, 8391])         # [control, treatment]

# Two-sided z-test for the difference in proportions
z_stat, p_value = proportions_ztest(count=conversions, nobs=users, alternative="two-sided")

rate_control, rate_treatment = conversions / users
abs_lift = rate_treatment - rate_control
rel_lift = abs_lift / rate_control

# 95% confidence interval for the difference (Wald approximation)
se = np.sqrt(rate_control * (1 - rate_control) / users[0]
             + rate_treatment * (1 - rate_treatment) / users[1])
ci_low, ci_high = abs_lift - 1.96 * se, abs_lift + 1.96 * se

print(f"Control: {rate_control:.3%} | Treatment: {rate_treatment:.3%}")
print(f"Absolute lift: {abs_lift:.3%} | Relative lift: {rel_lift:.2%}")
print(f"Z = {z_stat:.3f}, p-value = {p_value:.4f}")
print(f"95% CI for difference: [{ci_low:.3%}, {ci_high:.3%}]")

Continuous Metrics: Welch’s T-Test

from scipy import stats

# revenue_control / revenue_treatment: per-user revenue arrays
t_stat, p_value = stats.ttest_ind(
    revenue_treatment, revenue_control,
    equal_var=False,        # Welch's correction: does not assume equal variances
)

For heavily skewed revenue data, consider bootstrap confidence intervals or winsorizing extreme outliers.

Interpret Correctly

Result Interpretation Decision
p < α, CI excludes 0, lift ≥ MDE Statistically and practically significant Ship
p < α, lift < MDE Significant but too small to matter Weigh cost vs. benefit
p ≥ α, CI narrow around 0 No meaningful effect Do not ship
p ≥ α, CI wide Inconclusive (underpowered) Collect more data or redesign

A p-value is the probability of seeing data this extreme if H₀ were true. It is not the probability that the variant is better.

Common Statistical Pitfalls

Pitfall Why It Breaks Results Fix
Peeking Inflates false positive rate Fixed horizon or sequential tests
Multiple comparisons More metrics → more false hits Bonferroni or Benjamini-Hochberg
Novelty effect Users click new things temporarily Run longer; segment new vs. returning
Simpson’s paradox Aggregate trend reverses in segments Check balanced allocation per segment
Interference Users affect each other (marketplaces, social) Cluster or switchback randomization
Low power Real effects go undetected Pre-test power analysis

Real-World Use Cases

E-Commerce: Single-Page Checkout

An online retailer tests a one-page checkout against its three-step flow. The primary metric is checkout completion rate. Guardrails include payment error rate and page load time. A 1.3-point absolute lift with a CI excluding zero justifies the engineering rollout.

SaaS: Onboarding Email Sequence

A SaaS team tests a shortened onboarding email series. The metric is 7-day activation rate. Because activation unfolds over days, the team waits for the full 7-day observation window on the last enrolled user before analysis.

Media: Headline Testing

A publisher tests two headline styles. Click-through rate is the primary metric, but the novelty effect is strong. The team analyzes returning readers separately to confirm the lift persists.

Pre-Launch Checklist

  • [ ] Hypothesis written with metric, direction, and MDE
  • [ ] Single primary metric and guardrails defined
  • [ ] Sample size and duration calculated
  • [ ] Randomization unit and hashing logic tested
  • [ ] Logging verified for both variants
  • [ ] SRM check scripted
  • [ ] Decision rule documented before launch

Frequently Asked Questions

How long should an A/B test run?

Run it until it reaches the pre-calculated sample size, rounded up to whole weeks. Most tests need one to four weeks. Stopping earlier because results look significant inflates false positives.

What is a good sample size for an A/B test?

It depends on the baseline rate, MDE, α, and power. Detecting a lift from 12% to 13.5% at α = 0.05 and 80% power requires about 8,500 users per group. Smaller expected lifts require substantially larger samples.

What is statistical significance in A/B testing?

A result is statistically significant when its p-value falls below the chosen α (usually 0.05). This means the observed difference would be unlikely if there were truly no effect. It does not measure the size or business value of the effect.

Can I run an A/B test with low traffic?

Yes, with trade-offs. Target a larger MDE, test higher-impact changes, use a more sensitive metric, apply CUPED variance reduction, or consider Bayesian methods. Low traffic makes small effects undetectable.

Hot this week

Humanoids in Factories and Warehouses: What They Actually Do Today

Which humanoid robots do real work in factories and warehouses today? See deployment data, ROI math and ROS 2 code. Read the 2026 guide.

Debugging Robot Code: Logs, Serial Output, rosbag, and Foxglove

Debug robot code faster with ROS 2 logging, serial capture, rosbag2 recording, and Foxglove. Copy the configs and scripts into your stack today.

Build a Mapping Robot With SLAM From Scratch

Build a SLAM mapping robot from scratch: wiring, odometry, LiDAR, ROS 2 slam_toolbox config and tuning. Follow the full guide and map your first room.

Depth Cameras: Stereo, Time-of-Flight, and Structured Light Compared

Compare stereo, time-of-flight, and structured light depth cameras: accuracy, range, sunlight, ROS2 setup. Pick the right sensor for your robot.

Python vs C++ for Robotics: Which Should You Use?

Python vs C++ for robotics compared: latency, ROS2 performance, control loops, and code examples. Learn which language fits your robot. Read the guide.

Topics

Humanoids in Factories and Warehouses: What They Actually Do Today

Which humanoid robots do real work in factories and warehouses today? See deployment data, ROI math and ROS 2 code. Read the 2026 guide.

Debugging Robot Code: Logs, Serial Output, rosbag, and Foxglove

Debug robot code faster with ROS 2 logging, serial capture, rosbag2 recording, and Foxglove. Copy the configs and scripts into your stack today.

Build a Mapping Robot With SLAM From Scratch

Build a SLAM mapping robot from scratch: wiring, odometry, LiDAR, ROS 2 slam_toolbox config and tuning. Follow the full guide and map your first room.

Depth Cameras: Stereo, Time-of-Flight, and Structured Light Compared

Compare stereo, time-of-flight, and structured light depth cameras: accuracy, range, sunlight, ROS2 setup. Pick the right sensor for your robot.

Python vs C++ for Robotics: Which Should You Use?

Python vs C++ for robotics compared: latency, ROS2 performance, control loops, and code examples. Learn which language fits your robot. Read the guide.

Build a ROS 2 Rover: Raspberry Pi 5, Motors, and LiDAR

Build a ROS 2 rover with Raspberry Pi 5, RPLIDAR C1, and a Pico motor controller. Copy the code, wire it, and map your room. Start building now.

Imitation Learning for Robots: Teach by Demonstration

Learn imitation learning for robots: behavior cloning, DAgger, ACT, and diffusion policies with ROS2 code. Build your first policy today.

LiDAR for Robots: 2D, 3D, and Solid-State Compared

Compare 2D, 3D, and solid-state LiDAR for robots. See specs, ROS2 configs, code, and a selection workflow. Pick the right sensor now.

Related Articles

Popular Categories