A/B Testing Fundamentals: Design, Run, and Analyze Your First Experiment

You changed a button color and conversions rose 4%. Was it the button, or random noise? A/B testing answers that question with statistical rigor. It splits users into randomized groups, applies one change to one group, and measures whether the observed difference exceeds what chance alone would produce.

This guide walks through the full workflow: framing a testable hypothesis, calculating sample size, randomizing users, running the test, and analyzing results in Python with statsmodels and SciPy.

Quick Takeaways

• A/B testing is a randomized controlled experiment: control (A) vs. variant (B), one primary metric, one pre-defined decision rule.
• Calculate sample size before launch. Power analysis using the baseline rate, MDE, α, and power (1−β) prevents underpowered tests.
• Never peek and stop early. Repeated significance checks inflate the false positive rate far beyond your nominal α = 0.05.
• Report effect size and confidence intervals, not just the p-value.

What Is A/B Testing?

A/B testing compares two versions of a single element to determine which performs better on a target metric. Users are randomly assigned to control (A) or treatment (B). Randomization balances both observed and unobserved confounders across groups, so any systematic difference in the outcome is attributable to the change.

Statistically, A/B testing is a two-sample hypothesis test. The most common forms:

• Two-proportion z-test for binary outcomes (converted vs. not).
• Welch’s t-test for continuous outcomes (revenue per user, session duration).
• Mann-Whitney U test for skewed, non-normal distributions.

Step 1: Design the Experiment

Define a Testable Hypothesis

A strong hypothesis states the change, the expected direction, and the metric.

• Weak: “A new checkout page will be better.”
• Strong: “Replacing the 3-step checkout with a single-page checkout will increase the checkout completion rate from 12% to at least 13.5%.”

Formalize it:

• H₀ (null): p_B = p_A (no difference in conversion rate).
• H₁ (alternative): p_B ≠ p_A (two-sided) or p_B > p_A (one-sided).

Use two-sided tests by default. They protect against shipping a variant that actually hurts performance.

Choose Metrics

Pick one primary metric. Testing many metrics without correction inflates false positives (the multiple comparisons problem).

Select the Randomization Unit

Randomize at the level at which the metric is measured. For a user-level conversion metric, randomize by user ID, not by session or page view. Randomizing by session lets one user see both variants, which violates independence assumptions and contaminates results.

Use deterministic hashing so users always see the same variant:

import hashlib

def assign_variant(user_id: str, experiment_name: str, split: float = 0.5) -> str:
    """Deterministically assign a user to 'control' or 'treatment'."""
    # Salt with the experiment name so assignments are independent across experiments
    key = f"{experiment_name}:{user_id}".encode("utf-8")
    # Hash to a stable integer, then map to [0, 1)
    bucket = int(hashlib.md5(key).hexdigest(), 16) % 10_000 / 10_000
    return "control" if bucket < split else "treatment"

print(assign_variant("user_48213", "single_page_checkout_v1"))

Step 2: Calculate Sample Size

Running a test without a sample size calculation is the most common beginner mistake. An underpowered test misses real effects and produces unstable estimates.

Four inputs determine sample size:

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

baseline = 0.12          # Current conversion rate
target = 0.135           # Smallest lift worth detecting (12% -> 13.5%)
alpha = 0.05             # Two-sided false positive rate
power = 0.80             # 80% chance of detecting a true effect

# Cohen's h: standardized effect size for two proportions
effect_size = proportion_effectsize(target, baseline)

analysis = NormalIndPower()
n_per_group = analysis.solve_power(
    effect_size=effect_size,
    alpha=alpha,
    power=power,
    ratio=1.0,           # Equal group sizes
    alternative="two-sided",
)

print(f"Required sample size per group: {int(n_per_group):,}")

This yields roughly 8,500 users per group. Divide by daily eligible traffic to estimate duration. Round up to full weeks to capture day-of-week seasonality.

Step 3: Run the Experiment

Follow these operational rules.

1. Run for full weekly cycles. User behavior differs on weekdays and weekends.
2. Do not change the experiment mid-flight. Altering traffic splits or variants invalidates the analysis.
3. Do not peek and stop early. Checking p-values daily and stopping at the first p < 0.05 can raise the real false positive rate to 20-30%. If you need early stopping, use sequential testing or always-valid p-values.
4. Validate randomization with an SRM check. A Sample Ratio Mismatch signals a bug in assignment or logging.

from scipy.stats import chisquare

observed = [8612, 8391]            # Users actually assigned: control, treatment
expected = [sum(observed) / 2] * 2 # Expected 50/50 split

stat, p_value = chisquare(f_obs=observed, f_exp=expected)
print(f"SRM p-value: {p_value:.4f}")

# A very low p-value (commonly < 0.001) indicates an SRM: do not trust the results
if p_value < 0.001:
    print("WARNING: Sample Ratio Mismatch detected.")

Step 4: Analyze the Results

Two-Proportion Z-Test

import numpy as np
from statsmodels.stats.proportion import proportions_ztest, proportion_confint

# Observed data
conversions = np.array([1034, 1148])   # [control, treatment]
users = np.array([8612, 8391])         # [control, treatment]

# Two-sided z-test for the difference in proportions
z_stat, p_value = proportions_ztest(count=conversions, nobs=users, alternative="two-sided")

rate_control, rate_treatment = conversions / users
abs_lift = rate_treatment - rate_control
rel_lift = abs_lift / rate_control

# 95% confidence interval for the difference (Wald approximation)
se = np.sqrt(rate_control * (1 - rate_control) / users[0]
             + rate_treatment * (1 - rate_treatment) / users[1])
ci_low, ci_high = abs_lift - 1.96 * se, abs_lift + 1.96 * se

print(f"Control: {rate_control:.3%} | Treatment: {rate_treatment:.3%}")
print(f"Absolute lift: {abs_lift:.3%} | Relative lift: {rel_lift:.2%}")
print(f"Z = {z_stat:.3f}, p-value = {p_value:.4f}")
print(f"95% CI for difference: [{ci_low:.3%}, {ci_high:.3%}]")

Continuous Metrics: Welch’s T-Test

from scipy import stats

# revenue_control / revenue_treatment: per-user revenue arrays
t_stat, p_value = stats.ttest_ind(
    revenue_treatment, revenue_control,
    equal_var=False,        # Welch's correction: does not assume equal variances
)

For heavily skewed revenue data, consider bootstrap confidence intervals or winsorizing extreme outliers.

Interpret Correctly

A p-value is the probability of seeing data this extreme if H₀ were true. It is not the probability that the variant is better.

Common Statistical Pitfalls

Real-World Use Cases

E-Commerce: Single-Page Checkout

An online retailer tests a one-page checkout against its three-step flow. The primary metric is checkout completion rate. Guardrails include payment error rate and page load time. A 1.3-point absolute lift with a CI excluding zero justifies the engineering rollout.

SaaS: Onboarding Email Sequence

A SaaS team tests a shortened onboarding email series. The metric is 7-day activation rate. Because activation unfolds over days, the team waits for the full 7-day observation window on the last enrolled user before analysis.

Media: Headline Testing

A publisher tests two headline styles. Click-through rate is the primary metric, but the novelty effect is strong. The team analyzes returning readers separately to confirm the lift persists.

Pre-Launch Checklist

• [ ] Hypothesis written with metric, direction, and MDE
• [ ] Single primary metric and guardrails defined
• [ ] Sample size and duration calculated
• [ ] Randomization unit and hashing logic tested
• [ ] Logging verified for both variants
• [ ] SRM check scripted
• [ ] Decision rule documented before launch

Frequently Asked Questions

How long should an A/B test run?

Run it until it reaches the pre-calculated sample size, rounded up to whole weeks. Most tests need one to four weeks. Stopping earlier because results look significant inflates false positives.

What is a good sample size for an A/B test?

It depends on the baseline rate, MDE, α, and power. Detecting a lift from 12% to 13.5% at α = 0.05 and 80% power requires about 8,500 users per group. Smaller expected lifts require substantially larger samples.

What is statistical significance in A/B testing?

A result is statistically significant when its p-value falls below the chosen α (usually 0.05). This means the observed difference would be unlikely if there were truly no effect. It does not measure the size or business value of the effect.

Can I run an A/B test with low traffic?

Yes, with trade-offs. Target a larger MDE, test higher-impact changes, use a more sensitive metric, apply CUPED variance reduction, or consider Bayesian methods. Low traffic makes small effects undetectable.