Hypothesis Testing Explained: From Null Hypothesis to Decision

Every A/B test, drug trial, and model comparison asks the same question: is the pattern in this sample real, or is it noise? Hypothesis testing gives you a formal procedure to answer it. You state a claim, measure how surprising your data would be if the claim were false, and then make a decision with a known error rate. This guide walks through the full workflow, from defining H₀ to interpreting a p-value, with production-ready Python code.

Quick Takeaways

• Hypothesis testing compares observed sample data against a null hypothesis (H₀) to decide whether an effect is statistically significant.
• The p-value is the probability of seeing data at least as extreme as yours, assuming H₀ is true. It is not the probability that H₀ is true.
• Pick the test from your data type and assumptions: t-test for means, chi-square for categorical counts, Mann-Whitney U when normality fails.
• Always report effect size and a confidence interval next to the p-value. Significance alone says nothing about practical importance.

What Is Hypothesis Testing?

Hypothesis testing is a decision framework built on proof by contradiction. You assume nothing interesting is happening, then check whether your data contradicts that assumption strongly enough to reject it.

The framework has five components:

1. Null hypothesis (H₀): The default claim. No effect, no difference, no relationship.
2. Alternative hypothesis (H₁ or Hₐ): The claim you want evidence for.
3. Test statistic: A number summarizing how far your sample sits from H₀.
4. p-value: The probability of a test statistic at least this extreme under H₀.
5. Significance level (α): Your pre-set threshold for rejecting H₀, typically 0.05.

Null vs. Alternative Hypothesis

Write both hypotheses before you look at the data. Doing it afterward invites p-hacking.

You never “accept” H₀. You either reject it or fail to reject it. Failing to reject means the data lacks enough evidence, not that H₀ is proven.

One-Tailed vs. Two-Tailed Tests

• Two-tailed: Tests for any difference (H₁: μ ≠ μ₀). Use as the default.
• Right-tailed: Tests for an increase (H₁: μ > μ₀).
• Left-tailed: Tests for a decrease (H₁: μ < μ₀).

Choose the direction before collecting data. Switching to a one-tailed test after seeing the result to get under 0.05 is a form of data dredging.

The Hypothesis Testing Workflow

Step 1: State Hypotheses and Set α

Define H₀, H₁, and α. Also decide your minimum detectable effect and required sample size through a power analysis (covered below).

Step 2: Explore the Data and Check Assumptions

Parametric tests carry assumptions. Check them before trusting any output.

With n > 30 per group, the Central Limit Theorem makes the sampling distribution of the mean approximately normal, so moderate non-normality is tolerable. Heavy skew and outliers still break things.

import numpy as np
import pandas as pd
from scipy import stats

rng = np.random.default_rng(seed=42)  # reproducible randomness

# Simulated page load times (seconds) for control and variant
control = rng.normal(loc=3.20, scale=0.80, size=120)
variant = rng.normal(loc=2.95, scale=0.85, size=115)

df = pd.DataFrame({
    "load_time": np.concatenate([control, variant]),
    "group": ["control"] * len(control) + ["variant"] * len(variant),
})

# Normality check per group (H0: data is normally distributed)
for name, grp in df.groupby("group"):
    stat, p = stats.shapiro(grp["load_time"])
    print(f"{name}: Shapiro-Wilk W={stat:.3f}, p={p:.3f}")

# Equal variance check (H0: variances are equal)
lev_stat, lev_p = stats.levene(control, variant)
print(f"Levene: stat={lev_stat:.3f}, p={lev_p:.3f}")

Step 3: Compute the Test Statistic and p-value

For a two-sample comparison of means, use Welch’s t-test. It does not assume equal variances and loses almost nothing when variances are equal, so it is the safer default.

The Welch t-statistic is:

t = (x̄₁ − x̄₂) / √(s₁²/n₁ + s₂²/n₂)

# Welch's t-test: equal_var=False handles unequal variances
t_stat, p_value = stats.ttest_ind(
    control, variant,
    equal_var=False,
    alternative="two-sided",  # also: "less" or "greater"
)
print(f"t = {t_stat:.3f}, p = {p_value:.4f}")

Step 4: Apply the Decision Rule

The critical-value approach gives the same answer. Reject H₀ when the test statistic falls in the rejection region beyond the critical value.

Step 5: Quantify Effect Size and Confidence Intervals

A tiny effect becomes “significant” with a huge sample. Report magnitude too.

def cohens_d(a: np.ndarray, b: np.ndarray) -> float:
    """Standardized mean difference using pooled standard deviation."""
    n1, n2 = len(a), len(b)
    pooled_var = ((n1 - 1) * a.var(ddof=1) + (n2 - 1) * b.var(ddof=1)) / (n1 + n2 - 2)
    return (a.mean() - b.mean()) / np.sqrt(pooled_var)

d = cohens_d(control, variant)

# 95% confidence interval for the mean difference (Welch-Satterthwaite df)
diff = control.mean() - variant.mean()
se = np.sqrt(control.var(ddof=1)/len(control) + variant.var(ddof=1)/len(variant))
df_w = se**4 / (
    (control.var(ddof=1)/len(control))**2 / (len(control)-1)
    + (variant.var(ddof=1)/len(variant))**2 / (len(variant)-1)
)
ci = stats.t.interval(0.95, df_w, loc=diff, scale=se)

print(f"Cohen's d = {d:.3f}")
print(f"Mean diff = {diff:.3f}, 95% CI = ({ci[0]:.3f}, {ci[1]:.3f})")

Type I and Type II Errors

Every decision carries one of two risks.

• α is the false positive rate you choose.
• β is the false negative rate. Power = 1 − β, with 0.80 as the common target.
• Lowering α raises β unless you increase sample size.

Power Analysis in Python

Compute the required sample size before running an experiment.

from statsmodels.stats.power import TTestIndPower

analysis = TTestIndPower()
n_per_group = analysis.solve_power(
    effect_size=0.3,   # expected Cohen's d
    alpha=0.05,
    power=0.80,
    ratio=1.0,         # equal group sizes
    alternative="two-sided",
)
print(f"Required n per group: {np.ceil(n_per_group):.0f}")

Choosing the Right Test

Non-Parametric and Categorical Examples

# Mann-Whitney U: no normality assumption, compares distributions/ranks
u_stat, p_mw = stats.mannwhitneyu(control, variant, alternative="two-sided")
print(f"Mann-Whitney U={u_stat:.1f}, p={p_mw:.4f}")

# Chi-square test of independence on a 2x2 contingency table
#                 converted  not_converted
table = np.array([[48, 352],    # control
                  [71, 329]])   # variant
chi2, p_chi, dof, expected = stats.chi2_contingency(table)
print(f"chi2={chi2:.3f}, dof={dof}, p={p_chi:.4f}")
print("Expected counts:\n", expected.round(1))  # all should be >= 5

Real-World Use Cases

Use Case 1: A/B Testing a Checkout Redesign

An e-commerce team tests whether a new checkout page raises conversion. The metric is binary, so they use a two-proportion z-test.

from statsmodels.stats.proportion import proportions_ztest, proportion_confint

conversions = np.array([71, 48])      # variant, control
visitors = np.array([400, 400])

z_stat, p_val = proportions_ztest(conversions, visitors, alternative="larger")
print(f"z={z_stat:.3f}, p={p_val:.4f}")

# Confidence interval for each conversion rate (Wilson method)
for label, c, n in zip(["variant", "control"], conversions, visitors):
    lo, hi = proportion_confint(c, n, alpha=0.05, method="wilson")
    print(f"{label}: {c/n:.3f} (95% CI {lo:.3f}-{hi:.3f})")

Decision: If p ≤ 0.05, roll out the variant. Also check that the lift justifies engineering cost.

Use Case 2: Detecting Sensor Drift in Manufacturing

A plant compares current vibration readings against the calibrated baseline mean of 0.50 mm/s. A one-sample t-test flags drift before equipment fails.

readings = rng.normal(loc=0.54, scale=0.10, size=60)
t_stat, p_val = stats.ttest_1samp(readings, popmean=0.50)
print(f"t={t_stat:.3f}, p={p_val:.4f}")

Use Case 3: Comparing Two Models on the Same Folds

Compare cross-validated scores from two models on identical folds using a paired test. Note that CV folds share training data, so scores are not fully independent. Treat the p-value as approximate, or use a corrected resampled t-test or McNemar’s test on a held-out set.

scores_rf  = np.array([0.861, 0.874, 0.858, 0.869, 0.866])
scores_xgb = np.array([0.872, 0.880, 0.866, 0.879, 0.871])
t_stat, p_val = stats.ttest_rel(scores_xgb, scores_rf)
print(f"Paired t={t_stat:.3f}, p={p_val:.4f}")

Common Mistakes That Invalidate Results

Correcting for Multiple Comparisons

from statsmodels.stats.multitest import multipletests

p_values = [0.012, 0.034, 0.041, 0.220, 0.003]

# Benjamini-Hochberg controls the false discovery rate
reject, p_adj, _, _ = multipletests(p_values, alpha=0.05, method="fdr_bh")
for raw, adj, r in zip(p_values, p_adj, reject):
    print(f"raw p={raw:.3f}  adj p={adj:.3f}  reject H0={r}")

How to Interpret a p-value Correctly

A p-value of 0.03 means: if H₀ were true, data this extreme would appear about 3% of the time. It does not mean:

• There is a 3% chance H₀ is true.
• There is a 97% chance the effect is real.
• The effect is large or important.

Bayesian methods answer the “probability the hypothesis is true” question directly, but they require a prior. Frequentist testing avoids priors and controls long-run error rates instead.

Frequently Asked Questions

What is the difference between the null hypothesis and the alternative hypothesis?

The null hypothesis (H₀) states that no effect or difference exists. The alternative hypothesis (H₁) states that an effect exists. Testing evaluates whether sample data provides enough evidence to reject H₀ in favor of H₁.

What does a p-value of 0.05 mean?

A p-value of 0.05 means there is a 5% probability of observing results at least as extreme as yours if the null hypothesis is true. It is the conventional cutoff for statistical significance, but it is not a measure of effect size or the probability that H₀ is true.

When should I use a t-test vs. a z-test?

Use a z-test when the population standard deviation is known or the sample is very large. Use a t-test when the population standard deviation is unknown and estimated from the sample, which covers most real-world cases.

Can I accept the null hypothesis?

No. You fail to reject the null hypothesis when evidence is insufficient. A non-significant result may reflect a small sample or low power rather than a true absence of effect. To argue for equivalence, use an equivalence test such as TOST.