A p-value answers one narrow question: if the null hypothesis were true, how often would you see data at least this extreme? It does not tell you whether your hypothesis is true, whether your effect matters, or whether your result will replicate. Misreading it has distorted published research for decades, and the same errors show up in A/B tests, model comparisons, and feature selection pipelines every day.
Quick Takeaways
- A p-value is P(data this extreme or more | H₀ true). It is not P(H₀ true | data).
- A small p-value signals incompatibility with the null model. It says nothing about effect size or practical importance.
- Running many tests without correction inflates false positives. Twenty independent tests at α = 0.05 give a 64% chance of at least one false hit.
- Report the effect size, the confidence interval, and the exact p-value together.
| Concept | What it measures | Common misreading |
|---|---|---|
| p-value | Data compatibility with H₀ | “Probability H₀ is true” |
| α (alpha) | Pre-set false positive rate | “Probability my result is wrong” |
| Effect size | Magnitude of difference | Ignored when p < 0.05 |
| Confidence interval | Plausible parameter range | “95% chance the true value is inside” |
| Power (1 − β) | Chance of detecting a real effect | Rarely computed before the study |
What a P-Value Actually Is
Every hypothesis test follows the same logic.
- State a null hypothesis (H₀), usually “no effect” or “no difference.”
- Choose a test statistic (t, z, χ², F) that summarizes the data.
- Compute the sampling distribution of that statistic assuming H₀ is true.
- Calculate the probability of observing a statistic at least as extreme as yours.
That probability is the p-value. Formally, for a two-sided test:
p = P(|T| ≥ |t_obs| | H₀)
Consider a coin you suspect is biased. You flip it 100 times and get 62 heads. Under H₀ (a fair coin), the chance of 62 or more heads, or 38 or fewer, is about 0.021. The data look unusual if the coin is fair. That is all the p-value tells you.
The Role of Alpha
The significance level α is a threshold you set before collecting data. If p ≤ α, you reject H₀. The conventional α = 0.05 is a convention, not a law of nature. Fields like particle physics use 5σ (p ≈ 3×10⁻⁷). Clinical trials with safety implications often use stricter thresholds.
Why 0.05 Is Not a Cliff
A result with p = 0.049 is not meaningfully different from one with p = 0.051. Treating the threshold as a bright line produces the “significant vs. not significant” fallacy: two studies with nearly identical evidence get opposite conclusions.
Seeing a P-Value in Python
Run this simulation to build intuition. It draws two samples, runs a Welch’s t-test, and computes the effect size alongside the p-value.
import numpy as np
from scipy import stats
rng = np.random.default_rng(seed=42) # reproducible randomness
# Two groups: true mean difference of 0.4 standard deviations
control = rng.normal(loc=100, scale=15, size=80)
treatment = rng.normal(loc=106, scale=15, size=80)
# Welch's t-test: does not assume equal variances
t_stat, p_value = stats.ttest_ind(treatment, control, equal_var=False)
# Cohen's d: standardized effect size
pooled_sd = np.sqrt((control.var(ddof=1) + treatment.var(ddof=1)) / 2)
cohens_d = (treatment.mean() - control.mean()) / pooled_sd
# 95% confidence interval for the mean difference
diff = treatment.mean() - control.mean()
se = np.sqrt(control.var(ddof=1)/len(control) + treatment.var(ddof=1)/len(treatment))
ci_low, ci_high = diff - 1.96 * se, diff + 1.96 * se
print(f"t = {t_stat:.3f}, p = {p_value:.4f}")
print(f"Cohen's d = {cohens_d:.3f}")
print(f"Mean difference = {diff:.2f}, 95% CI [{ci_low:.2f}, {ci_high:.2f}]")
Always report all three outputs. The p-value alone hides whether the effect is tiny or large.
Simulating the Null Distribution (Permutation Test)
A permutation test makes the definition concrete. You shuffle group labels thousands of times to build the distribution of differences under H₀, then see where your observed difference falls.
import numpy as np
rng = np.random.default_rng(seed=7)
observed_diff = treatment.mean() - control.mean()
pooled = np.concatenate([control, treatment])
n_treat = len(treatment)
n_permutations = 10_000
null_diffs = np.empty(n_permutations)
for i in range(n_permutations):
shuffled = rng.permutation(pooled) # break any real group structure
null_diffs[i] = shuffled[:n_treat].mean() - shuffled[n_treat:].mean()
# Two-sided p-value: share of null differences at least as extreme as observed
p_perm = np.mean(np.abs(null_diffs) >= np.abs(observed_diff))
print(f"Permutation p-value: {p_perm:.4f}")
The permutation p-value is literally the fraction of shuffled datasets that look as extreme as yours. No formula, no distributional assumptions.
The 7 Most Common P-Value Mistakes
1. Treating p as the Probability That H₀ Is True
A p-value of 0.03 does not mean there is a 3% chance the null is true. It means that if the null is true, data this extreme occur 3% of the time. Getting P(H₀ | data) requires Bayes’ theorem and a prior.
2. Equating Significance with Importance
With enough data, trivial effects become significant. A study of 2 million users can return p < 0.001 for a 0.02% conversion lift that is worthless in business terms. Always pair the p-value with Cohen’s d, odds ratio, or the raw difference.
3. Reading Non-Significance as “No Effect”
p = 0.20 means the data are not incompatible with H₀. It does not prove H₀. Underpowered tests fail to detect real effects constantly. Check the confidence interval: a wide interval spanning both zero and large effects means the test was inconclusive.
4. P-Hacking and Multiple Comparisons
Test enough features, subgroups, or metrics and something crosses 0.05 by chance. The family-wise error rate grows quickly:
FWER = 1 − (1 − α)^m
where m is the number of tests. Use correction methods.
from statsmodels.stats.multitest import multipletests
p_values = [0.001, 0.012, 0.034, 0.048, 0.20, 0.51]
# Holm is uniformly more powerful than Bonferroni; fdr_bh controls false discovery rate
reject, p_adj, _, _ = multipletests(p_values, alpha=0.05, method="holm")
for raw, adj, r in zip(p_values, p_adj, reject):
print(f"raw p = {raw:.3f} | adjusted p = {adj:.3f} | reject H0: {r}")
5. Optional Stopping in A/B Tests
Checking results daily and stopping when p < 0.05 inflates the false positive rate well above 5%. Fix the sample size in advance using a power calculation, or use sequential testing methods designed for continuous monitoring.
6. Ignoring Test Assumptions
A t-test assumes roughly normal residuals and independent observations. Violations, especially non-independence from repeated measures or clustered data, make p-values unreliable. Use Mann-Whitney U, bootstrap methods, or mixed-effects models when assumptions fail.
7. Misinterpreting the Confidence Interval Alongside It
A 95% CI does not mean “95% probability the true value lies inside.” It means that if you repeated the procedure many times, 95% of the intervals would contain the true parameter. Any single interval either contains it or does not.
Choosing the Right Approach: A Comparison
| Method | Answers | Strength | Weakness |
|---|---|---|---|
| Frequentist p-value | Is the data compatible with H₀? | Simple, standardized | Misread constantly |
| Confidence interval | What effect sizes are plausible? | Shows magnitude and precision | Counterintuitive interpretation |
| Bayes factor | How much does data favor H₁ over H₀? | Quantifies evidence for both sides | Sensitive to prior choice |
| Bayesian posterior | What is P(effect | data)? | Direct probability statements | Requires a prior; heavier compute |
| Permutation test | Is the observed gap unusual under shuffling? | Few assumptions | Slower on large data |
| Correction method | Controls | Conservativeness | Best for |
|---|---|---|---|
| Bonferroni | FWER | Very high | Few tests, strict control |
| Holm | FWER | High (less than Bonferroni) | General-purpose FWER control |
| Benjamini-Hochberg | FDR | Moderate | Many tests, e.g., genomics |
Real-World Use Cases
Case 1: A/B Testing a Checkout Redesign
An e-commerce team tests a new checkout flow. Control converts at 4.10%, treatment at 4.35%, with 120,000 users per arm. A two-proportion z-test gives a small p-value.
from statsmodels.stats.proportion import proportions_ztest, proportion_confint
conversions = np.array([5220, 4920]) # treatment, control
visitors = np.array([120000, 120000])
z_stat, p_val = proportions_ztest(conversions, visitors)
rate_t, rate_c = conversions / visitors
print(f"Treatment: {rate_t:.3%} | Control: {rate_c:.3%}")
print(f"z = {z_stat:.2f}, p = {p_val:.5f}")
print(f"Absolute lift: {rate_t - rate_c:.3%}")
The decision should rest on the absolute lift and its interval, weighed against implementation cost, not on the p-value alone.
Case 2: Feature Selection in Machine Learning
Univariate feature screening with SelectKBest and an F-test ranks features by p-value. With 5,000 candidate features, roughly 250 pass at α = 0.05 even if all are noise. Apply Benjamini-Hochberg correction, and validate on a held-out set.
Case 3: Detecting Drift in Sensor Streams
A Kolmogorov-Smirnov test comparing a recent window of sensor readings to a baseline flags distribution drift. In streaming settings you run the test repeatedly, so adjust thresholds for multiplicity or use control-chart methods instead of raw p-values.
How to Report Results Correctly
Use this checklist for every test you publish or present.
- State the hypotheses and the test used with its assumptions checked.
- Report the exact p-value (p = 0.032), not just “p < 0.05.”
- Include the effect size and confidence interval.
- Document how many tests you ran and which multiplicity correction applied.
- Report the sample size and power analysis done before collection.
- Share the code and data when possible.
A well-formed sentence looks like this: “Treatment increased conversion by 0.25 percentage points (95% CI [0.12, 0.38], Welch z-test p = 0.0004, Cohen’s h = 0.012).”
Frequently Asked Questions
What does a p-value of 0.05 mean?
It means that if the null hypothesis were true, you would see results at least this extreme in about 5% of repeated experiments. It is not a 5% probability that your result is wrong or that the null is true.
Is a lower p-value always better?
No. A lower p-value indicates stronger incompatibility with H₀, but it reflects both effect size and sample size. A tiny, irrelevant effect can produce a tiny p-value in a large dataset. Judge results by effect size and confidence intervals as well.
What is the difference between a p-value and a significance level?
The significance level (α) is a threshold chosen before the study. The p-value is computed from the data after the study. You reject H₀ when the p-value is at or below α.
Can a p-value be exactly zero?
No. Software that prints p = 0.000 is rounding a very small value. Report it as p < 0.001 instead.
