A single bad value can shift a mean, tilt a regression line, and wreck a feature scaler. A single rare but real value can be the fraud case, the failing sensor, or the whale customer your model exists to find. Outlier handling fails when you treat these two cases the same. Detect first, diagnose second, act third. Delete last.
Quick Takeaways
- Flag, don’t delete. Add an
is_outliercolumn and a reason code before you change any data. - Use robust statistics. Median and MAD (median absolute deviation) resist the contamination that breaks mean and standard deviation.
- Match the method to the data. Use IQR or modified Z-score for one feature, Isolation Forest or LOF for many features, and a domain rule for anything with a physical limit.
- Pick the treatment by cause. Errors get fixed or removed. Real extremes get capped, transformed, or modeled with a robust loss.
| Method | Best for | Handles multivariate? | Key parameter | Weakness |
|---|---|---|---|---|
| Z-score | Roughly normal, single feature | No | threshold 3.0 | Mean and SD are themselves distorted by outliers |
| Modified Z-score | Skewed single feature | No | threshold 3.5 | Fails when MAD = 0 |
| IQR fences | Quick EDA, any distribution | No | k = 1.5 (3.0 for extreme) | Ignores feature interactions |
| Isolation Forest | Tabular, high-dimensional | Yes | n_estimators=200, max_samples=256 |
Score threshold is hard to interpret |
| LOF | Clusters of varying density | Yes | n_neighbors=20 |
Slow on large data, sensitive to scaling |
| Mahalanobis (robust) | Correlated numeric features | Yes | chi-square cutoff | Assumes elliptical shape |
What Counts as an Outlier?
An outlier is an observation that sits far from the bulk of the data under some model of “normal.” The model matters. A $4M house is an outlier in a neighborhood of $400K homes. It is not an outlier in a dataset of luxury estates.
Three types need different handling:
- Point outliers. One value is extreme on its own, such as an age of 214.
- Contextual outliers. The value is normal globally but odd in context. A 35°C reading is fine in July and suspicious in January.
- Collective outliers. Each point looks fine, but the group is abnormal. A flat line of identical sensor readings is one example.
Next, identify the cause. Outliers come from data errors (typos, unit mix-ups, sensor faults), sampling mismatches (a wholesale buyer inside retail data), or genuine rare events. The first two justify correction or removal. The third is signal.
Univariate Detection: Fast and Interpretable
Start here. These checks run in milliseconds and give you a baseline.
Z-Score and Its Limits
The Z-score is z = (x − μ) / σ. Values with |z| > 3 are flagged. On a normal distribution, that covers about 0.27% of points.
The flaw is circular. A large outlier inflates both μ and σ, which shrinks its own Z-score. This is called masking. In small samples, the maximum possible Z-score is bounded by (n − 1) / √n, so with n = 10 no point can ever exceed 2.85.
Modified Z-Score (MAD-Based)
Replace mean with median and SD with MAD. The constant 0.6745 makes MAD comparable to σ under normality. The standard cutoff is 3.5 (Iglewicz and Hoaglin).
IQR Fences
Compute Q1 and Q3. Flag anything below Q1 − 1.5·IQR or above Q3 + 1.5·IQR. Use k = 3.0 to catch only extreme points. IQR needs no normality assumption, which makes it the default for EDA.
import numpy as np
import pandas as pd
def iqr_flags(s: pd.Series, k: float = 1.5) -> pd.Series:
q1, q3 = s.quantile([0.25, 0.75])
iqr = q3 - q1
return (s < q1 - k * iqr) | (s > q3 + k * iqr)
def modified_z_flags(s: pd.Series, thresh: float = 3.5) -> pd.Series:
med = s.median()
mad = (s - med).abs().median()
if mad == 0: # more than 50% identical values
return pd.Series(False, index=s.index)
mz = 0.6745 * (s - med) / mad
return mz.abs() > thresh
df["iqr_out"] = iqr_flags(df["price"])
df["mad_out"] = modified_z_flags(df["price"])
Skewed data warning: Right-skewed variables like income, latency, and order value produce false positives on the upper tail. Apply a log transform (
np.log1p) before running these tests.
Multivariate Detection: When the Combination Is the Anomaly
A 6’5″ person is normal. A 22-pound person is possible. A 6’5″ person weighing 22 pounds is a data error. Univariate checks miss this. Use model-based detectors.
Isolation Forest
Isolation Forest builds random trees that split features at random thresholds. Anomalies sit in sparse regions and isolate in fewer splits. The anomaly score is based on average path length across trees. It scales roughly linearly with sample size and needs no distance metric.
Local Outlier Factor (LOF)
LOF compares each point’s local density to the density of its k nearest neighbors. A score near 1 means similar density. A score well above 1 means the point is isolated relative to its neighbors. LOF handles datasets where clusters have different densities, which global methods struggle with. Scale your features first, because LOF relies on distances.
from sklearn.ensemble import IsolationForest
from sklearn.neighbors import LocalOutlierFactor
from sklearn.preprocessing import RobustScaler
X = RobustScaler().fit_transform(df[["height_cm", "weight_kg", "age"]])
iso = IsolationForest(
n_estimators=200,
max_samples=256,
contamination="auto", # or a float like 0.01 if you know the rate
random_state=42,
)
df["iso_label"] = iso.fit_predict(X) # -1 = outlier, 1 = inlier
df["iso_score"] = iso.score_samples(X) # lower = more anomalous
lof = LocalOutlierFactor(n_neighbors=20, contamination="auto")
df["lof_label"] = lof.fit_predict(X)
df["lof_score"] = lof.negative_outlier_factor_ # more negative = more anomalous
RobustScaler centers on the median and scales by IQR, so outliers cannot distort the scaling that feeds your detector.
Detector Agreement
Run two or three detectors and look at the overlap. A point flagged by IQR, Isolation Forest, and LOF deserves a manual look. A point flagged by only one is usually a boundary case.
flag_cols = ["iqr_out", "iso_label", "lof_label"]
df["votes"] = (
df["iqr_out"].astype(int)
+ (df["iso_label"] == -1).astype(int)
+ (df["lof_label"] == -1).astype(int)
)
review_queue = df[df["votes"] >= 2].sort_values("iso_score")
The Decision Framework: Keep, Fix, Cap, or Drop
Run every flagged row through these questions in order.
- Is it physically or logically impossible? Negative age, a timestamp in 2087, a percentage of 340. Fix the source or drop the row.
- Is it a unit or entry error? A price of 1,299,000 next to prices around 1,299 suggests a misplaced decimal. Correct it if the cause is provable.
- Is it a legitimate rare event? Fraud, equipment failure, viral traffic. Keep it. It is the signal.
- Does your model break on it? If yes, change the model or the representation, not the data.
| Treatment | When to use | Risk |
|---|---|---|
| Remove | Proven error, out-of-scope population | Shrinks sample, biases results if cause is non-random |
| Impute | Error in a feature, row is otherwise valid | Understates variance |
| Winsorize (cap at percentiles) | Real extremes that distort linear models | Flattens true tail behavior |
| Transform (log, Box-Cox, Yeo-Johnson) | Skewed features | Harder interpretation |
| Robust model or loss | Extremes are real and must stay | Slightly lower efficiency on clean data |
| Separate model | Outliers are a distinct regime | More system complexity |
from scipy.stats.mstats import winsorize
from sklearn.linear_model import HuberRegressor, LinearRegression
from sklearn.preprocessing import PowerTransformer
# 1. Cap the bottom and top 1%
df["income_w"] = winsorize(df["income"].to_numpy(), limits=[0.01, 0.01])
# 2. Reduce skew without discarding rows
pt = PowerTransformer(method="yeo-johnson")
df["income_pt"] = pt.fit_transform(df[["income"]])
# 3. Keep the data, change the loss function
ols = LinearRegression().fit(X_train, y_train)
huber = HuberRegressor(epsilon=1.35).fit(X_train, y_train) # 1.35 ≈ 95% efficiency on clean normal data
Huber loss treats small residuals quadratically and large ones linearly. The epsilon=1.35 default gives about 95% statistical efficiency on clean Gaussian data while limiting the pull of extreme points.
Preventing Data Leakage
Outlier logic is a preprocessing step. Treat it like a scaler.
- Fit thresholds on training data only. Compute IQR fences, winsorization percentiles, and Isolation Forest parameters on the train split.
- Apply the same thresholds to validation and test data. Never recompute them.
- Do not drop test rows. In production you will get outliers too. Evaluate how your model behaves on them.
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import RobustScaler
from sklearn.linear_model import HuberRegressor
pipe = Pipeline([
("scale", RobustScaler()),
("model", HuberRegressor(epsilon=1.35, max_iter=500)),
])
pipe.fit(X_train, y_train)
print(pipe.score(X_test, y_test))
Real-World Workflows
Fraud Detection: Outliers Are the Product
A payments team scores transactions with Isolation Forest on amount, merchant category, hour of day, and distance from home. They do not remove the flagged points. They route them to a review queue and feed confirmed cases back as labels. Early detection is unsupervised. Once labels accumulate, the team switches to a supervised classifier and keeps the isolation score as a feature.
Sensor and IoT Data: Contextual Outliers
A temperature sensor reports 41°C at 3 a.m. in a cold warehouse. A global threshold passes it. A rolling median with a rolling MAD flags it.
roll_med = s.rolling(window=60, center=True).median()
roll_mad = (s - roll_med).abs().rolling(window=60, center=True).median()
flag = (s - roll_med).abs() > 3.5 * roll_mad / 0.6745
Fix confirmed sensor faults by interpolating over the gap. Keep a was_imputed column so downstream models can discount those values.
Regression on Housing Prices: Cap, Don’t Cut
Luxury listings pull an OLS fit upward. Dropping them makes the model useless for the high-end segment. Log-transform the target, winsorize the top 0.5% of square footage, and compare MAE on a held-out luxury slice before and after. Report both numbers.
Audit Template
Log every decision in a table that ships with the dataset.
| Row ID | Feature | Value | Detector(s) | Cause | Action | Reviewer |
|---|---|---|---|---|---|---|
| 88214 | price | 1,299,000 | IQR, ISO | Decimal error | Corrected to 1,299.00 | A.K. |
| 90117 | amount | 48,200 | ISO, LOF | Verified wholesale order | Kept | A.K. |
Common Mistakes
- Removing outliers before splitting the data. This leaks test information into training.
- Using Z-score on skewed data. It over-flags the long tail.
- Treating
contaminationas known. Setting it to 0.05 forces exactly 5% of points to be flagged, regardless of truth. Use"auto"and inspect score distributions. - Skipping the cause check. Automated removal turns rare events into missing evidence.
- Ignoring the downstream metric. The right outlier policy is the one that improves performance on the cases you care about, measured on held-out data.
FAQ
How do you detect outliers in a dataset?
Start with IQR fences or a modified Z-score on each feature. Add a multivariate detector such as Isolation Forest or LOF for feature interactions. Review points flagged by two or more methods, and confirm the cause with domain knowledge before you act.
Should you always remove outliers?
No. Remove outliers only when they are proven errors or out-of-scope records. Real extreme values carry information. Cap them, transform them, or use a robust model such as Huber regression instead of deleting them.
What is the difference between the Z-score and IQR methods?
The Z-score uses mean and standard deviation and assumes a roughly normal distribution. The IQR method uses quartiles and makes no distribution assumption, so it resists distortion from the outliers themselves. Prefer IQR or the MAD-based modified Z-score on skewed data.
How does Isolation Forest detect outliers?
Isolation Forest builds random decision trees that split on random features and thresholds. Anomalies lie in sparse regions, so they isolate in fewer splits. Points with short average path length across the trees get the lowest scores and are labeled outliers.



