Handling Imbalanced Datasets: SMOTE, Class Weights, and Better Metrics

A fraud model that predicts “legitimate” for every transaction scores 99.9% accuracy on a dataset with 0.1% fraud, and catches zero fraud. Imbalanced data breaks the default workflow: the loss function favors the majority class, accuracy hides the failure, and naive resampling leaks information. This guide covers three fixes in the order you should apply them: better metrics, class weights, and SMOTE with its variants.

Quick Takeaways

• Fix the metric first. Use PR-AUC, F1/F-beta, and Matthews Correlation Coefficient (MCC) instead of accuracy or ROC-AUC.
• Start with class weights. class_weight="balanced" costs nothing, adds no synthetic data, and often matches SMOTE.
• Apply SMOTE inside the cross-validation fold only. Oversampling before the split inflates scores through data leakage.
• Tune the decision threshold. The default 0.5 cutoff is rarely optimal for skewed classes.

Why Imbalanced Data Breaks Models

Gradient-based learners minimize average loss across samples. With a 99:1 split, the majority class contributes 99% of the gradient signal. The model learns that predicting the majority class is cheap and usually right.

Three failure modes follow:

1. Biased decision boundary. Minority samples sit near or beyond the boundary, so the model ignores them.
2. Misleading validation. Accuracy rewards the trivial classifier.
3. Unstable estimates. With 50 positives in a test set, one misclassified sample moves recall by 2 points.

Measure your imbalance ratio (IR) before choosing a strategy:

import pandas as pd

y = pd.Series(y)
counts = y.value_counts()
print(counts)
print(f"Imbalance ratio: {counts.max() / counts.min():.1f}:1")

Rough guidance: IR under 10:1 is mild, 10:1 to 100:1 is moderate, and above 100:1 is severe. Severe cases often need anomaly-detection approaches alongside the techniques below.

Step 1: Choose Metrics That Expose the Problem

Pick the metric before the technique. Otherwise you cannot tell whether the technique helped.

Metrics Comparison

Why ROC-AUC Misleads

ROC-AUC uses the false positive rate: FP / (FP + TN). When negatives number in the millions, even 10,000 false positives yield a tiny FPR, so the curve looks excellent. Precision (TP / (TP + FP)) exposes the same errors directly. That is why PR-AUC separates good and bad models on rare-event problems where ROC-AUC does not.

Formulas Worth Memorizing

• F1 = 2 × (Precision × Recall) / (Precision + Recall)
• F-beta = (1 + β²) × (P × R) / (β² × P + R). Use β = 2 when missing a positive costs more than a false alarm. Use β = 0.5 for the opposite.
• MCC = (TP×TN − FP×FN) / √((TP+FP)(TP+FN)(TN+FP)(TN+FN)). It ranges from −1 to +1.

Evaluation Code

from sklearn.metrics import (
    average_precision_score, roc_auc_score, f1_score,
    matthews_corrcoef, balanced_accuracy_score, classification_report
)

proba = model.predict_proba(X_test)[:, 1]
pred = (proba >= 0.5).astype(int)

print("PR-AUC :", average_precision_score(y_test, proba))
print("ROC-AUC:", roc_auc_score(y_test, proba))
print("F1     :", f1_score(y_test, pred))
print("MCC    :", matthews_corrcoef(y_test, pred))
print("BalAcc :", balanced_accuracy_score(y_test, pred))
print(classification_report(y_test, pred, digits=3))

Always use a stratified split so train and test keep the same class ratio:

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

Step 2: Class Weights (The Zero-Cost Baseline)

Class weights multiply each sample’s loss by a factor tied to its class. A misclassified minority sample then hurts more. No data changes, so there is no leakage risk and no synthetic noise.

The "balanced" heuristic in scikit-learn computes:

weight_c = n_samples / (n_classes × n_samples_in_class_c)

For 1,000 samples with 10 positives, the positive weight is 1000 / (2 × 10) = 50, and the negative weight is 1000 / (2 × 990) ≈ 0.505.

from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier

lr = LogisticRegression(class_weight="balanced", max_iter=1000)
rf = RandomForestClassifier(
    n_estimators=300, class_weight="balanced_subsample", n_jobs=-1, random_state=42
)

balanced_subsample recomputes weights on each bootstrap sample, which suits random forests.

Class Weights in Gradient Boosting and Deep Learning

# XGBoost: ratio of negatives to positives
import xgboost as xgb
spw = (y_train == 0).sum() / (y_train == 1).sum()
xgb_clf = xgb.XGBClassifier(scale_pos_weight=spw, eval_metric="aucpr")

# LightGBM
import lightgbm as lgb
lgb_clf = lgb.LGBMClassifier(is_unbalance=True)   # or scale_pos_weight=spw

# PyTorch: weighted binary cross-entropy
import torch, torch.nn as nn
pos_weight = torch.tensor([spw])
criterion = nn.BCEWithLogitsLoss(pos_weight=pos_weight)

Trade-off: Heavy weights (50× or more) can destabilize training and push probabilities out of calibration. If you need trustworthy probabilities, recalibrate afterward with CalibratedClassifierCV.

Step 3: SMOTE and Its Variants

SMOTE (Synthetic Minority Over-sampling Technique, Chawla et al., 2002) creates new minority samples by interpolation. For a minority point x, it finds its k nearest minority neighbors (default k = 5), picks one neighbor x_nn, and generates:

x_new = x + λ × (x_nn − x), where λ is drawn uniformly from [0, 1].

The new point lies on the line segment between two real minority samples. This fills in the minority region instead of duplicating points, as random oversampling does.

The Critical Rule: Resample Inside the Fold

Resampling before cross-validation leaks synthetic copies of validation-fold samples into training. Scores look great and production performance collapses. Use imblearn.pipeline.Pipeline, which applies samplers only during fit.

from imblearn.pipeline import Pipeline
from imblearn.over_sampling import SMOTE
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import StratifiedKFold, cross_validate

pipe = Pipeline([
    ("scale", StandardScaler()),
    ("smote", SMOTE(sampling_strategy=0.5, k_neighbors=5, random_state=42)),
    ("clf", RandomForestClassifier(n_estimators=300, random_state=42, n_jobs=-1)),
])

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_validate(
    pipe, X_train, y_train, cv=cv,
    scoring={"pr_auc": "average_precision", "f1": "f1", "mcc": "matthews_corrcoef"}  # mcc scorer is built into scikit-learn
)
for k, v in scores.items():
    if k.startswith("test_"):
        print(k, round(v.mean(), 4), "±", round(v.std(), 4))

Install the library with pip install imbalanced-learn. Note that sampling_strategy=0.5 sets the minority-to-majority ratio to 1:2 after resampling. You rarely need a perfect 1:1 balance.

SMOTE Variants

When SMOTE Fails

• High-dimensional sparse data (text TF-IDF, one-hot features). Euclidean neighbors lose meaning. Use class weights instead.
• Pure categorical data. Interpolating category codes produces nonsense. Use SMOTE-NC or skip it.
• Noisy minority labels. SMOTE amplifies noise by interpolating between mislabeled points.
• Tree ensembles. Random forests and boosting often do as well with class weights alone. Test both.
• Images and text with deep learning. Interpolating raw pixels or tokens does not yield valid samples. Use augmentation or weighted sampling.

Step 4: Tune the Decision Threshold

Most classifiers output probabilities. The 0.5 cutoff assumes balanced classes and symmetric costs. Neither holds here. Pick the threshold from a validation set using the precision-recall curve.

import numpy as np
from sklearn.metrics import precision_recall_curve

proba_val = model.predict_proba(X_val)[:, 1]
prec, rec, thr = precision_recall_curve(y_val, proba_val)

f1 = 2 * prec * rec / (prec + rec + 1e-12)
best = np.argmax(f1[:-1])          # last point has no threshold
print(f"Best threshold: {thr[best]:.3f} | F1: {f1[best]:.3f}")

final_pred = (proba_test >= thr[best]).astype(int)

For cost-sensitive work, replace F1 with expected cost: cost = FN × C_fn + FP × C_fp. Choose the threshold that minimizes it. If a missed fraud case costs $500 and a false alarm costs $5, the optimal threshold sits far below 0.5.

Real-World Workflows

Fraud Detection (IR ≈ 500:1)

1. Use PR-AUC as the primary metric and track recall at fixed precision (for example, recall at 90% precision).
2. Train LightGBM with scale_pos_weight.
3. Skip SMOTE at first. Add Borderline-SMOTE only if recall stalls.
4. Tune the threshold on a time-based validation split, not a random one. Fraud patterns drift.

Medical Screening (IR ≈ 20:1)

1. Optimize F2 or recall, since a missed case costs more than a false alarm.
2. Use LogisticRegression(class_weight="balanced") for interpretability.
3. Calibrate probabilities with isotonic regression before showing risk scores to clinicians.
4. Report confidence intervals. Small positive counts make point estimates unstable.

Churn Prediction (IR ≈ 8:1)

1. Mild imbalance often needs no resampling. Try the baseline with PR-AUC first.
2. Compare class weights, SMOTE, and threshold tuning in one pipeline experiment.
3. Keep whichever technique wins on stratified CV PR-AUC, not on accuracy.

Decision Playbook

Common Mistakes

• Oversampling before the train/test split. The test set then contains synthetic neighbors of training points.
• Resampling the test set. Evaluate on the real class distribution only.
• Reporting accuracy. It tells you nothing here.
• Ignoring variance. Run repeated stratified CV (RepeatedStratifiedKFold) when positives are few.
• Stacking every technique. Weights plus SMOTE plus a tuned threshold can double-correct the bias and hurt precision. Add one change at a time and measure each.

FAQ

What is the best way to handle an imbalanced dataset?

Start with the right metric (PR-AUC, F1, or MCC), then use class weights, then tune the decision threshold. Add SMOTE only if those steps leave recall too low. No single method wins everywhere, so compare them with stratified cross-validation.

Is SMOTE better than class weights?

Not by default. Class weights are faster, cause no leakage, and perform comparably on tree ensembles and linear models. SMOTE helps most on small, numeric, tabular datasets with moderate imbalance, and it works best paired with models that benefit from denser minority regions.

Which metric should I use for imbalanced classification?

Use PR-AUC (average precision) for ranking quality and F1, F2, or MCC for a chosen threshold. Avoid accuracy. ROC-AUC can look overly optimistic when negatives vastly outnumber positives.

Should I apply SMOTE before or after splitting the data?

After. Split first, then apply SMOTE only to the training portion. In cross-validation, put SMOTE inside an imblearn Pipeline so it runs on each training fold and never touches validation data.