Train/Test Splits Done Right: Best Practices and Common Mistakes

Your model scored 97% in validation and 71% in production. In most cases the model is fine and the split is the problem. A flawed train/test split lets information from the test set leak into training, so your metrics describe a model that doesn’t exist. This guide covers how to choose a splitting strategy, how to implement it in scikit-learn, and how to catch the mistakes that inflate scores.

Quick Takeaways

  • Match the split to the data’s structure. Use stratified splits for imbalanced classes, group splits for repeated entities, and time-based splits for temporal data.
  • Split first, preprocess second. Fit scalers, imputers, encoders and feature selectors on the training set only.
  • Use three sets, not two. Train (~60-80%), validation for tuning, and a test set you touch once.
  • Deduplicate before splitting. Near-duplicate rows across sets are the most common silent leak, and they hit LLM fine-tuning and benchmark evaluation especially hard.
Data Type Recommended Split scikit-learn Tool Typical Ratio
i.i.d. tabular, balanced Random train_test_split 80/20
Imbalanced classification Stratified train_test_split(stratify=y) 80/20
Multiple rows per user/patient Group GroupShuffleSplit, GroupKFold 80/20
Time series, forecasting Chronological TimeSeriesSplit Expanding window
Small data (<5k rows) K-fold CV StratifiedKFold 5 or 10 folds
LLM fine-tuning Deduplicated + held-out by source Custom (MinHash) 90/5/5

Why Splits Matter: Generalization Error

A model trained and scored on the same data measures memorization. The test set exists to estimate generalization error, which is how the model performs on data it has never seen and that matches the deployment distribution.

That estimate is only valid when two conditions hold:

  1. Independence. No test example shares information with a training example.
  2. Representativeness. The test distribution matches what the model will face in production.

Every mistake below breaks one of these.

The Three-Set Framework

Two sets are not enough once you tune anything. Each time you adjust a hyperparameter after viewing test scores, the test set becomes a second validation set. Your final number is then optimistic.

Set Purpose Touch Frequency Typical Share
Train Fit model weights Every epoch/iteration 60-80%
Validation Tune hyperparameters, early stopping, model selection Often 10-20%
Test Report final performance Once 10-20%

For datasets under roughly 10,000 rows, replace the fixed validation set with k-fold cross-validation on the training portion. Keep the test set held out.

For datasets over 1 million rows, a 98/1/1 split is often fine. One percent of a million is still 10,000 examples, enough for a tight confidence interval.

Choosing the Right Splitting Strategy

Random Split

Use it only when rows are independent and identically distributed (i.i.d.).

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y,
    test_size=0.2,
    random_state=42,   # reproducibility
    shuffle=True
)

Always set random_state. Without it, your results can’t be reproduced and comparisons between runs are noise.

Stratified Split

With a 2% positive class (fraud, rare disease, churn), a random 20% test set can end up with far fewer positives than expected. Stratification preserves class proportions in every set.

X_train, X_test, y_train, y_test = train_test_split(
    X, y,
    test_size=0.2,
    stratify=y,        # preserve class ratio
    random_state=42
)

For cross-validation, use StratifiedKFold(n_splits=5, shuffle=True, random_state=42). For regression, bin the target into quantiles and stratify on the bins.

Group Split

If one patient, user, device or document produces multiple rows, a random split puts some of that entity’s rows in train and others in test. The model memorizes the entity and looks accurate on its “unseen” rows.

from sklearn.model_selection import GroupShuffleSplit

gss = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
train_idx, test_idx = next(gss.split(X, y, groups=df["patient_id"]))

X_train, X_test = X.iloc[train_idx], X.iloc[test_idx]
y_train, y_test = y.iloc[train_idx], y.iloc[test_idx]

# Sanity check: no overlap
assert set(df.iloc[train_idx]["patient_id"]).isdisjoint(
    set(df.iloc[test_idx]["patient_id"])
)

For cross-validation, use GroupKFold. When you need both stratification and grouping, use StratifiedGroupKFold.

Time-Based Split

For forecasting, never shuffle. Training on the future to predict the past is leakage by definition.

from sklearn.model_selection import TimeSeriesSplit

tscv = TimeSeriesSplit(n_splits=5, gap=7)  # 7-step gap prevents lag leakage

for fold, (train_idx, val_idx) in enumerate(tscv.split(X)):
    print(f"Fold {fold}: train up to {train_idx[-1]}, validate {val_idx[0]}-{val_idx[-1]}")

The gap parameter drops samples between train and validation. If your features include a 7-day rolling mean, a gap of at least 7 stops window features from overlapping the boundary.

Common Mistakes (and How to Fix Them)

# Mistake Symptom Fix
1 Preprocessing before splitting Test score unrealistically high Wrap steps in a Pipeline
2 Duplicate/near-duplicate rows across sets Score drops sharply after dedup Deduplicate before splitting
3 Shuffling time series Great backtest, poor live results Chronological split with gap
4 Ignoring groups Model memorizes entities GroupKFold
5 Tuning on the test set Test score optimistic Separate validation set
6 Unstratified rare classes Unstable, high-variance metrics stratify=y
7 Target leakage in features Suspiciously perfect accuracy Audit features for post-outcome data
8 Test set too small Wide confidence intervals Compute required sample size

Mistake 1: Preprocessing Before Splitting

This is the most common leak. Fitting a StandardScaler on the full dataset lets the test set’s mean and variance shape the training features. The effect is subtle with scaling but severe with target encoding, imputation by group mean, PCA, and feature selection.

Wrong:

from sklearn.preprocessing import StandardScaler

X_scaled = StandardScaler().fit_transform(X)       # sees test data
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y)

Right:

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

pipe = Pipeline([
    ("scale", StandardScaler()),
    ("clf", LogisticRegression(max_iter=1000))
])

# The scaler is re-fit inside each training fold only
scores = cross_val_score(pipe, X_train, y_train, cv=5, scoring="roc_auc")
print(f"ROC-AUC: {scores.mean():.3f} ± {scores.std():.3f}")

A Pipeline guarantees each transformer is fit on training folds only. Use it every time.

Mistake 2: Duplicates and Near-Duplicates

Web-scraped data, customer records and product catalogs contain repeats. If the same row lands in both train and test, you’re measuring recall, not generalization.

# Exact duplicates
df = df.drop_duplicates(subset=feature_cols)

# Near-duplicates in text: normalize, then hash
import hashlib, re

def fingerprint(text: str) -> str:
    norm = re.sub(r"\W+", " ", text.lower()).strip()
    return hashlib.md5(norm.encode()).hexdigest()

df["fp"] = df["text"].map(fingerprint)
df = df.drop_duplicates(subset="fp")

For fuzzy matches, use MinHash LSH (the datasketch library) with a Jaccard threshold around 0.8. For embeddings, flag pairs above 0.95 cosine similarity.

Mistake 3: Target Leakage

A feature that wouldn’t exist at prediction time makes the model look brilliant. Examples include a discharge_status column in a hospital readmission model, or days_since_last_payment in a default predictor when the label is computed from payment behavior.

Audit with two checks:

  • Train a single-feature model on each column. Any feature that alone gives near-perfect AUC deserves scrutiny.
  • Ask for each feature: “Would I have this value at the moment of prediction?”

Mistake 4: Test Set Too Small

A 1,000-row test set with 90% accuracy has a 95% confidence interval of roughly ±1.9 points. Compare two models that differ by 1 point and you’re comparing noise. Estimate it with the normal approximation:

import math

def accuracy_ci(acc: float, n: int, z: float = 1.96) -> tuple[float, float]:
    margin = z * math.sqrt(acc * (1 - acc) / n)
    return acc - margin, acc + margin

print(accuracy_ci(0.90, 1000))   # (0.881, 0.919)

Splitting Strategies for LLM Work

Splits for large language models carry extra risks because training corpora are huge and opaque.

Scenario Risk Mitigation
Fine-tuning on instruction data Same prompt template or near-identical Q&A across train/eval MinHash dedup; split by source or topic
Benchmark evaluation Benchmark contamination (test items in pretraining data) Use private or freshly written held-out sets
RAG evaluation Eval questions answerable from the exact chunk used for tuning Split by document, not by chunk
Synthetic data generation Generated examples paraphrase eval items Decontaminate with 13-gram overlap checks

For RAG pipelines, always split at the document level. If chunks from one PDF appear in both your retrieval-tuning set and your evaluation set, retrieval metrics inflate.

A minimal n-gram decontamination check:

def ngrams(text: str, n: int = 13) -> set:
    tokens = text.lower().split()
    return {" ".join(tokens[i:i+n]) for i in range(len(tokens) - n + 1)}

train_grams = set().union(*(ngrams(t) for t in train_texts))
clean_eval = [t for t in eval_texts if ngrams(t).isdisjoint(train_grams)]
print(f"Kept {len(clean_eval)}/{len(eval_texts)} eval items")

Practical Workflow: A Leak-Proof Pipeline

Follow this sequence on any new supervised learning project.

  1. Inspect structure. Identify entities (users, patients), timestamps and class balance.
  2. Deduplicate. Remove exact and near-duplicate rows.
  3. Pick the split type using the table at the top of this guide.
  4. Carve out the test set first. Save it and don’t look at it.
  5. Build a Pipeline. Put every fitted transformation inside it.
  6. Cross-validate on training data to tune hyperparameters.
  7. Run one final evaluation on the test set and report it with a confidence interval.
  8. Re-check with a time-forward holdout if the model will run in production over months.

End-to-end example combining stratification, a pipeline and a final test evaluation:

import pandas as pd
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import roc_auc_score

df = pd.read_csv("data.csv").drop_duplicates()
X, y = df.drop(columns="label"), df["label"]

# 1. Hold out the test set first
X_dev, X_test, y_dev, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

# 2. Tune with CV on the dev set only
pipe = Pipeline([
    ("scale", StandardScaler()),
    ("clf", GradientBoostingClassifier(random_state=42))
])
grid = GridSearchCV(
    pipe,
    {"clf__n_estimators": [100, 300], "clf__max_depth": [2, 3]},
    cv=5, scoring="roc_auc", n_jobs=-1
)
grid.fit(X_dev, y_dev)

# 3. Single, final test evaluation
test_auc = roc_auc_score(y_test, grid.predict_proba(X_test)[:, 1])
print(f"CV AUC: {grid.best_score_:.3f} | Test AUC: {test_auc:.3f}")

A large gap between CV AUC and test AUC (more than a few points) signals either leakage in your CV setup or a distribution shift between the sets.

Real-World Scenarios

Healthcare, readmission prediction. A hospital has 80,000 admissions from 30,000 patients. A random row split puts the same patient in both sets, and the model learns patient-specific patterns. Switching to GroupKFold on patient_id typically drops reported AUC noticeably, and the lower number is the honest one.

E-commerce, demand forecasting. Shuffled K-fold gives low error because neighboring days leak seasonality and promotions. A rolling-origin evaluation with TimeSeriesSplit reveals the true forecast error at the planning horizon.

Enterprise RAG for support tickets. Tickets contain many near-identical templated replies. Without deduplication and document-level splitting, retrieval recall in testing looks near-perfect. After MinHash dedup and a split by customer account, the metric reflects actual retrieval quality.

FAQ

What is the best train/test split ratio?

80/20 is the common default, but the right ratio depends on data size. Use 70/30 or cross-validation for small datasets, and 98/1/1 for datasets above a million rows. The test set must be large enough to give a confidence interval narrow enough to separate your candidate models.

What is data leakage in machine learning?

Data leakage is when information from outside the training set influences the model during training. Common causes include preprocessing before splitting, duplicate rows across sets, features that encode the target, and shuffling time-ordered data. Leakage inflates validation metrics and causes poor production performance.

Should I split data before or after preprocessing?

Split first. Fit scalers, imputers, encoders and feature selectors on the training data only, then apply them to validation and test data. Use a scikit-learn Pipeline so this happens automatically inside each cross-validation fold.

When should I use stratified splitting?

Use it whenever classes are imbalanced, such as fraud at 1-5% prevalence. stratify=y keeps the class ratio the same in train and test sets, which stabilizes metrics like precision, recall and F1.

Hot this week

Best AI Music and Voice Generators in 2026

Compare Suno v6, ElevenLabs Music, Udio, Eleven v4, Cartesia and Fish Audio on price, rights and latency. Pick your tool and start creating today.

AI Engineer vs ML Engineer vs Data Scientist: What’s the Difference?

AI engineer vs ML engineer vs data scientist: compare skills, tools, and roles side by side. Find your best-fit career path and start learning today.

Chain-of-Thought, Few-Shot, and Role Prompts: When Each Technique Helps

Chain-of-thought, few-shot, and role prompts each fix different failures. Learn when each works, when it backfires, and copy the templates.

Outliers: How to Detect and Handle Them Without Losing Signal

Learn outlier detection with Z-score, IQR, Isolation Forest, and LOF. Copy the Python code and handle outliers without losing real signal.

Hotend and Extruder Basics: How Filament Becomes Plastic Layers

Learn how a hotend and extruder turn filament into layers. Compare direct vs Bowden, nozzles, and temps, then tune yours. Read the full guide.

Topics

Best AI Music and Voice Generators in 2026

Compare Suno v6, ElevenLabs Music, Udio, Eleven v4, Cartesia and Fish Audio on price, rights and latency. Pick your tool and start creating today.

AI Engineer vs ML Engineer vs Data Scientist: What’s the Difference?

AI engineer vs ML engineer vs data scientist: compare skills, tools, and roles side by side. Find your best-fit career path and start learning today.

Chain-of-Thought, Few-Shot, and Role Prompts: When Each Technique Helps

Chain-of-thought, few-shot, and role prompts each fix different failures. Learn when each works, when it backfires, and copy the templates.

Outliers: How to Detect and Handle Them Without Losing Signal

Learn outlier detection with Z-score, IQR, Isolation Forest, and LOF. Copy the Python code and handle outliers without losing real signal.

Hotend and Extruder Basics: How Filament Becomes Plastic Layers

Learn how a hotend and extruder turn filament into layers. Compare direct vs Bowden, nozzles, and temps, then tune yours. Read the full guide.

Multiboard vs Pegboard vs Gridfinity: Which Wall Storage System Should You Print?

Multiboard, pegboard or Gridfinity? Compare load, filament, print time and cost, then pick the right wall storage system to print. Read the guide.

3D Printer Fire Safety: Preventing Thermal Failures and What to Do If Something Goes Wrong

Learn how to stop thermal runaway, test your firmware, pick the right extinguisher, and respond fast. Get the full 3D printer fire safety checklist.

Bambu Studio Tips and Tricks Every Owner Should Know

Master Bambu Studio with 20+ proven tips on walls, supports, color, G-code and new 2.9 features. Read the guide and print cleaner, faster.

Related Articles

Popular Categories