Your model scored 97% in validation and 71% in production. In most cases the model is fine and the split is the problem. A flawed train/test split lets information from the test set leak into training, so your metrics describe a model that doesn’t exist. This guide covers how to choose a splitting strategy, how to implement it in scikit-learn, and how to catch the mistakes that inflate scores.
Quick Takeaways
- Match the split to the data’s structure. Use stratified splits for imbalanced classes, group splits for repeated entities, and time-based splits for temporal data.
- Split first, preprocess second. Fit scalers, imputers, encoders and feature selectors on the training set only.
- Use three sets, not two. Train (~60-80%), validation for tuning, and a test set you touch once.
- Deduplicate before splitting. Near-duplicate rows across sets are the most common silent leak, and they hit LLM fine-tuning and benchmark evaluation especially hard.
| Data Type | Recommended Split | scikit-learn Tool | Typical Ratio |
|---|---|---|---|
| i.i.d. tabular, balanced | Random | train_test_split |
80/20 |
| Imbalanced classification | Stratified | train_test_split(stratify=y) |
80/20 |
| Multiple rows per user/patient | Group | GroupShuffleSplit, GroupKFold |
80/20 |
| Time series, forecasting | Chronological | TimeSeriesSplit |
Expanding window |
| Small data (<5k rows) | K-fold CV | StratifiedKFold |
5 or 10 folds |
| LLM fine-tuning | Deduplicated + held-out by source | Custom (MinHash) | 90/5/5 |
Why Splits Matter: Generalization Error
A model trained and scored on the same data measures memorization. The test set exists to estimate generalization error, which is how the model performs on data it has never seen and that matches the deployment distribution.
That estimate is only valid when two conditions hold:
- Independence. No test example shares information with a training example.
- Representativeness. The test distribution matches what the model will face in production.
Every mistake below breaks one of these.
The Three-Set Framework
Two sets are not enough once you tune anything. Each time you adjust a hyperparameter after viewing test scores, the test set becomes a second validation set. Your final number is then optimistic.
| Set | Purpose | Touch Frequency | Typical Share |
|---|---|---|---|
| Train | Fit model weights | Every epoch/iteration | 60-80% |
| Validation | Tune hyperparameters, early stopping, model selection | Often | 10-20% |
| Test | Report final performance | Once | 10-20% |
For datasets under roughly 10,000 rows, replace the fixed validation set with k-fold cross-validation on the training portion. Keep the test set held out.
For datasets over 1 million rows, a 98/1/1 split is often fine. One percent of a million is still 10,000 examples, enough for a tight confidence interval.
Choosing the Right Splitting Strategy
Random Split
Use it only when rows are independent and identically distributed (i.i.d.).
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
X, y,
test_size=0.2,
random_state=42, # reproducibility
shuffle=True
)
Always set random_state. Without it, your results can’t be reproduced and comparisons between runs are noise.
Stratified Split
With a 2% positive class (fraud, rare disease, churn), a random 20% test set can end up with far fewer positives than expected. Stratification preserves class proportions in every set.
X_train, X_test, y_train, y_test = train_test_split(
X, y,
test_size=0.2,
stratify=y, # preserve class ratio
random_state=42
)
For cross-validation, use StratifiedKFold(n_splits=5, shuffle=True, random_state=42). For regression, bin the target into quantiles and stratify on the bins.
Group Split
If one patient, user, device or document produces multiple rows, a random split puts some of that entity’s rows in train and others in test. The model memorizes the entity and looks accurate on its “unseen” rows.
from sklearn.model_selection import GroupShuffleSplit
gss = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
train_idx, test_idx = next(gss.split(X, y, groups=df["patient_id"]))
X_train, X_test = X.iloc[train_idx], X.iloc[test_idx]
y_train, y_test = y.iloc[train_idx], y.iloc[test_idx]
# Sanity check: no overlap
assert set(df.iloc[train_idx]["patient_id"]).isdisjoint(
set(df.iloc[test_idx]["patient_id"])
)
For cross-validation, use GroupKFold. When you need both stratification and grouping, use StratifiedGroupKFold.
Time-Based Split
For forecasting, never shuffle. Training on the future to predict the past is leakage by definition.
from sklearn.model_selection import TimeSeriesSplit
tscv = TimeSeriesSplit(n_splits=5, gap=7) # 7-step gap prevents lag leakage
for fold, (train_idx, val_idx) in enumerate(tscv.split(X)):
print(f"Fold {fold}: train up to {train_idx[-1]}, validate {val_idx[0]}-{val_idx[-1]}")
The gap parameter drops samples between train and validation. If your features include a 7-day rolling mean, a gap of at least 7 stops window features from overlapping the boundary.
Common Mistakes (and How to Fix Them)
| # | Mistake | Symptom | Fix |
|---|---|---|---|
| 1 | Preprocessing before splitting | Test score unrealistically high | Wrap steps in a Pipeline |
| 2 | Duplicate/near-duplicate rows across sets | Score drops sharply after dedup | Deduplicate before splitting |
| 3 | Shuffling time series | Great backtest, poor live results | Chronological split with gap |
| 4 | Ignoring groups | Model memorizes entities | GroupKFold |
| 5 | Tuning on the test set | Test score optimistic | Separate validation set |
| 6 | Unstratified rare classes | Unstable, high-variance metrics | stratify=y |
| 7 | Target leakage in features | Suspiciously perfect accuracy | Audit features for post-outcome data |
| 8 | Test set too small | Wide confidence intervals | Compute required sample size |
Mistake 1: Preprocessing Before Splitting
This is the most common leak. Fitting a StandardScaler on the full dataset lets the test set’s mean and variance shape the training features. The effect is subtle with scaling but severe with target encoding, imputation by group mean, PCA, and feature selection.
Wrong:
from sklearn.preprocessing import StandardScaler
X_scaled = StandardScaler().fit_transform(X) # sees test data
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y)
Right:
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
pipe = Pipeline([
("scale", StandardScaler()),
("clf", LogisticRegression(max_iter=1000))
])
# The scaler is re-fit inside each training fold only
scores = cross_val_score(pipe, X_train, y_train, cv=5, scoring="roc_auc")
print(f"ROC-AUC: {scores.mean():.3f} ± {scores.std():.3f}")
A Pipeline guarantees each transformer is fit on training folds only. Use it every time.
Mistake 2: Duplicates and Near-Duplicates
Web-scraped data, customer records and product catalogs contain repeats. If the same row lands in both train and test, you’re measuring recall, not generalization.
# Exact duplicates
df = df.drop_duplicates(subset=feature_cols)
# Near-duplicates in text: normalize, then hash
import hashlib, re
def fingerprint(text: str) -> str:
norm = re.sub(r"\W+", " ", text.lower()).strip()
return hashlib.md5(norm.encode()).hexdigest()
df["fp"] = df["text"].map(fingerprint)
df = df.drop_duplicates(subset="fp")
For fuzzy matches, use MinHash LSH (the datasketch library) with a Jaccard threshold around 0.8. For embeddings, flag pairs above 0.95 cosine similarity.
Mistake 3: Target Leakage
A feature that wouldn’t exist at prediction time makes the model look brilliant. Examples include a discharge_status column in a hospital readmission model, or days_since_last_payment in a default predictor when the label is computed from payment behavior.
Audit with two checks:
- Train a single-feature model on each column. Any feature that alone gives near-perfect AUC deserves scrutiny.
- Ask for each feature: “Would I have this value at the moment of prediction?”
Mistake 4: Test Set Too Small
A 1,000-row test set with 90% accuracy has a 95% confidence interval of roughly ±1.9 points. Compare two models that differ by 1 point and you’re comparing noise. Estimate it with the normal approximation:
import math
def accuracy_ci(acc: float, n: int, z: float = 1.96) -> tuple[float, float]:
margin = z * math.sqrt(acc * (1 - acc) / n)
return acc - margin, acc + margin
print(accuracy_ci(0.90, 1000)) # (0.881, 0.919)
Splitting Strategies for LLM Work
Splits for large language models carry extra risks because training corpora are huge and opaque.
| Scenario | Risk | Mitigation |
|---|---|---|
| Fine-tuning on instruction data | Same prompt template or near-identical Q&A across train/eval | MinHash dedup; split by source or topic |
| Benchmark evaluation | Benchmark contamination (test items in pretraining data) | Use private or freshly written held-out sets |
| RAG evaluation | Eval questions answerable from the exact chunk used for tuning | Split by document, not by chunk |
| Synthetic data generation | Generated examples paraphrase eval items | Decontaminate with 13-gram overlap checks |
For RAG pipelines, always split at the document level. If chunks from one PDF appear in both your retrieval-tuning set and your evaluation set, retrieval metrics inflate.
A minimal n-gram decontamination check:
def ngrams(text: str, n: int = 13) -> set:
tokens = text.lower().split()
return {" ".join(tokens[i:i+n]) for i in range(len(tokens) - n + 1)}
train_grams = set().union(*(ngrams(t) for t in train_texts))
clean_eval = [t for t in eval_texts if ngrams(t).isdisjoint(train_grams)]
print(f"Kept {len(clean_eval)}/{len(eval_texts)} eval items")
Practical Workflow: A Leak-Proof Pipeline
Follow this sequence on any new supervised learning project.
- Inspect structure. Identify entities (users, patients), timestamps and class balance.
- Deduplicate. Remove exact and near-duplicate rows.
- Pick the split type using the table at the top of this guide.
- Carve out the test set first. Save it and don’t look at it.
- Build a
Pipeline. Put every fitted transformation inside it. - Cross-validate on training data to tune hyperparameters.
- Run one final evaluation on the test set and report it with a confidence interval.
- Re-check with a time-forward holdout if the model will run in production over months.
End-to-end example combining stratification, a pipeline and a final test evaluation:
import pandas as pd
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import roc_auc_score
df = pd.read_csv("data.csv").drop_duplicates()
X, y = df.drop(columns="label"), df["label"]
# 1. Hold out the test set first
X_dev, X_test, y_dev, y_test = train_test_split(
X, y, test_size=0.2, stratify=y, random_state=42
)
# 2. Tune with CV on the dev set only
pipe = Pipeline([
("scale", StandardScaler()),
("clf", GradientBoostingClassifier(random_state=42))
])
grid = GridSearchCV(
pipe,
{"clf__n_estimators": [100, 300], "clf__max_depth": [2, 3]},
cv=5, scoring="roc_auc", n_jobs=-1
)
grid.fit(X_dev, y_dev)
# 3. Single, final test evaluation
test_auc = roc_auc_score(y_test, grid.predict_proba(X_test)[:, 1])
print(f"CV AUC: {grid.best_score_:.3f} | Test AUC: {test_auc:.3f}")
A large gap between CV AUC and test AUC (more than a few points) signals either leakage in your CV setup or a distribution shift between the sets.
Real-World Scenarios
Healthcare, readmission prediction. A hospital has 80,000 admissions from 30,000 patients. A random row split puts the same patient in both sets, and the model learns patient-specific patterns. Switching to GroupKFold on patient_id typically drops reported AUC noticeably, and the lower number is the honest one.
E-commerce, demand forecasting. Shuffled K-fold gives low error because neighboring days leak seasonality and promotions. A rolling-origin evaluation with TimeSeriesSplit reveals the true forecast error at the planning horizon.
Enterprise RAG for support tickets. Tickets contain many near-identical templated replies. Without deduplication and document-level splitting, retrieval recall in testing looks near-perfect. After MinHash dedup and a split by customer account, the metric reflects actual retrieval quality.
FAQ
What is the best train/test split ratio?
80/20 is the common default, but the right ratio depends on data size. Use 70/30 or cross-validation for small datasets, and 98/1/1 for datasets above a million rows. The test set must be large enough to give a confidence interval narrow enough to separate your candidate models.
What is data leakage in machine learning?
Data leakage is when information from outside the training set influences the model during training. Common causes include preprocessing before splitting, duplicate rows across sets, features that encode the target, and shuffling time-ordered data. Leakage inflates validation metrics and causes poor production performance.
Should I split data before or after preprocessing?
Split first. Fit scalers, imputers, encoders and feature selectors on the training data only, then apply them to validation and test data. Use a scikit-learn Pipeline so this happens automatically inside each cross-validation fold.
When should I use stratified splitting?
Use it whenever classes are imbalanced, such as fraud at 1-5% prevalence. stratify=y keeps the class ratio the same in train and test sets, which stabilizes metrics like precision, recall and F1.




