One-Hot vs Label vs Target Encoding: Which Should You Use?

Models need numbers, and your city, product_id, or browser columns are strings. The encoder you pick decides whether a linear model invents fake order, whether a tree ensemble drowns in sparse columns, or whether your validation score quietly leaks the target. This guide compares the three most common encoders, shows production-grade scikit-learn code, and gives you a decision rule you can apply in under a minute.

Quick Takeaways

• One-hot encoding is the safe default for low-cardinality nominal features (roughly under 15–20 categories), especially with linear models, SVMs, and neural nets.
• Label (ordinal) encoding is correct for truly ordered categories (S < M < L). For unordered categories, use it only with tree-based models.
• Target encoding handles high-cardinality features (ZIP codes, user IDs) in one column, but it leaks the target unless you use smoothing and cross-fitting.
• Fit every encoder inside a pipeline on training folds only. Fitting on the full dataset is the most common encoding bug.

What Categorical Encoding Actually Does

A categorical encoder maps a discrete set of values to a numeric representation. The mapping defines the geometry your model sees.

• A linear model computes w·x. If you encode red=1, green=2, blue=3, the model assumes green sits exactly between red and blue and that the effect is linear in that order.
• A decision tree splits on thresholds. It can carve x<=1 and x<=2 to isolate categories, so arbitrary integer codes cost it far less.
• A neural network learns from distances. Arbitrary integers inject distances that do not exist.

The question is never “which encoder is best.” It is “what structure does this encoder imply, and does my model respect it?”

One-Hot Encoding

One-hot encoding creates one binary column per category. A row with color="green" becomes [0, 1, 0].

How it works

For a feature with k categories, you get k columns (or k−1 with drop="first" to avoid the dummy variable trap in unregularized linear regression). No order is implied, and every category sits at equal distance from every other.

Strengths

• Makes no ordering assumption.
• Works with every model family.
• Zero leakage risk, because the mapping uses no target information.

Weaknesses

• Dimensionality explosion. A feature with 10,000 categories adds 10,000 columns.
• Sparse matrices slow down tree libraries and inflate memory.
• Tree models split poorly on sparse binary columns: each split isolates one category, so deep trees are needed.

Code: one-hot with rare-category grouping

from sklearn.preprocessing import OneHotEncoder

ohe = OneHotEncoder(
    handle_unknown="infrequent_if_exist",  # unseen -> infrequent bucket
    min_frequency=0.01,                    # categories under 1% of rows are grouped
    sparse_output=True,                    # keep memory low
)

min_frequency and max_categories (available in scikit-learn 1.1+) cap column growth without custom code.

Label (Ordinal) Encoding

Label encoding maps each category to an integer: {"S": 0, "M": 1, "L": 2}.

Two very different uses

1. Ordinal features. Education level, T-shirt size, and satisfaction ratings have a real order. Encode them explicitly in that order.
2. Nominal features with tree models. city has no order, but random forests, XGBoost, and LightGBM can still split on integer codes. It is compact and often works, though the splits are arbitrary.

Strengths

• One column, regardless of cardinality.
• Fast and memory-light.
• The correct choice when order is real.

Weaknesses

• On nominal data, linear models, SVMs, KNN, and neural nets read the integers as magnitudes. That adds noise or bias.
• Alphabetical default ordering from LabelEncoder is meaningless and a common source of bugs.

Code: ordinal encoding with explicit order

from sklearn.preprocessing import OrdinalEncoder

size_order = [["S", "M", "L", "XL"]]  # one list per column

oe = OrdinalEncoder(
    categories=size_order,
    handle_unknown="use_encoded_value",
    unknown_value=-1,                 # unseen sizes -> -1
)

Use OrdinalEncoder for features and reserve LabelEncoder for the target variable only. LabelEncoder accepts a single 1-D array and gives you no explicit category order.

Target Encoding

Target encoding replaces each category with a statistic of the target, usually the mean. For binary classification, city="Austin" becomes the observed fraction of positives among Austin rows.

The smoothing formula

Raw means are noisy for rare categories. A category seen twice with one positive gets an encoding of 0.5, which is mostly noise. Smoothing blends the category mean with the global mean:

encoding = (n × category_mean + m × global_mean) / (n + m)

• n is the number of rows in the category.
• m is the smoothing strength (larger m pulls rare categories toward the global mean).

When n is small, the encoding collapses toward the global mean. When n is large, it trusts the category’s own mean.

The leakage problem

If you compute category means on the training set and then train on those same rows, each row’s encoding contains its own label. The model sees a feature that partially encodes the answer. Training scores look excellent. Test scores collapse.

The fix is cross-fitting (also called out-of-fold encoding): split the training data into K folds, and encode each fold using statistics computed from the other K−1 folds. Scikit-learn’s TargetEncoder (added in 1.3) does this automatically inside fit_transform.

Code: target encoding done safely

from sklearn.preprocessing import TargetEncoder

te = TargetEncoder(
    smooth="auto",     # empirical-Bayes smoothing, no manual m
    cv=5,              # 5-fold cross-fitting inside fit_transform
    target_type="auto",
    random_state=42,
)

X_train_enc = te.fit_transform(X_train[["zip_code"]], y_train)  # cross-fitted
X_test_enc  = te.transform(X_test[["zip_code"]])                # full-train stats

Note the asymmetry: fit_transform uses cross-fitting, while transform uses statistics from the full training set. Never call fit(X, y).transform(X) on training data. That path skips cross-fitting and leaks.

Strengths

• One column for any cardinality.
• Encodes predictive signal directly, which helps gradient boosting on ID-like features.

Weaknesses

• Leakage if implemented naively.
• Higher variance on rare categories.
• Brittle under distribution shift: if category-target relationships drift, the encoding goes stale.

Decision Framework: Which Encoder Should You Use?

Work through these questions in order.

1. Is there a real order? Use ordinal encoding with an explicit category list.
2. Is cardinality low (under ~15–20) and the model linear, SVM, or neural? Use one-hot.
3. Is cardinality high (50+) and the model is a gradient-boosted tree? Use target encoding (cross-fitted) or the library’s native categorical handling.
4. Is cardinality moderate and the model tree-based? Test ordinal against one-hot with cross-validation. Both are defensible.
5. Is cardinality very high and the model is a neural net? Use learned embeddings instead of any of these three.

Practical Workflow: A Leak-Free Pipeline

Put encoders inside a ColumnTransformer and a Pipeline. Cross-validation then refits every encoder on each training fold automatically.

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, TargetEncoder
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import cross_val_score

df = pd.read_csv("customers.csv")
X = df.drop(columns="churned")
y = df["churned"]

preprocess = ColumnTransformer(
    transformers=[
        ("onehot", OneHotEncoder(handle_unknown="ignore", min_frequency=0.01),
         ["browser", "payment_method"]),
        ("ordinal", OrdinalEncoder(
            categories=[["basic", "plus", "pro"]],
            handle_unknown="use_encoded_value", unknown_value=-1),
         ["plan_tier"]),
        ("target", TargetEncoder(smooth="auto", cv=5, random_state=42),
         ["zip_code", "referrer_domain"]),
    ],
    remainder="passthrough",
)

model = Pipeline([
    ("prep", preprocess),
    ("clf", HistGradientBoostingClassifier(max_iter=300, learning_rate=0.05)),
])

scores = cross_val_score(model, X, y, cv=5, scoring="roc_auc")
print(f"AUC: {scores.mean():.3f} ± {scores.std():.3f}")

This pipeline applies each encoder to the columns where it fits and refits everything per fold. No statistics from validation rows ever reach training.

Real-World Use Cases

Fraud detection with high-cardinality merchants

A payments team has 80,000 merchant IDs. One-hot would add 80,000 columns. Ordinal codes are arbitrary. A cross-fitted target encoder compresses each merchant into one fraud-rate feature. The team adds a second feature, merchant transaction count, so the model can judge how much to trust the rate.

Churn prediction with mixed feature types

A SaaS company combines plan_tier (ordinal), signup_channel (6 values, one-hot), and company_domain (12,000 values, target encoded). The pipeline above reflects this structure, and cross-validation catches any leakage before launch.

Insurance pricing with linear models

Regulators often require interpretable coefficients. One-hot encoding keeps one coefficient per category, which regulators and actuaries can audit. Rare levels are grouped with min_frequency to stabilize estimates.

Common Mistakes to Avoid

• Fitting encoders before the train/test split. Statistics from test rows contaminate training.
• Using LabelEncoder on input features. It is built for targets and ignores your intended order.
• Target encoding without smoothing. Rare categories get extreme values.
• Ignoring unseen categories. Production data has values your training set never saw. Set handle_unknown explicitly.
• Applying the dummy-variable trap fix everywhere. drop="first" matters for unregularized linear regression. It is unnecessary for trees and regularized models.

Beyond the Big Three

CatBoost uses ordered target statistics internally, which prevents the leakage problem by design. If your data is mostly categorical, test it before building a custom encoding stack.

FAQ

Is one-hot or label encoding better?

Neither wins universally. One-hot is better for nominal features in linear models, SVMs, and neural networks because it implies no order. Label (ordinal) encoding is better for ordered categories and for tree-based models on unordered features, where its single column keeps the data compact.

When should I use target encoding?

Use target encoding for high-cardinality features, typically 50 or more categories, when one-hot would create too many columns. Apply smoothing and cross-fitting (scikit-learn’s TargetEncoder with cv=5 does both) to prevent target leakage.

Does label encoding work for tree-based models?

Yes. Random forests, XGBoost, and LightGBM can split on integer codes, so label encoding on nominal data often works well. Results vary by dataset, so compare it against one-hot with cross-validation.

How do I avoid data leakage with target encoding?

Fit the encoder only on training folds, use out-of-fold encoding for training rows, and apply smoothing. Wrap the encoder in a Pipeline so cross-validation refits it per fold. Never run fit and transform separately on the same training data.