What Is Feature Engineering? Turn Raw Data Into Better Predictions

A model is only as good as the signals you feed it. Feature engineering is the practice of transforming raw data into inputs that expose those signals clearly. A gradient-boosted tree trained on well-crafted features routinely beats a deep network trained on raw columns. This guide covers the core techniques, working code, and where feature engineering fits now that embeddings and LLMs handle much of the representation work.

Quick Takeaways

• Feature engineering converts raw variables into numeric representations that make patterns easier for a model to learn.
• Core techniques: encoding, scaling, binning, interaction terms, date/time decomposition, text vectorization, and feature selection.
• Tabular data still rewards manual feature work. Unstructured data (text, images) increasingly relies on learned embeddings such as 768-, 1,536-, or 3,072-dimension vectors.
• The biggest silent risk is data leakage. Fit every transformer on training data only.

What Feature Engineering Actually Means

A feature is a measurable input variable. Feature engineering is the creation, transformation, and selection of those variables so an algorithm can exploit them.

Consider a raw timestamp column. Most models cannot extract meaning from 2026-03-14 18:42:07. Split it into hour of day, day of week, is_weekend, and days since last purchase, and the model suddenly sees behavioral rhythm.

Three activities fall under the umbrella:

1. Feature creation: deriving new variables from existing ones.
2. Feature transformation: reshaping distributions or encoding types.
3. Feature selection: removing noise, redundancy, and leakage.

Why It Matters for Model Performance

Algorithms have inductive biases. Linear models assume additive relationships. Distance-based models assume comparable scales. Trees assume axis-aligned splits. Feature engineering bridges the gap between your data’s shape and the model’s assumptions.

Gains show up in three places:

• Accuracy: better signal, lower error.
• Training speed: fewer, denser features converge faster.
• Interpretability: debt_to_income_ratio explains a decision better than two raw columns.

Core Techniques, With Code

All examples use Python with scikit-learn 1.5+ and pandas 2.x.

1. Handling Missing Values

Imputation is the first gate. Missingness itself is often informative, so flag it.

import pandas as pd
from sklearn.impute import SimpleImputer

df["income_missing"] = df["income"].isna().astype(int)  # keep the signal
imputer = SimpleImputer(strategy="median")
df[["income"]] = imputer.fit_transform(df[["income"]])

Median imputation resists outliers. For categorical columns, use strategy="most_frequent" or an explicit "Unknown" category.

2. Encoding Categorical Variables

from sklearn.preprocessing import OneHotEncoder

enc = OneHotEncoder(handle_unknown="ignore", sparse_output=False)
X_cat = enc.fit_transform(df[["city"]])

Setting handle_unknown="ignore" prevents crashes when production data contains unseen categories.

3. Scaling and Normalization

Distance- and gradient-based models need comparable ranges.

Standardization applies z = (x − μ) / σ, producing mean 0 and unit variance. Min-max scaling maps values to [0, 1]. Robust scaling uses the median and IQR, which suits outlier-heavy data.

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)   # fit on train only
X_test_scaled = scaler.transform(X_test)          # reuse train statistics

Tree ensembles (Random Forest, XGBoost, LightGBM) do not need scaling.

4. Transforming Skewed Distributions

Right-skewed variables like price or income distort linear models. Apply a log transform or a Yeo-Johnson power transform.

import numpy as np
df["log_price"] = np.log1p(df["price"])  # log1p handles zeros safely

5. Binning and Discretization

Binning converts continuous values into ranges, which captures non-linear effects in linear models.

df["age_group"] = pd.cut(df["age"], bins=[0, 18, 35, 55, 120],
                         labels=["teen", "young", "mid", "senior"])

Use pd.qcut for equal-frequency bins.

6. Interaction and Polynomial Features

Interactions capture relationships a model cannot see alone. Price per square foot beats price and area in isolation.

df["price_per_sqft"] = df["price"] / df["sqft"]

from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=2, interaction_only=True, include_bias=False)
X_poly = poly.fit_transform(X[["sqft", "bedrooms", "age"]])

Degree-2 expansion on n features yields roughly n(n+1)/2 new columns. Watch dimensionality.

7. Date and Time Features

df["hour"] = df["ts"].dt.hour
df["dow"] = df["ts"].dt.dayofweek
df["is_weekend"] = (df["dow"] >= 5).astype(int)

# Cyclical encoding so hour 23 sits next to hour 0
df["hour_sin"] = np.sin(2 * np.pi * df["hour"] / 24)
df["hour_cos"] = np.cos(2 * np.pi * df["hour"] / 24)

For forecasting, add lag features and rolling statistics:

df["sales_lag_7"] = df["sales"].shift(7)
df["sales_roll_30_mean"] = df["sales"].shift(1).rolling(30).mean()

The shift(1) before the rolling window prevents the current row from leaking into its own feature.

8. Text Features

Classic approach: TF-IDF with unigrams and bigrams.

from sklearn.feature_extraction.text import TfidfVectorizer

tfidf = TfidfVectorizer(ngram_range=(1, 2), max_features=20000, stop_words="english")
X_text = tfidf.fit_transform(df["review"])

Modern approach: dense embeddings from a sentence-transformer or an embedding API, typically 384 to 3,072 dimensions, fed into a downstream classifier.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")   # 384-dim vectors
X_emb = model.encode(df["review"].tolist(), batch_size=64)

Feature Selection: Keep Signal, Drop Noise

More features do not guarantee better models. Irrelevant columns raise variance and overfitting risk.

from sklearn.feature_selection import mutual_info_classif, SelectKBest

selector = SelectKBest(mutual_info_classif, k=25)
X_sel = selector.fit_transform(X_train, y_train)

For tree models, compare permutation importance against built-in impurity importance. Impurity scores favor high-cardinality features and can mislead.

The Leakage Trap: Pipelines Done Right

Leakage occurs when information from the test set or the future contaminates training. It produces inflated validation scores and failed deployments.

Wrap all steps in a Pipeline with a ColumnTransformer so each cross-validation fold fits transformers independently.

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import cross_val_score

num_cols = ["age", "income", "tenure"]
cat_cols = ["city", "plan"]

preprocess = ColumnTransformer([
    ("num", Pipeline([("imp", SimpleImputer(strategy="median")),
                      ("sc", StandardScaler())]), num_cols),
    ("cat", OneHotEncoder(handle_unknown="ignore"), cat_cols),
])

clf = Pipeline([("prep", preprocess),
                ("model", HistGradientBoostingClassifier(random_state=42))])

scores = cross_val_score(clf, X, y, cv=5, scoring="roc_auc")
print(scores.mean())

For time series, replace cv=5 with TimeSeriesSplit so folds respect chronology.

Feature Engineering vs. Feature Learning

Deep nets learn representations, but benchmarks on tabular data continue to show well-tuned gradient boosting with sound features competing strongly against neural approaches. Treat them as complementary: use embeddings for unstructured fields, engineered features for structured ones, and combine both in one model.

LLMs as Feature Engineers

LLMs now extract structured features from messy text at scale. A support ticket becomes sentiment, urgency, product_area, and is_refund_request columns that feed a classic tabular model.

import anthropic, json

client = anthropic.Anthropic()

def extract_features(ticket: str) -> dict:
    msg = client.messages.create(
        model="claude-sonnet-5-5",
        max_tokens=300,
        temperature=0,   # deterministic extraction
        system="Return only valid JSON with keys: sentiment (-1..1), "
               "urgency (1-5), product_area (string), refund_request (bool).",
        messages=[{"role": "user", "content": ticket}],
    )
    return json.loads(msg.content[0].text)

Practical settings: use temperature 0, enforce a JSON schema, validate outputs, and cache results. Batch processing cuts cost on large datasets. Always spot-check a labeled sample, since extraction errors become feature noise.

Real-World Workflows

Churn prediction (SaaS)

Result: behavioral ratios and recency signals typically outperform raw counts.

Fraud detection (payments)

Build velocity features (transactions in the last 1 hour, 24 hours), deviation features (amount vs. the customer’s 90-day median), and geo-distance between consecutive transactions. Compute them point-in-time to avoid leaking future events.

A repeatable process

1. Profile the data: distributions, missingness, cardinality.
2. Baseline first: train a simple model on minimal features.
3. Add features in groups and measure the validation delta for each.
4. Prune features that add nothing.
5. Validate with leakage-safe cross-validation.
6. Productionize the same pipeline for training and inference.

Common Mistakes

• Fitting scalers or encoders before the train/test split.
• Target encoding without out-of-fold computation.
• One-hot encoding a column with 50,000 categories.
• Using future information in time-series features.
• Ignoring training-serving skew, where production computes features differently than training did.

A feature store such as Feast or Tecton helps keep definitions consistent across both environments.

FAQ

What is feature engineering in simple terms?

Feature engineering is the process of turning raw data into informative input variables that help a machine learning model find patterns. It includes creating new variables, transforming existing ones, and selecting the most useful subset.

Is feature engineering still needed with deep learning?

Yes, for tabular and structured data. Deep learning reduces manual work on text, images, and audio through learned representations. But domain-driven features like ratios, lags, and aggregates still improve accuracy and interpretability on structured datasets.

What is the difference between feature engineering and feature selection?

Feature engineering creates and transforms variables. Feature selection chooses which variables to keep. Selection is one stage within the broader engineering process and aims to reduce noise, overfitting, and training cost.

Which feature engineering techniques matter most?

Start with missing-value handling, categorical encoding, scaling (for distance and gradient models), date/time decomposition, and domain-specific ratios or aggregates. These deliver the largest gains for the least effort on most tabular problems.