Models learn only from the columns you give them. This page covers how to turn raw data into informative, leak-free features: encoding, scaling, interactions, time features, and wrapping it all in a reproducible pipeline.
A feature is a numeric input column a model learns from. Feature engineering is the work of turning raw records — timestamps, categories, free text, event logs — into columns that expose the signal a model needs, in a form the algorithm can use. It covers cleaning, encoding categories as numbers, putting numeric columns on comparable scales, creating new columns from combinations of old ones, and summarising histories into fixed-width aggregates.
The pipeline above is the mental model for the whole page. Raw data is cleaned, transformed and combined into a feature matrix X. The yellow line at the bottom is the rule that matters most: every step that learns something from data (a mean, a vocabulary, a category frequency) must learn it from training rows only.
For tabular problems, good features usually move results more than switching algorithms. A gradient-boosted tree given days_since_last_purchase will beat the same model given only a raw list of purchase timestamps, because the tree cannot compute the subtraction itself. Linear models depend on features even more: they can only draw straight boundaries, so curvature and interactions must be supplied as columns.
Deep learning reduced manual feature work for images, audio and text, where networks learn representations from raw pixels or tokens. For business data in tables, however, careful features remain the main lever, and they also make models easier to explain: "risk rises with debt-to-income ratio" is clearer than an opaque embedding.
Most algorithms need numbers, so categories must be encoded. One-hot encoding creates one binary column per category; it implies no ordering and suits linear models, but explodes for high-cardinality columns like ZIP codes. Ordinal encoding maps each category to an integer; it is compact and fine for tree models, but wrong for linear models unless the categories truly are ordered (small < medium < large). Target encoding replaces a category with the mean of the target for that category, for example the historical click rate per advertiser. It is powerful for high cardinality but leaks the label unless computed out-of-fold and smoothed toward the global mean:
enc(c) = (nc · meanc + m · global_mean) / (nc + m)
where nc is the count of rows in category c and m is a smoothing weight. Rare categories are pulled toward the overall mean so a single lucky row cannot dominate. Other options include frequency encoding (replace with the count) and hashing (map into a fixed number of buckets).
Standardization rescales a column to mean 0 and standard deviation 1: z = (x − μ) / σ. Min-max scaling maps it to [0, 1]: (x − min) / (max − min). Scaling matters for anything driven by distances or gradients — k-nearest neighbours, SVMs, regularised linear models, neural networks — and does nothing for tree models, which split on thresholds.
Skewed quantities such as income or page views often benefit from log(1 + x), which compresses the long tail. Binning turns a number into ranges, useful when the relationship is step-like. Interactions multiply or divide columns: debt / income, price per square metre, or the product of two flags. Date features extract hour, weekday, month, holiday flags and "time since" values; cyclic quantities like hour-of-day are often encoded as sin(2πh/24) and cos(2πh/24) so 23:00 sits next to 00:00. Aggregates summarise history per entity: counts and means over the last 7 or 30 days.
Missing values deserve their own thought. Imputing with the median is a baseline, but adding a binary was_missing indicator lets the model learn when absence itself is informative.
Leakage is any path by which information unavailable at prediction time reaches the training data. The left panel shows the subtle form: fitting a scaler or encoder on the full dataset lets test-set statistics shape the training features, so validation scores become optimistic. The right panel is the fix: split first, fit on training rows, and only transform validation and test rows.
The blatant form is a feature that encodes the answer: a refund_issued column when predicting fraud, or a 30-day aggregate computed with a window that extends past the prediction timestamp. The defence is to ask, for every feature, "would I know this value at the moment I need the prediction?" and to build time-based aggregates with an explicit cut-off.
A Pipeline with a ColumnTransformer bundles every step so cross-validation refits them per fold automatically and the same object is used at serving time.
import numpy as np, pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler, FunctionTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
rng = np.random.default_rng(0)
n = 2000
df = pd.DataFrame({
"income": rng.lognormal(10.5, 0.6, n),
"debt": rng.lognormal(9.5, 0.8, n),
"plan": rng.choice(["basic", "plus", "pro"], n),
"signup_hour": rng.integers(0, 24, n),
})
df.loc[rng.random(n) < 0.05, "income"] = np.nan
y = ((df["debt"] / df["income"].fillna(df["income"].median())) > 0.45).astype(int)
def add_features(X):
X = X.copy()
X["dti"] = X["debt"] / X["income"] # interaction (ratio)
X["hour_sin"] = np.sin(2 * np.pi * X["signup_hour"] / 24)
X["hour_cos"] = np.cos(2 * np.pi * X["signup_hour"] / 24)
return X
num = ["income", "debt", "dti", "hour_sin", "hour_cos"]
prep = ColumnTransformer([
("num", Pipeline([("impute", SimpleImputer(strategy="median", add_indicator=True)),
("log", FunctionTransformer(np.log1p)),
("scale", StandardScaler())]), num),
("cat", OneHotEncoder(handle_unknown="ignore"), ["plan"]),
])
model = Pipeline([("features", FunctionTransformer(add_features)),
("prep", prep),
("clf", LogisticRegression(max_iter=1000))])
print(cross_val_score(model, df, y, cv=5, scoring="roc_auc").mean())Remove the dti line and the score drops noticeably, because a linear model cannot form the ratio from income and debt on its own. That one engineered column carries most of the signal. (Because the label here is built from that ratio, the example deliberately exaggerates the effect; real data is noisier.)
| Approach | Strength | Weakness | Use when |
|---|---|---|---|
| Hand-crafted features | Encodes domain knowledge, interpretable | Labour-intensive, can miss patterns | Tabular business data, small datasets |
| Automated (featuretools, tsfresh) | Generates many candidates fast | Bloat, leakage risk, needs selection | Relational or time-series data with many tables |
| Learned representations (deep nets) | No manual design for raw signals | Needs lots of data and compute, opaque | Images, audio, text |
| Pretrained embeddings | Transfers knowledge from large corpora | Domain mismatch, dimensionality | Text or IDs fed into tabular models |
handle_unknown="ignore" or map to an "other" bucket.