Feature Engineering
Creating and selecting features that improve model performance
Skills taught: ml.features, ml.encoding, ml.feature-engineering
What is Feature Engineering?
Feature engineering is the craft of transforming raw data into informative inputs (features) that help the model learn. It is where domain knowledge and ML skill intersect — and it is typically the single biggest lever on model performance.
Derived Features
Create new columns by combining or transforming existing ones. The goal is to expose relationships that the algorithm cannot discover on its own (especially for linear models).
import pandas as pd
import numpy as np
# --- Ratio feature ---
df['price_per_sqft'] = df['price'] / df['sqft']
# --- Interaction feature ---
df['income_x_age'] = df['income'] * df['age']
# --- Date decomposition ---
df['date'] = pd.to_datetime(df['date'])
df['day_of_week'] = df['date'].dt.dayofweek # 0=Mon, 6=Sun
df['month'] = df['date'].dt.month
df['is_weekend'] = df['day_of_week'] >= 5
df['days_since_launch'] = (df['date'] - df['date'].min()).dt.days
# --- Log transform for skewed target ---
df['log_revenue'] = np.log1p(df['revenue'])
Encoding Categorical Features
Most algorithms need numbers. Use one-hot encoding for unordered categories (city, colour). Use ordinal encoding for ordered categories (small < medium < large). For high-cardinality categoricals (1000+ unique values), use target encoding or embeddings.
import pandas as pd
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder
# One-hot encoding (unordered categories)
ohe = OneHotEncoder(handle_unknown='ignore', sparse_output=False)
city_enc = ohe.fit_transform(X_train[['city']])
# Ordinal encoding (ordered categories)
ordinal = OrdinalEncoder(categories=[['low', 'medium', 'high']])
risk_enc = ordinal.fit_transform(X_train[['risk_level']])
# Target encoding for high-cardinality (mean of target per category)
target_means = X_train.groupby('category')['target'].mean()
X_train['cat_encoded'] = X_train['category'].map(target_means)
X_test['cat_encoded'] = X_test['category'].map(target_means).fillna(target_means.mean())
Binning Continuous Variables
import pandas as pd
import numpy as np
from sklearn.preprocessing import KBinsDiscretizer
# Manual bins with meaningful labels
df['age_group'] = pd.cut(
df['age'],
bins=[0, 18, 35, 60, 100],
labels=['child', 'young_adult', 'adult', 'senior']
)
# Quantile bins (equal frequency)
df['income_quartile'] = pd.qcut(df['income'], q=4, labels=['Q1', 'Q2', 'Q3', 'Q4'])
# Automated binning with sklearn
kbins = KBinsDiscretizer(n_bins=5, encode='ordinal', strategy='quantile')
df['age_bin'] = kbins.fit_transform(df[['age']])
Tip: Domain knowledge beats fancy algorithms. A well-engineered feature often improves accuracy more than switching from logistic regression to a gradient boosted tree.
Warning: Avoid feature leakage: never include a feature that directly encodes the target label (e.g., using tomorrow's stock price to predict today's). The model will learn a spurious pattern and fail completely in production.
Polynomial Features
from sklearn.preprocessing import PolynomialFeatures
# Generate x, x^2, x1*x2 interactions
poly = PolynomialFeatures(degree=2, include_bias=False)
X_poly = poly.fit_transform(X_train)
print(f'Original features: {X_train.shape[1]}')
print(f'Polynomial features: {X_poly.shape[1]}')
print(f'Feature names: {poly.get_feature_names_out()[:10]}')
Polynomial features grow quickly: 10 original features → 65 degree-2 features. Only use them when the dataset is not high-dimensional, or combine with regularisation (Lasso) to remove unimportant ones.
Frequently Asked Questions
What is feature leakage?
Including a feature that directly encodes the target label (e.g., using tomorrow's price to predict today's). It makes the model look accurate but it won't generalise — it has memorised the target, not learned a pattern.
Should I always do feature engineering before trying a model?
Start with a simple baseline model on raw features first. This gives you a performance floor. Then engineer features and see if they improve it. This prevents over-engineering features that don't help.
How do I know which features to create?
Talk to domain experts, look at correlations with the target, and think about which combinations are physically or logically meaningful. Automated tools like FeatureTools can help explore interaction features systematically.