Regularization
L1, L2, dropout, and preventing overfitting
Skills taught: ml.regularization, ml.lasso, ml.ridge, ml.bias-variance
Overfitting and Regularization
Overfitting means the model memorises training data — including its random noise — and fails on new examples. The gap between training score and validation score is the tell-tale sign. Regularisation adds a penalty on complexity to keep the model general.
The Bias-Variance Trade-off
Bias is the error from underfitting — the model is too simple to capture the pattern. Variance is the error from overfitting — the model is too sensitive to the specific training data. The goal is to find the sweet spot where total error is minimised.
- High bias / low variance: simple model, underfits — training and test error are both high
- Low bias / high variance: complex model, overfits — training error is low, test error is high
- Good balance: validation error is close to training error and both are acceptably low
L2 Regularization: Ridge
from sklearn.linear_model import Ridge, RidgeCV
from sklearn.preprocessing import StandardScaler
import numpy as np
# Ridge adds alpha * sum(coef^2) to the loss
# Higher alpha = stronger regularisation = smaller coefficients
ridge = Ridge(alpha=1.0)
ridge.fit(X_train_sc, y_train)
print(f'Ridge R²: {ridge.score(X_test_sc, y_test):.3f}')
print(f'Coef range: {ridge.coef_.min():.3f} to {ridge.coef_.max():.3f}')
# Automatically choose alpha with cross-validation
alphas = [0.001, 0.01, 0.1, 1.0, 10.0, 100.0]
ridge_cv = RidgeCV(alphas=alphas, cv=5)
ridge_cv.fit(X_train_sc, y_train)
print(f'Best alpha: {ridge_cv.alpha_}')
L1 Regularization: Lasso (Feature Selection)
from sklearn.linear_model import Lasso, LassoCV
import numpy as np
# Lasso adds alpha * sum(|coef|) to the loss
# This drives unimportant coefficients to exactly 0
lasso = Lasso(alpha=0.1)
lasso.fit(X_train_sc, y_train)
n_nonzero = np.sum(lasso.coef_ != 0)
print(f'Non-zero features: {n_nonzero} / {X_train_sc.shape[1]}')
print(f'Selected features: {list(X.columns[lasso.coef_ != 0])}')
print(f'Lasso R²: {lasso.score(X_test_sc, y_test):.3f}')
# Cross-validate to find best alpha
lasso_cv = LassoCV(alphas=None, cv=5) # None = auto-generate alphas
lasso_cv.fit(X_train_sc, y_train)
print(f'Best alpha: {lasso_cv.alpha_:.4f}')
L1 (Lasso) vs L2 (Ridge) Regularization
- L1 / Lasso: Penalises sum of absolute coefficient values
- L2 / Ridge: Penalises sum of squared coefficient values
ElasticNet: Combining L1 and L2
from sklearn.linear_model import ElasticNet, ElasticNetCV
# ElasticNet = L1 * l1_ratio + L2 * (1-l1_ratio)
# l1_ratio=1 is pure Lasso; l1_ratio=0 is pure Ridge
elastic = ElasticNet(alpha=0.1, l1_ratio=0.5)
elastic.fit(X_train_sc, y_train)
print(f'ElasticNet R²: {elastic.score(X_test_sc, y_test):.3f}')
print(f'Non-zero: {(elastic.coef_ != 0).sum()}')
Tip: Always scale features before applying Ridge or Lasso. The penalty applies to the coefficient magnitude, so unscaled features with large ranges will have smaller coefficients and receive less regularisation.
Frequently Asked Questions
How do I choose the regularization strength (alpha)?
Use RidgeCV or LassoCV which perform cross-validation automatically over a range of alpha values. For a Pipeline, use GridSearchCV to search over alpha values as a hyperparameter.
What is the difference between regularisation and pruning (for decision trees)?
They address the same problem (overfitting) in different ways. Regularisation adds a penalty to the loss function for linear models. Pruning constrains the tree's growth (max_depth, min_samples_leaf) or removes branches after training.
Should I use Ridge or Lasso first?
Start with Ridge — it is stable and works well when most features contribute. If you suspect only a few features are important and want automatic feature selection, try Lasso. ElasticNet is a safe default for high-dimensional data.