Feature Selection
Removing irrelevant features to improve models
Skills taught: ml.feature-selection, ml.permutation-importance, ml.rfe
Why Select Features?
Fewer, better features reduce overfitting, speed up training, and make models more interpretable. Including irrelevant or redundant features introduces noise and can actually hurt performance. The goal: keep only the features that genuinely help predict the target.
- Reduces overfitting — fewer parameters to learn from noise
- Speeds up training and prediction — less computation
- Improves interpretability — easier to explain 10 features than 500
- Reduces data collection costs — if 20 features are enough, don't measure 200
Method 1: Filter Methods (Statistical Tests)
Filter methods rank features by their statistical relationship with the target — independently of any model. They are fast but can miss complex interactions between features.
from sklearn.feature_selection import SelectKBest, f_classif, mutual_info_classif
import pandas as pd
# F-statistic (ANOVA): tests linear relationship, good for numeric features
selector_f = SelectKBest(score_func=f_classif, k=10)
X_new_f = selector_f.fit_transform(X_train, y_train)
selected_mask_f = selector_f.get_support()
print('Selected by F-test:', list(X.columns[selected_mask_f]))
# Mutual information: captures non-linear relationships
selector_mi = SelectKBest(score_func=mutual_info_classif, k=10)
X_new_mi = selector_mi.fit_transform(X_train, y_train)
selected_mask_mi = selector_mi.get_support()
print('Selected by MI:', list(X.columns[selected_mask_mi]))
# View scores
scores = pd.Series(selector_mi.scores_, index=X.columns)
print(scores.sort_values(ascending=False).head(15))
Method 2: Recursive Feature Elimination (RFE)
RFE trains a model, removes the least important feature, then retrains — repeating until only the desired number of features remain. It captures feature interactions that filter methods miss.
from sklearn.feature_selection import RFE, RFECV
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import StratifiedKFold
# Standard RFE: select exactly N features
rf = RandomForestClassifier(n_estimators=100, random_state=42)
rfe = RFE(estimator=rf, n_features_to_select=15, step=1)
rfe.fit(X_train, y_train)
selected = [feat for feat, sel in zip(X.columns, rfe.support_) if sel]
print(f'Selected {len(selected)} features:', selected)
print(f'Score with selected features: {rfe.score(X_test, y_test):.3f}')
# RFECV: automatically find the optimal number of features
rfecv = RFECV(
estimator=RandomForestClassifier(n_estimators=100, random_state=42),
cv=StratifiedKFold(5),
scoring='f1',
min_features_to_select=5
)
rfecv.fit(X_train, y_train)
print(f'Optimal number of features: {rfecv.n_features_}')
Method 3: Permutation Importance
Permutation importance measures how much model performance drops when a feature's values are randomly shuffled. If shuffling a feature doesn't hurt performance, it was not useful. This works with any fitted model and is more reliable than tree-based feature importance.
from sklearn.inspection import permutation_importance
import pandas as pd
import matplotlib.pyplot as plt
# Train the model first
model = RandomForestClassifier(n_estimators=200, random_state=42)
model.fit(X_train, y_train)
# Permutation importance on the test set
result = permutation_importance(
model, X_test, y_test,
n_repeats=10, # shuffle each feature 10 times and average
random_state=42,
scoring='f1',
n_jobs=-1
)
perm_df = pd.DataFrame({
'feature': X.columns,
'importance_mean': result.importances_mean,
'importance_std': result.importances_std
}).sort_values('importance_mean', ascending=False)
print(perm_df.head(15))
# Remove features that hurt performance when shuffled (negative or near-zero importance)
useful_features = perm_df[perm_df['importance_mean'] > 0.001]['feature'].tolist()
print(f'Keeping {len(useful_features)} useful features')
Method 4: Lasso (Embedded Feature Selection)
from sklearn.linear_model import LassoCV
from sklearn.preprocessing import StandardScaler
import numpy as np
scaler = StandardScaler()
X_train_sc = scaler.fit_transform(X_train)
# LassoCV finds optimal alpha via CV and drives unimportant coefficients to zero
lasso = LassoCV(cv=5, random_state=42)
lasso.fit(X_train_sc, y_train)
selected_mask = lasso.coef_ != 0
selected_features = X.columns[selected_mask]
print(f'Lasso selected {selected_mask.sum()} / {len(X.columns)} features')
print(f'Selected: {list(selected_features)}')
Tip: Prefer permutation importance over tree-based feature importance (model.feature_importances_). Tree importance is biased towards high-cardinality features and can be misleading. Permutation importance is model-agnostic and evaluated on the test set.
Warning: Always do feature selection on the training set only. If you rank features by correlation with the target on the full dataset first, you've used test-set information — a form of data leakage.
Frequently Asked Questions
Can feature importance from Random Forest be misleading?
Yes. Random Forest's built-in feature importance (Gini importance) tends to favour high-cardinality features (many unique values) because there are more possible split points. Use permutation importance for a more reliable, unbiased estimate.
How many features should I keep?
Use RFECV to find the optimal number automatically, or look at the diminishing returns in the permutation importance ranking. As a rule of thumb, start with sqrt(n_features) and increase if the model underfits.
Should I do feature selection before or after hyperparameter tuning?
Ideally, feature selection and hyperparameter tuning should be done together (nested cross-validation). In practice, do feature selection first with a default model, then tune hyperparameters on the reduced feature set.