Cross-Validation
K-fold, stratified folds, and avoiding data leakage
Skills taught: ml.cross-validation, ml.hyperparameter-tuning, ml.pipeline
Why Cross-Validate?
A single train/test split is unreliable. If your test set happens to contain easy examples, you overestimate performance. If it contains hard ones, you underestimate. Cross-validation evaluates the model on multiple independent splits and averages the results for a fairer estimate.
Basic K-Fold Cross-Validation
from sklearn.model_selection import cross_val_score, KFold, StratifiedKFold
from sklearn.ensemble import RandomForestClassifier
import numpy as np
model = RandomForestClassifier(n_estimators=100, random_state=42)
# Simple 5-fold (not stratified)
kf = KFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X, y, cv=kf, scoring='f1')
print(f'F1 (5-fold): {scores.mean():.3f} ± {scores.std():.3f}')
# Stratified — preserves class ratios in each fold (use for classification)
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
st_scores = cross_val_score(model, X, y, cv=skf, scoring='f1')
print(f'F1 (stratified): {st_scores.mean():.3f} ± {st_scores.std():.3f}')
Using Pipelines to Prevent Leakage
If you scale the data before cross-validation, the scaler sees the test fold's statistics — this is data leakage. Using a Pipeline ensures that preprocessing is re-fit inside each fold on the training data only.
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, StratifiedKFold
# Pipeline: scaler is fit inside each CV fold, preventing leakage
pipe = Pipeline([
('scaler', StandardScaler()),
('model', LogisticRegression(max_iter=1000))
])
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(pipe, X, y, cv=skf, scoring='roc_auc')
print(f'AUC-ROC: {scores.mean():.3f} ± {scores.std():.3f}')
Hyperparameter Tuning with GridSearchCV
from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
pipe = Pipeline([
('scaler', StandardScaler()),
('rf', RandomForestClassifier(random_state=42))
])
param_grid = {
'rf__n_estimators': [50, 100, 200],
'rf__max_depth': [5, 10, None],
'rf__min_samples_leaf': [1, 5, 10]
}
search = GridSearchCV(
pipe,
param_grid,
cv=StratifiedKFold(5, shuffle=True, random_state=42),
scoring='f1',
n_jobs=-1, # use all CPU cores
verbose=1
)
search.fit(X_train, y_train)
print(f'Best params: {search.best_params_}')
print(f'Best CV F1: {search.best_score_:.3f}')
print(f'Test F1: {f1_score(y_test, search.predict(X_test)):.3f}')
RandomizedSearchCV for Large Grids
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint, uniform
# When the grid is too large for exhaustive search
param_dist = {
'rf__n_estimators': randint(50, 500),
'rf__max_depth': [5, 10, 20, None],
'rf__min_samples_leaf': randint(1, 20),
'rf__max_features': uniform(0.3, 0.7) # fraction of features
}
rand_search = RandomizedSearchCV(
pipe,
param_dist,
n_iter=50, # sample 50 random combinations
cv=5,
scoring='f1',
random_state=42,
n_jobs=-1
)
rand_search.fit(X_train, y_train)
print(f'Best CV F1: {rand_search.best_score_:.3f}')
Tip: Always use a Pipeline when cross-validating. Any step that fits from data (scaler, imputer, encoder) must be inside the pipeline so it only sees training folds, not the test fold.
Warning: Don't use the test set for hyperparameter selection. Use cross-validation on the training set to pick hyperparameters, then evaluate on the test set exactly once at the end. Using the test set for tuning inflates your performance estimate.
Frequently Asked Questions
What is a Pipeline in scikit-learn?
A Pipeline chains preprocessing steps and the model together so they can be fitted and cross-validated as a single unit. This prevents data leakage by ensuring preprocessing is only fitted on training folds.
How many folds should I use?
5 or 10 folds are the standard. More folds give a lower-bias estimate but take longer. Use 10 for small datasets where you need more training data per fold. For very large datasets, 5 or even 3 folds may be sufficient.
What is the difference between GridSearchCV and RandomizedSearchCV?
GridSearchCV tries every combination of hyperparameters exhaustively — good when the grid is small. RandomizedSearchCV samples a fixed number of random combinations — faster for large grids, and often finds near-optimal settings with fewer trials.