Ensemble Methods
Bagging, boosting, Random Forest, and XGBoost
Skills taught: ml.ensembles, ml.random-forest, ml.gradient-boosting, ml.xgboost
Ensemble Learning
Ensembles combine multiple models to outperform any single model. The key insight: if models make independent errors, averaging their predictions cancels errors out. The two main strategies are bagging (parallel, reduces variance) and boosting (sequential, reduces bias).
Bagging: Random Forest
Random Forest trains hundreds of decision trees, each on a random bootstrap sample of data and a random subset of features. Final predictions are made by majority vote (classification) or averaging (regression). The randomness makes trees independent, so their errors cancel.
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score, StratifiedKFold
import numpy as np
import pandas as pd
rf = RandomForestClassifier(
n_estimators=200, # number of trees
max_depth=None, # let trees grow deep (bagging handles variance)
min_samples_leaf=2,
max_features='sqrt', # sqrt(n_features) per tree — key for decorrelation
bootstrap=True, # sample with replacement
n_jobs=-1, # parallel training
random_state=42
)
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(rf, X, y, cv=skf, scoring='f1')
print(f'F1: {scores.mean():.3f} ± {scores.std():.3f}')
# Feature importances
rf.fit(X_train, y_train)
importances = pd.Series(rf.feature_importances_, index=X.columns)
print(importances.sort_values(ascending=False).head(10))
Boosting: XGBoost
XGBoost builds trees sequentially: each new tree focuses on the residual errors of the previous ones. It uses gradient descent in function space, which is why it's called 'gradient boosting'. XGBoost adds hardware optimisations and regularisation on top.
from xgboost import XGBClassifier
from sklearn.model_selection import cross_val_score, StratifiedKFold
xgb = XGBClassifier(
n_estimators=500,
learning_rate=0.05, # smaller = more trees needed but better generalisation
max_depth=6,
subsample=0.8, # row sampling per tree
colsample_bytree=0.8, # feature sampling per tree
reg_alpha=0.1, # L1 regularisation
reg_lambda=1.0, # L2 regularisation
eval_metric='logloss',
early_stopping_rounds=20,
random_state=42,
n_jobs=-1
)
# Train with early stopping
xgb.fit(
X_train, y_train,
eval_set=[(X_val, y_val)],
verbose=50
)
print(f'Best iteration: {xgb.best_iteration}')
print(f'Test F1: {f1_score(y_test, xgb.predict(X_test)):.3f}')
Stacking: Blending Models
from sklearn.ensemble import StackingClassifier, RandomForestClassifier, GradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
# Level 0: diverse base models
base_models = [
('rf', RandomForestClassifier(n_estimators=100, random_state=42)),
('gb', GradientBoostingClassifier(n_estimators=100, random_state=42)),
('svm', SVC(probability=True, kernel='rbf'))
]
# Level 1: meta-learner combines base model predictions
stacking = StackingClassifier(
estimators=base_models,
final_estimator=LogisticRegression(),
cv=5, # use CV to generate training predictions for meta-learner
stack_method='predict_proba'
)
stacking.fit(X_train, y_train)
print(f'Stacking F1: {f1_score(y_test, stacking.predict(X_test)):.3f}')
Random Forest (Bagging) vs XGBoost (Boosting)
- Random Forest: Parallel ensemble — trains trees independently and averages
- XGBoost / LightGBM: Sequential ensemble — each tree corrects previous errors
Tip: XGBoost and LightGBM win most structured-data Kaggle competitions. Use them as a strong baseline. But start with Random Forest first — it requires almost no tuning and gives a good performance floor.
Frequently Asked Questions
When should I use boosting vs bagging?
Bagging (Random Forest) works well out of the box and is hard to overfit — good for a quick, reliable baseline. Boosting (XGBoost) is usually more accurate but needs careful tuning of learning_rate, n_estimators, and max_depth.
Do I need to scale features for Random Forest or XGBoost?
No. Both are tree-based algorithms that split on feature thresholds — the absolute scale doesn't matter. This is a big advantage over linear models and k-NN.
What is the difference between XGBoost and LightGBM?
Both are gradient boosting frameworks. LightGBM is significantly faster on large datasets because it grows trees leaf-wise (rather than level-wise) and uses histogram-based splitting. XGBoost is more established and sometimes more stable on small datasets.