Decision Trees
Splits, depth, and tree-based models
Skills taught: ml.trees, ml.decision-tree, ml.interpretability
How Decision Trees Work
A decision tree learns a sequence of if-then-else rules. At each node, it asks a question about one feature (e.g., age > 30?). The answer routes the sample left or right. At a leaf node, it makes a prediction. The goal is to split data so each leaf is as pure as possible.
Training a Decision Tree
from sklearn.tree import DecisionTreeClassifier, export_text, plot_tree
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
import matplotlib.pyplot as plt
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,
stratify=y, random_state=42)
# Shallow tree = interpretable, less likely to overfit
model = DecisionTreeClassifier(
max_depth=5, # limit depth to prevent overfitting
min_samples_leaf=10, # each leaf needs at least 10 samples
criterion='gini', # 'gini' or 'entropy'
random_state=42
)
model.fit(X_train, y_train)
preds = model.predict(X_test)
print(f'Accuracy: {model.score(X_test, y_test):.3f}')
print(classification_report(y_test, preds))
# Print the tree as text rules
rules = export_text(model, feature_names=list(X.columns))
print(rules[:1000]) # first 1000 chars
Visualising the Tree
import matplotlib.pyplot as plt
from sklearn.tree import plot_tree
fig, ax = plt.subplots(figsize=(20, 10))
plot_tree(
model,
feature_names=list(X.columns),
class_names=['No', 'Yes'],
filled=True, # colour nodes by class
rounded=True,
ax=ax
)
plt.savefig('tree.png', dpi=150, bbox_inches='tight')
plt.show()
# Feature importances
importances = model.feature_importances_
for feat, imp in sorted(zip(X.columns, importances), key=lambda x: -x[1]):
print(f'{feat:20s}: {imp:.3f}')
- Gini Impurity
- A measure of how often a randomly chosen element would be incorrectly classified. Gini=0 means a node is perfectly pure (all one class). It's computed as 1 - Σ(pᵢ²) over all classes.
- Information Gain
- The reduction in entropy (uncertainty) achieved by splitting on a feature. The tree selects the feature and threshold that maximises information gain at each node.
Overfitting and Pruning
An unconstrained decision tree will grow until every leaf contains a single sample — perfectly classifying training data but failing completely on new data. This is overfitting.
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
# Compare different depths
results = []
for depth in range(1, 20):
dt = DecisionTreeClassifier(max_depth=depth, random_state=42)
cv_scores = cross_val_score(dt, X, y, cv=5, scoring='f1')
dt.fit(X_train, y_train)
results.append({
'depth': depth,
'train_score': dt.score(X_train, y_train),
'cv_score': cv_scores.mean()
})
# The gap between train_score and cv_score reveals overfitting
for r in results:
print(f"depth={r['depth']:2d} train={r['train_score']:.3f} cv={r['cv_score']:.3f}")
Decision Tree vs Random Forest
- Decision Tree: Single tree — interpretable but high variance
- Random Forest: Ensemble of trees — accurate but less interpretable
Tip: Unconstrained trees overfit. Use max_depth=5 or min_samples_leaf=10 as starting hyperparameters. Tune with cross-validation.
Frequently Asked Questions
What is the difference between a decision tree and a random forest?
A random forest is an ensemble of many decision trees. Each tree is trained on a random subset of data (bootstrap) and features. Final predictions are made by majority vote or averaging. This dramatically reduces variance and overfitting compared to a single tree.
Should I use Gini impurity or entropy?
In practice the results are nearly identical. Gini is slightly faster to compute. Use Gini unless you have a specific reason to prefer entropy.
Do decision trees need feature scaling?
No. Decision trees split on individual features one at a time, so the absolute scale of features doesn't matter. This is a key advantage over linear models and k-NN.