Classification
Logistic regression, decision boundaries, and class probabilities
Skills taught: ml.classification, ml.logistic-regression, ml.class-imbalance
Classification vs Regression
Classification predicts a discrete label (spam/not-spam, cat/dog, disease/healthy). Regression predicts a continuous number. The key difference: classification boundaries divide the feature space into regions; regression fits a continuous surface.
Logistic Regression
Despite the name, logistic regression is a classifier. It computes a linear combination of features, passes it through the sigmoid function to get a probability, then applies a threshold (default 0.5) to assign a class label.
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
import numpy as np
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,
stratify=y, random_state=42)
scaler = StandardScaler()
X_train_sc = scaler.fit_transform(X_train)
X_test_sc = scaler.transform(X_test)
model = LogisticRegression(max_iter=1000, C=1.0)
model.fit(X_train_sc, y_train)
preds = model.predict(X_test_sc)
probs = model.predict_proba(X_test_sc)[:, 1] # probability of class 1
print(classification_report(y_test, preds))
print(confusion_matrix(y_test, preds))
Understanding the Confusion Matrix
from sklearn.metrics import ConfusionMatrixDisplay
import matplotlib.pyplot as plt
# confusion matrix: rows = actual, cols = predicted
# Predicted 0 Predicted 1
# Actual 0 TN FP
# Actual 1 FN TP
cm = confusion_matrix(y_test, preds)
TN, FP, FN, TP = cm.ravel()
print(f'True Positives: {TP} (correctly identified positives)')
print(f'True Negatives: {TN} (correctly identified negatives)')
print(f'False Positives: {FP} (false alarms)')
print(f'False Negatives: {FN} (missed positives)')
# Visualise
fig, ax = plt.subplots()
ConfusionMatrixDisplay(cm).plot(ax=ax)
plt.show()
Adjusting the Decision Threshold
The default threshold of 0.5 is not always optimal. In fraud detection, you want to catch more fraud (high recall) even if it means more false alarms. In medical screening, you must minimise missed cases. Adjust the threshold based on the relative cost of false positives vs false negatives.
import numpy as np
from sklearn.metrics import precision_recall_curve
# Try different thresholds
for threshold in [0.3, 0.4, 0.5, 0.6, 0.7]:
preds_at_t = (probs >= threshold).astype(int)
tp = ((preds_at_t == 1) & (y_test == 1)).sum()
fp = ((preds_at_t == 1) & (y_test == 0)).sum()
fn = ((preds_at_t == 0) & (y_test == 1)).sum()
precision = tp / (tp + fp + 1e-9)
recall = tp / (tp + fn + 1e-9)
print(f'Threshold {threshold:.1f}: Precision={precision:.3f} Recall={recall:.3f}')
Handling Class Imbalance
When one class is rare (e.g., 1% fraud in a dataset), a model that predicts 'not fraud' every time achieves 99% accuracy but is useless. Use class_weight='balanced' to give the minority class more influence during training.
from sklearn.linear_model import LogisticRegression
from imblearn.over_sampling import SMOTE
# Option 1: class_weight='balanced' (simplest)
model = LogisticRegression(class_weight='balanced', max_iter=1000)
model.fit(X_train_sc, y_train)
# Option 2: SMOTE oversampling (synthesises minority class examples)
smote = SMOTE(random_state=42)
X_res, y_res = smote.fit_resample(X_train_sc, y_train)
model.fit(X_res, y_res)
Tip: On imbalanced datasets, accuracy is a misleading metric. Use precision, recall, F1, or AUC-ROC instead. These metrics measure how well you find the minority class.
Frequently Asked Questions
How do I handle imbalanced classes?
Use class_weight='balanced' in sklearn to give more weight to the minority class during training. For more severe imbalance, try oversampling (SMOTE) or undersampling. Always evaluate using F1, precision-recall, or AUC-ROC rather than accuracy.
What is the difference between binary and multi-class classification?
Binary classification has two classes (spam/not-spam). Multi-class has 3+ classes (cat/dog/bird). Logistic regression handles multi-class with one-vs-rest (OvR) or softmax (multinomial) strategies — set multi_class='ovr' or 'multinomial' in sklearn.
What is the C parameter in LogisticRegression?
C is the inverse regularisation strength (1/λ). A smaller C applies stronger regularisation (simpler model). A larger C allows the model to fit the training data more closely. Default is C=1.0. Tune with cross-validation.