Evaluation Metrics
Accuracy, precision, recall, F1, AUC-ROC, and MSE
Skills taught: ml.metrics, ml.precision-recall, ml.roc
Choosing the Right Metric
Accuracy alone is misleading on imbalanced datasets. On a dataset where 99% of transactions are legitimate, a model predicting 'not fraud' always has 99% accuracy — but zero recall for fraud. Use precision, recall, and F1 for classification; RMSE or MAE for regression.
Classification Metrics: Precision, Recall, F1
from sklearn.metrics import (
accuracy_score, precision_score, recall_score,
f1_score, roc_auc_score, classification_report,
confusion_matrix, ConfusionMatrixDisplay
)
import matplotlib.pyplot as plt
# Full report with all per-class metrics
print(classification_report(y_test, preds, target_names=['Not Fraud', 'Fraud']))
# Individual metrics
print(f'Accuracy: {accuracy_score(y_test, preds):.3f}')
print(f'Precision: {precision_score(y_test, preds):.3f}') # TP / (TP+FP)
print(f'Recall: {recall_score(y_test, preds):.3f}') # TP / (TP+FN)
print(f'F1 Score: {f1_score(y_test, preds):.3f}') # harmonic mean of P & R
print(f'AUC-ROC: {roc_auc_score(y_test, probs):.3f}') # needs probabilities
- Precision
- Of all positive predictions the model made (TP + FP), what fraction were actually positive (TP)? High precision means few false alarms. Precision = TP / (TP + FP).
- Recall (Sensitivity, TPR)
- Of all actual positives (TP + FN), what fraction did the model find (TP)? High recall means few missed cases. Recall = TP / (TP + FN).
- F1 Score
- The harmonic mean of precision and recall. Useful single metric when you need to balance both. F1 = 2 × (Precision × Recall) / (Precision + Recall).
The Precision-Recall Trade-off
Lowering the classification threshold catches more positives (higher recall) but also flags more false alarms (lower precision). The right balance depends on your use case.
from sklearn.metrics import precision_recall_curve, average_precision_score
import matplotlib.pyplot as plt
precisions, recalls, thresholds = precision_recall_curve(y_test, probs)
ap = average_precision_score(y_test, probs)
plt.figure(figsize=(8, 5))
plt.plot(recalls, precisions, lw=2)
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.title(f'Precision-Recall Curve (AP = {ap:.3f})')
plt.grid(True)
plt.show()
# Find threshold that maximises F1
f1_scores = 2 * precisions[:-1] * recalls[:-1] / (precisions[:-1] + recalls[:-1] + 1e-9)
best_idx = f1_scores.argmax()
print(f'Best threshold: {thresholds[best_idx]:.3f}')
print(f'At that threshold: P={precisions[best_idx]:.3f} R={recalls[best_idx]:.3f}')
AUC-ROC: Measuring Discrimination
from sklearn.metrics import roc_curve, roc_auc_score
import matplotlib.pyplot as plt
fpr, tpr, thresholds = roc_curve(y_test, probs)
auc = roc_auc_score(y_test, probs)
plt.figure(figsize=(7, 6))
plt.plot(fpr, tpr, lw=2, label=f'ROC curve (AUC = {auc:.3f})')
plt.plot([0, 1], [0, 1], 'k--', label='Random classifier (AUC = 0.5)')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate (Recall)')
plt.title('ROC Curve')
plt.legend()
plt.grid(True)
plt.show()
Regression Metrics
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
import numpy as np
rmse = np.sqrt(mean_squared_error(y_test, preds))
mae = mean_absolute_error(y_test, preds)
r2 = r2_score(y_test, preds)
mape = np.mean(np.abs((y_test - preds) / y_test)) * 100 # % error
print(f'RMSE: {rmse:.2f} (penalises large errors more)')
print(f'MAE: {mae:.2f} (robust to outliers)')
print(f'R²: {r2:.3f} (1 = perfect, 0 = predicts mean)')
print(f'MAPE: {mape:.1f}% (percentage error, scale-independent)')
Tip: For cancer detection, missing a true case (low recall) is more dangerous than a false alarm. For spam filtering, sending important emails to spam (low precision) is more annoying than missing some spam. Choose your primary metric based on the cost of each error type.
Frequently Asked Questions
What is AUC-ROC?
The Area Under the ROC (Receiver Operating Characteristic) Curve. The ROC curve plots True Positive Rate vs False Positive Rate at every classification threshold. AUC summarises this — 1.0 is perfect, 0.5 is no better than random.
When should I use PR-AUC instead of ROC-AUC?
PR-AUC (Precision-Recall AUC) is more informative than ROC-AUC on heavily imbalanced datasets. ROC-AUC can look optimistic when the negative class dominates; PR-AUC focuses on how well you find the rare positive class.
How do I choose between RMSE and MAE for regression?
If large errors are especially costly (a prediction off by 10x is much worse than off by 2x), use RMSE — it penalises large errors more. If outliers shouldn't dominate your evaluation, use MAE.