MLOps
Tracking experiments, CI/CD for ML, and monitoring
Skills taught: ml.mlops, ml.experiment-tracking, ml.mlflow, ml.model-monitoring
What is MLOps?
MLOps (ML + DevOps) is the practice of applying software engineering discipline to machine learning systems. It covers experiment tracking, data versioning, automated testing, CI/CD pipelines for model deployment, and monitoring in production.
- Reproducibility — anyone can recreate any past experiment exactly
- Automation — retraining and deployment happen without manual steps
- Monitoring — production models are watched for quality degradation
- Collaboration — team members can share, compare, and build on each other's work
- Auditability — full history of which data trained which model in production
Experiment Tracking with MLflow
MLflow records every experiment run: parameters used, metrics achieved, and model artefacts saved. The MLflow UI lets you compare runs visually, pick the best model, and register it for deployment.
import mlflow
import mlflow.sklearn
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import f1_score, roc_auc_score
import pandas as pd
mlflow.set_experiment('churn-prediction') # group runs by experiment
params = {
'n_estimators': 200,
'max_depth': 10,
'min_samples_leaf': 5
}
with mlflow.start_run(run_name='rf-baseline'):
# Log parameters
mlflow.log_params(params)
mlflow.log_param('training_rows', len(X_train))
# Train
model = RandomForestClassifier(**params, random_state=42)
model.fit(X_train, y_train)
# Evaluate and log metrics
preds = model.predict(X_test)
probs = model.predict_proba(X_test)[:, 1]
metrics = {
'f1': f1_score(y_test, preds),
'auc': roc_auc_score(y_test, probs),
'accuracy': model.score(X_test, y_test)
}
mlflow.log_metrics(metrics)
# Save model
mlflow.sklearn.log_model(model, 'model')
print(f'F1: {metrics["f1"]:.3f} AUC: {metrics["auc"]:.3f}')
# View all runs: mlflow ui (opens at http://localhost:5000)
Model Registry and Versioning
import mlflow
from mlflow.tracking import MlflowClient
client = MlflowClient()
# Register a model from a run
run_id = 'your-run-id'
mlflow.register_model(
model_uri=f'runs:/{run_id}/model',
name='churn-classifier'
)
# Promote to 'Staging' after review
client.transition_model_version_stage(
name='churn-classifier',
version=1,
stage='Staging'
)
# Load from registry for serving
model_uri = 'models:/churn-classifier/Production'
model = mlflow.sklearn.load_model(model_uri)
print(f'Loaded from Production registry')
preds = model.predict(X_new)
Data Versioning with DVC
# DVC commands (run in terminal, not Python)
# pip install dvc
# Initialise DVC in a git repo
# dvc init
# Track a dataset file
# dvc add data/train.csv
# git add data/train.csv.dvc .gitignore
# git commit -m 'Track training data with DVC'
# Store data remotely (S3, GCS, local)
# dvc remote add -d myremote s3://my-bucket/dvc
# dvc push
# Later — reproduce the exact dataset used in a past commit
# git checkout abc1234
# dvc pull
# (now data/train.csv is the exact version from that commit)
# DVC pipeline — define stages as code
# dvc run -n train \
# -d src/train.py -d data/train.csv \
# -o models/model.joblib \
# python src/train.py
print('DVC commands should be run in a terminal')
Production Monitoring
Models degrade over time as the world changes. Data drift means the input distribution has changed. Concept drift means the relationship between inputs and the target has changed. Both require retraining.
from evidently import ColumnMapping
from evidently.report import Report
from evidently.metric_preset import DataDriftPreset, ClassificationPreset
import pandas as pd
# Load reference data (what training data looked like)
reference_df = pd.read_parquet('data/train.parquet')
# Load current production data (last week of predictions)
current_df = pd.read_parquet('data/production_last_week.parquet')
# Data drift report
report = Report(metrics=[DataDriftPreset(), ClassificationPreset()])
report.run(
reference_data=reference_df,
current_data=current_df,
column_mapping=ColumnMapping(target='churn', prediction='prediction')
)
report.save_html('drift_report.html')
# Check for drift programmatically
result = report.as_dict()
drift_detected = result['metrics'][0]['result']['dataset_drift']
if drift_detected:
print('DATA DRIFT DETECTED — consider retraining')
- MLflow
- Open-source platform for tracking ML experiments, packaging code reproducibly, and deploying models. Provides a UI for comparing runs and a model registry for managing production versions.
- Feature Store
- A centralised repository for computed features that ensures consistency between training and serving. Prevents training-serving skew — the same features computed the same way. Tools: Feast, Hopsworks.
Tip: Version your training data as well as your code. A model is only reproducible if you can recreate the exact dataset used to train it. Use DVC to store data versioning metadata in git.
Frequently Asked Questions
What is a feature store?
A centralised repository for computed features that ensures consistency between training and serving. Training-serving skew (features computed differently at serving time vs training time) is a major cause of production ML failures. Tools include Feast (open-source) and Hopsworks.
When should I retrain my model?
Retrain when: (1) monitored metrics drop below a threshold, (2) data drift is detected, (3) new labelled data is available, or (4) on a fixed schedule (weekly, monthly). Automated retraining pipelines trigger on these conditions automatically.
What is the difference between DVC and MLflow?
DVC focuses on data and pipeline versioning — it tracks which dataset and code produced which model, integrated with git. MLflow focuses on experiment tracking and model management — logging parameters, metrics, and comparing runs. They complement each other well.