Unsupervised Learning & Clustering
K-means, DBSCAN, PCA, and finding hidden structure
Skills taught: ml.clustering, ml.kmeans, ml.pca, ml.dimensionality-reduction
Unsupervised Learning
Unsupervised learning finds hidden structure in unlabelled data. There is no target variable — the algorithm discovers patterns on its own. The two most important techniques are clustering (grouping similar points) and dimensionality reduction (compressing high-dimensional data).
K-Means Clustering
K-means partitions data into K clusters by minimising within-cluster variance. Each point belongs to the cluster with the nearest centroid. The algorithm iterates: assign → recompute centroids → reassign — until convergence.
from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import silhouette_score
import numpy as np
import matplotlib.pyplot as plt
# Always scale before clustering
scaler = StandardScaler()
X_sc = scaler.fit_transform(X)
# Fit K-means
kmeans = KMeans(n_clusters=4, init='k-means++', n_init=10, random_state=42)
labels = kmeans.fit_predict(X_sc)
print(f'Inertia (within-cluster SS): {kmeans.inertia_:.2f}')
print(f'Silhouette score: {silhouette_score(X_sc, labels):.3f}') # -1 to 1, higher is better
print(f'Cluster sizes: {np.bincount(labels)}')
Choosing K: The Elbow Method
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
import matplotlib.pyplot as plt
inertias = []
silhouettes = []
K_range = range(2, 12)
for k in K_range:
km = KMeans(n_clusters=k, init='k-means++', n_init=10, random_state=42)
labels = km.fit_predict(X_sc)
inertias.append(km.inertia_)
silhouettes.append(silhouette_score(X_sc, labels))
# Plot elbow — look for the 'kink'
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))
ax1.plot(K_range, inertias, 'bx-')
ax1.set_xlabel('K')
ax1.set_ylabel('Inertia')
ax1.set_title('Elbow Method')
ax2.plot(K_range, silhouettes, 'rx-')
ax2.set_xlabel('K')
ax2.set_ylabel('Silhouette Score')
ax2.set_title('Silhouette Analysis')
plt.tight_layout()
plt.show()
best_k = K_range[np.argmax(silhouettes)]
print(f'Best K by silhouette: {best_k}')
DBSCAN: Density-Based Clustering
DBSCAN finds clusters as dense regions separated by low-density areas. It doesn't require specifying K in advance. Points in sparse regions are labelled as outliers (label = -1). It handles arbitrary cluster shapes that K-means cannot.
from sklearn.cluster import DBSCAN
from sklearn.neighbors import NearestNeighbors
import numpy as np
import matplotlib.pyplot as plt
# Find good eps using k-NN distance plot
k = 4
nbrs = NearestNeighbors(n_neighbors=k).fit(X_sc)
distances, _ = nbrs.kneighbors(X_sc)
distances = np.sort(distances[:, k-1]) # k-th nearest neighbour distance
# Look for the 'knee' in this plot — that's your eps
plt.plot(distances)
plt.xlabel('Points (sorted)')
plt.ylabel(f'{k}-NN distance')
plt.title('K-NN Distance Plot (find the knee for eps)')
plt.show()
# Fit DBSCAN
db = DBSCAN(eps=0.5, min_samples=5)
labels = db.fit_predict(X_sc)
n_clusters = len(set(labels)) - (1 if -1 in labels else 0)
n_outliers = (labels == -1).sum()
print(f'Found {n_clusters} clusters, {n_outliers} outliers')
PCA: Dimensionality Reduction
Principal Component Analysis projects high-dimensional data into fewer dimensions while retaining as much variance as possible. Use it for visualisation (reduce to 2D/3D), noise removal, or to speed up downstream models.
from sklearn.decomposition import PCA
import matplotlib.pyplot as plt
import numpy as np
# Fit PCA
pca = PCA(n_components=0.95) # keep 95% of variance
X_pca = pca.fit_transform(X_sc)
print(f'Reduced from {X_sc.shape[1]} to {pca.n_components_} components')
print(f'Explained variance ratio: {pca.explained_variance_ratio_.cumsum()[-1]:.3f}')
# Visualise in 2D
pca2d = PCA(n_components=2)
X_2d = pca2d.fit_transform(X_sc)
plt.figure(figsize=(8, 6))
plt.scatter(X_2d[:, 0], X_2d[:, 1], c=labels, cmap='tab10', alpha=0.6, s=10)
plt.xlabel(f'PC1 ({pca2d.explained_variance_ratio_[0]*100:.1f}% variance)')
plt.ylabel(f'PC2 ({pca2d.explained_variance_ratio_[1]*100:.1f}% variance)')
plt.title('PCA Projection')
plt.colorbar(label='Cluster')
plt.show()
K-Means vs DBSCAN
- K-Means: Partition-based — assigns every point to exactly one of K clusters
- DBSCAN: Density-based — groups dense regions, marks sparse points as outliers
Tip: Always standardise features before clustering. K-means uses Euclidean distance — features with large numeric ranges will dominate the distance calculation if not scaled.
Frequently Asked Questions
How do I evaluate clustering quality without labels?
Use the Silhouette Score (how similar a point is to its cluster vs other clusters — values near 1.0 are good) or the Davies-Bouldin Index (lower is better). The Elbow Method uses inertia (within-cluster sum of squares) to suggest a good K.
Should I use PCA before clustering?
Often yes for high-dimensional data. PCA removes noise dimensions and speeds up K-means. Keep enough components to explain 90-95% of variance (n_components=0.95). Don't use PCA if the number of features is small (< 20).
What is the difference between PCA and t-SNE?
PCA is a linear transform that preserves global variance — it's fast and reproducible. t-SNE is a non-linear transform that preserves local neighbourhood structure — it's better for visualisation but very slow on large datasets and results depend on the perplexity parameter.