Data Preparation
Cleaning, normalising, handling missing values and outliers
Skills taught: ml.data-prep, ml.imputation, ml.scaling
Data is Everything
Garbage in, garbage out. Most ML engineers spend 60–80% of their project time on data, not modelling. A clean dataset with a simple model almost always beats a messy dataset with a sophisticated one.
Step 1: Explore the Data First
import pandas as pd
import numpy as np
df = pd.read_csv('data.csv')
# Shape and types
print(df.shape) # (rows, cols)
print(df.dtypes) # column data types
# Missing value counts
print(df.isnull().sum())
# Basic statistics
print(df.describe())
# Duplicate rows
print(f'Duplicates: {df.duplicated().sum()}')
Step 2: Handle Missing Values
Three strategies: drop rows/columns with missing values (safe only if the fraction is small), impute with a statistic (median for numeric, most-frequent for categorical), or flag the missingness as a new binary feature.
from sklearn.impute import SimpleImputer
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Impute numeric columns with median
num_imputer = SimpleImputer(strategy='median')
X_train_num = num_imputer.fit_transform(X_train[numeric_cols])
X_test_num = num_imputer.transform(X_test[numeric_cols]) # transform only!
# Impute categoricals with most frequent value
cat_imputer = SimpleImputer(strategy='most_frequent')
X_train_cat = cat_imputer.fit_transform(X_train[cat_cols])
X_test_cat = cat_imputer.transform(X_test[cat_cols])
Warning: Always call fit_transform() on training data, then transform() (not fit_transform()) on test data. Fitting on test data leaks information and inflates your performance metrics.
Step 3: Encode Categorical Features
Most ML algorithms require numeric inputs. One-hot encoding works for unordered categories (city, colour). Ordinal encoding works for ordered categories (small < medium < large). pd.get_dummies() is a quick option for small cardinality.
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder
import pandas as pd
# One-hot (creates a column per category)
ohe = OneHotEncoder(handle_unknown='ignore', sparse_output=False)
city_encoded = ohe.fit_transform(X_train[['city']])
# Ordinal (preserves order)
ordinal = OrdinalEncoder(categories=[['small','medium','large']])
size_encoded = ordinal.fit_transform(X_train[['size']])
# Quick option with pandas
X_dummies = pd.get_dummies(X_train, columns=['city'], drop_first=True)
Step 4: Scale Numeric Features
StandardScaler (mean=0, std=1) works best for linear models and neural networks. MinMaxScaler (0 to 1) is useful when you need a bounded range. Tree-based models (Random Forest, XGBoost) don't need scaling.
from sklearn.preprocessing import StandardScaler, MinMaxScaler
import numpy as np
# Standard scaling (z-score)
scaler = StandardScaler()
X_train_sc = scaler.fit_transform(X_train)
X_test_sc = scaler.transform(X_test)
# Log-transform skewed feature before scaling
X_train['log_income'] = np.log1p(X_train['income']) # log1p handles zeros
print(f'Mean: {X_train_sc.mean(axis=0).round(3)}')
print(f'Std: {X_train_sc.std(axis=0).round(3)}')
Tip: For skewed features (like income or house price), apply a log transform before scaling. This compresses extreme values and helps linear models converge faster.
The full preprocessing pipeline can be built with sklearn.pipeline.Pipeline and ColumnTransformer, which apply different transforms to different columns automatically.
Frequently Asked Questions
Should I remove all rows with missing values?
Only if the fraction is very small (< 1–2%). Otherwise impute. Dropping many rows reduces your effective dataset size and can introduce bias if the missingness is not random.
What is the difference between StandardScaler and MinMaxScaler?
StandardScaler converts to zero mean and unit variance (unbounded range). MinMaxScaler maps to [0, 1] but is sensitive to outliers. Use StandardScaler for linear models and neural networks; MinMaxScaler for algorithms that need a bounded range like k-NN.
What is data leakage?
Data leakage occurs when information from the test set (or future data) is used during training. Common causes: fitting scalers on the full dataset, including future-derived features, or not splitting before preprocessing.