Linear Regression
Fitting a line, coefficients, and predicting continuous values
Skills taught: ml.regression, ml.linear-models, ml.model-interpretation
Linear Regression
Linear regression models the relationship between inputs and a continuous output as a weighted sum of features. The model learns a coefficient (weight) for each feature and an intercept.
The prediction formula is: ŷ = β₀ + β₁x₁ + β₂x₂ + ... + βₙxₙ. Training finds the β values that minimise the difference between predicted ŷ and actual y.
Training and Evaluating a Linear Regression Model
import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
from sklearn.preprocessing import StandardScaler
# Load data
df = pd.read_csv('houses.csv')
X = df[['sqft', 'bedrooms', 'bathrooms', 'age']]
y = df['price']
# Split and scale
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
scaler = StandardScaler()
X_train_sc = scaler.fit_transform(X_train)
X_test_sc = scaler.transform(X_test)
# Train
model = LinearRegression()
model.fit(X_train_sc, y_train)
# Predict
preds = model.predict(X_test_sc)
# Evaluate
rmse = np.sqrt(mean_squared_error(y_test, preds))
mae = mean_absolute_error(y_test, preds)
r2 = r2_score(y_test, preds)
print(f'RMSE: {rmse:,.0f}') # error in same units as target
print(f'MAE: {mae:,.0f}') # average absolute error
print(f'R²: {r2:.3f}') # 1.0 = perfect, 0 = predicts mean
Understanding Coefficients
# After fitting with scaled features
coef_df = pd.DataFrame({
'feature': X.columns,
'coefficient': model.coef_
}).sort_values('coefficient', ascending=False)
print(coef_df)
print(f'Intercept: {model.intercept_:,.0f}')
# A positive coefficient means increasing that feature raises the predicted price.
# With scaled features, coefficient magnitude tells you relative importance.
- RMSE
- Root Mean Squared Error — the square root of the average squared difference between predictions and actual values. In the same units as the target. Penalises large errors more than MAE.
- R² (R-squared)
- The proportion of variance in the target explained by the model. R²=1 means perfect prediction; R²=0 means the model is no better than predicting the mean; R²<0 means worse than predicting the mean.
RMSE vs MAE
- RMSE: Root Mean Squared Error — penalises large errors more
- MAE: Mean Absolute Error — treats all errors equally
Tip: Always scale features before linear regression. Without scaling, features with large numeric ranges (like income in thousands) will dominate features on small scales (like number of bedrooms), making coefficients misleading.
Warning: Linear regression assumes the relationship is linear. Check a residual plot — if residuals show a curved pattern, the model is underfitting a non-linear relationship. Try polynomial features or a tree-based model.
Assumptions to Check
- Linearity — the relationship between X and y is linear
- Homoscedasticity — residuals have constant variance (no funnel shape in residual plot)
- No multicollinearity — features are not highly correlated with each other
- No significant outliers — extreme values can distort the fitted line
Frequently Asked Questions
When is linear regression not appropriate?
When the relationship is strongly non-linear, when the target is categorical (use logistic regression or classifiers), or when there are strong interactions between features that you haven't manually added.
What is the difference between linear regression and logistic regression?
Linear regression predicts a continuous value. Logistic regression uses the same linear combination but passes it through a sigmoid function to output a probability for binary classification.
Why is RMSE in the same units as the target?
RMSE takes the square root of the mean squared error, bringing it back to the original units. This makes it directly interpretable — an RMSE of 10,000 means typical predictions are off by £10,000.