Neural Networks
Perceptrons, backpropagation, and deep learning with Keras
Skills taught: ml.neural-networks, ml.deep-learning, ml.keras, ml.backpropagation
Neural Network Basics
A neural network is a stack of layers, each applying a linear transformation followed by a non-linear activation function. The non-linearity is what allows deep networks to learn complex patterns that no linear model can capture.
Key Concepts: Layers and Activations
- Dense (Fully Connected) Layer
- Every neuron connects to every neuron in the previous layer. Computes z = Wx + b (linear), then a = σ(z) (activation). The weights W and biases b are learned during training.
- ReLU Activation
- Rectified Linear Unit: f(x) = max(0, x). The most common hidden-layer activation. Fast to compute, avoids the vanishing gradient problem that plagued earlier sigmoids.
- Backpropagation
- The algorithm that computes gradients of the loss with respect to every weight in the network using the chain rule. These gradients tell the optimiser how to adjust each weight to reduce loss.
Building a Binary Classifier with Keras
import tensorflow as tf
from tensorflow import keras
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
import numpy as np
# Scale data (critical for neural networks)
scaler = StandardScaler()
X_train_sc = scaler.fit_transform(X_train)
X_test_sc = scaler.transform(X_test)
X_val_sc = scaler.transform(X_val)
# Build model
model = keras.Sequential([
keras.layers.Input(shape=(X_train.shape[1],)),
keras.layers.Dense(128, activation='relu'),
keras.layers.BatchNormalization(),
keras.layers.Dropout(0.3), # randomly zero 30% of neurons during training
keras.layers.Dense(64, activation='relu'),
keras.layers.Dropout(0.2),
keras.layers.Dense(1, activation='sigmoid') # binary classification → probability
])
model.compile(
optimizer=keras.optimizers.Adam(learning_rate=0.001),
loss='binary_crossentropy',
metrics=['accuracy', keras.metrics.AUC(name='auc')]
)
model.summary()
Training with Early Stopping
callbacks = [
keras.callbacks.EarlyStopping(
monitor='val_loss',
patience=10, # stop if no improvement for 10 epochs
restore_best_weights=True # keep the best model, not the last
),
keras.callbacks.ReduceLROnPlateau(
monitor='val_loss',
factor=0.5, # halve learning rate on plateau
patience=5
)
]
history = model.fit(
X_train_sc, y_train,
epochs=200,
batch_size=64,
validation_data=(X_val_sc, y_val),
callbacks=callbacks,
verbose=1
)
# Evaluate
loss, acc, auc = model.evaluate(X_test_sc, y_test)
print(f'Test accuracy: {acc:.3f} AUC: {auc:.3f}')
# Probabilities for precision/recall analysis
probs = model.predict(X_test_sc).flatten()
Multi-class Classification
# For N classes, change the final layer and loss
n_classes = 10
model = keras.Sequential([
keras.layers.Input(shape=(784,)), # e.g., MNIST 28x28 flattened
keras.layers.Dense(256, activation='relu'),
keras.layers.BatchNormalization(),
keras.layers.Dropout(0.4),
keras.layers.Dense(128, activation='relu'),
keras.layers.Dropout(0.3),
keras.layers.Dense(n_classes, activation='softmax') # outputs N probabilities summing to 1
])
model.compile(
optimizer='adam',
loss='sparse_categorical_crossentropy', # use when labels are integers
metrics=['accuracy']
)
# Or use 'categorical_crossentropy' if labels are one-hot encoded
Dropout: Regularisation for Neural Networks
Dropout randomly sets a fraction of neurons to zero during each training step. This forces the network to not rely on any single neuron, acting as an ensemble of many smaller networks. It is the standard regularisation technique for neural networks.
Neural Networks vs Gradient Boosting (Tabular Data)
- Neural Networks (Keras/PyTorch): Learns representations through stacked differentiable layers
- XGBoost / LightGBM: Gradient boosted decision tree ensemble
Tip: Use EarlyStopping(restore_best_weights=True) to automatically save the best checkpoint during training. Without this, you get the final weights which are usually worse than the best validation checkpoint.
Warning: Always scale inputs before feeding them to a neural network. Without scaling, gradients for different features will have wildly different magnitudes, making training slow and unstable.
Frequently Asked Questions
Should I use neural networks or gradient boosting for tabular data?
Gradient boosting (XGBoost, LightGBM) usually outperforms neural networks on tabular/structured data. Use neural networks for images, text, and audio — data where feature learning from raw inputs is the core challenge.
How many layers and neurons should I use?
Start small (2 hidden layers, 64-128 neurons each) and add complexity only if the validation score keeps improving. More capacity is not always better — it increases overfitting risk and training time.
What is the difference between batch size and epochs?
An epoch is one full pass through the training data. Batch size is how many samples are processed before updating weights. Smaller batches (32-64) introduce more noise (can help generalisation). Larger batches are more stable but need more memory.