Deep Learning Fundamentals

The core ideas that every neural network architecture shares: how a network computes, learns and generalises. Start here before the architecture-specific pages.

1. Overview

Deep learning is machine learning with neural networks of many layers, trained end to end by gradient descent. Whatever the architecture (feed-forward, convolutional, recurrent, graph, Transformer), the same loop runs underneath: a forward pass turns inputs into predictions, a loss function scores them, backpropagation computes how each weight affected the loss, and an optimizer nudges the weights. This section collects the building blocks that every architecture relies on, so the architecture pages can focus on what makes each one different.

Most training failures trace back to a fundamentals problem rather than an architecture problem: gradients that vanish because of a poor activation or initialisation, a loss that does not match the output layer, a learning rate that is too high, or a network that memorises the training set. Understanding these pieces makes debugging systematic.

2. How the Pieces Fit Together

• Neurons and layers compute weighted sums; activation functions add the non-linearity that gives depth its power and largely determine whether gradients flow.
• Loss functions define the goal: cross-entropy for classification (paired with softmax or sigmoid outputs), mean squared error for regression.
• Backpropagation applies the chain rule layer by layer; optimizers such as SGD with momentum and Adam turn gradients into updates.
• Initialisation and normalisation (He/Xavier init, batch and layer norm) keep signal scale stable through depth.
• Regularisation (weight decay, dropout, early stopping, data augmentation) manages the bias-variance trade-off.

A suggested reading order. Begin with activation functions, because they explain why depth needs non-linearity and why some networks will not train at all. Next, revisit the bias-variance trade-off, since almost every deep learning technique beyond the basics is a way of controlling capacity and generalisation. Then move to the architecture pages: the multi-layer perceptron shows the training loop in its plainest form, convolutional networks add weight sharing for images, recurrent networks add state for sequences, and graph networks generalise message passing to arbitrary structures. Each architecture page assumes the vocabulary introduced here, so returning to this section when something is unclear is the fastest way to fill gaps.

3. Which Choice Goes Where

4. Related Topics