Backpropagation, Explained Step by Step with a Worked Example

Many people can train a neural network in PyTorch — call loss.backward(), watch the loss fall — without being able to say what that call does. Backpropagation answers one question for every weight: if I nudged this weight a little, how much would the final loss change? This article works through that question by hand, on a network small enough to check with a pocket calculator, then shows how the same pattern scales to millions of parameters.

The credit assignment problem

A network predicts through a chain of intermediate values. The output layer is easy to judge: compare prediction with target and you have an error. Hidden units are harder — no one told them what their output should have been, so you must infer their responsibility from how they influenced later layers.

That inference problem is called credit assignment. Paul Werbos described a solution in 1974, and Rumelhart, Hinton and Williams made it widely known in 1986, showing hidden layers trained this way could learn useful representations on their own. Their method is backpropagation: run the network forward, measure the loss, then propagate a gradient signal backward, layer by layer, using the chain rule.

The chain rule is the whole engine. If loss L depends on weight w through intermediate z, then dL/dw = (dL/dz) × (dz/dw). Backpropagation applies that repeatedly, reusing each intermediate gradient instead of recomputing it for every weight feeding into it. That reuse makes deep training practical.

A network small enough to do by hand

Our worked example has one input, one hidden unit, and one output. It is tiny, but contains every ingredient of a real network: weighted sum, bias, non-linear activation, and squared-error loss.

Input: x = 2.0
Hidden weights: w1 = 0.5, b1 = 0.0, with a ReLU activation
Output weights: w2 = 2.0, b2 = 0.0, with a linear output
Target: t = 4.0
Loss: L = (y − t)², where y is the prediction

In symbols, the forward computation is h_pre = w1·x + b1, then h = ReLU(h_pre), then y = w2·h + b2, then L = (y − t)². Keep those four lines in view. The backward pass is just those lines read in reverse, with a derivative attached to each one.

The forward pass, step by step

First, compute and record every intermediate value. Backpropagation needs them later, which is why frameworks cache activations during the forward pass.

h_pre = 0.5 × 2.0 + 0.0 = 1.0
h = ReLU(1.0) = 1.0 — ReLU passes positive values through unchanged.
y = 2.0 × 1.0 + 0.0 = 2.0
L = (2.0 − 4.0)² = (−2.0)² = 4.0

The network predicted 2.0 against a target of 4.0, so the loss is 4.0. That tells us it is wrong, but not which weight to move, by how much, or in which direction. The backward pass answers all three.

The backward pass: the chain rule, one link at a time

Start at the loss and work backward. At each step, separate the upstream gradient arriving from the right from the local derivative of the current operation. Multiply them to get the gradient to pass further left.

Step 1: From the loss to the prediction

For L = (y − t)², the derivative is dL/dy = 2(y − t). Substituting the forward values gives 2 × (2.0 − 4.0) = −4.0. The sign matters: the gradient is negative, meaning that increasing y would decrease the loss. That matches intuition — the prediction is too low.

Step 2: Through the output layer

The output is y = w2·h + b2. Its local derivatives are simple: dy/dw2 = h = 1.0, dy/db2 = 1, and dy/dh = w2 = 2.0. Multiply each by the upstream gradient −4.0:

dL/dw2 = −4.0 × 1.0 = −4.0
dL/db2 = −4.0 × 1 = −4.0
dL/dh = −4.0 × 2.0 = −8.0 — this is the signal to send into the hidden unit.

Notice the pattern for a multiply gate: the gradient for one input is the upstream gradient times the other input. For an add gate, the gradient is copied to both inputs unchanged. Those two patterns, plus a router for max operations, cover most cases in practice.

Step 3: Through the activation and into the hidden layer

ReLU has derivative 1 when its input is positive and 0 when it is negative. Here h_pre = 1.0, so the derivative is 1 and the signal passes through: dL/dh_pre = −8.0 × 1 = −8.0. If h_pre had been negative, the gradient would have been zero and the hidden weights below would have learned nothing from this example — the familiar dead-ReLU case.

Now h_pre = w1·x + b1, so dh_pre/dw1 = x = 2.0, dh_pre/db1 = 1, and dh_pre/dx = w1 = 0.5:

dL/dw1 = −8.0 × 2.0 = −16.0
dL/db1 = −8.0 × 1 = −8.0
dL/dx = −8.0 × 0.5 = −4.0 (we do not update inputs, but the value is useful for checking and for stacking layers)

We now have a gradient for every parameter: w1: −16.0, b1: −8.0, w2: −4.0, b2: −4.0. One backward sweep produced all four. It did not require four separate forward evaluations.

Taking a step and checking that it worked

Gradient descent updates each parameter in the opposite direction to its gradient: new = old − η × gradient, where η is the learning rate. With η = 0.01:

w1 = 0.5 − 0.01 × (−16.0) = 0.66
b1 = 0.0 − 0.01 × (−8.0) = 0.08
w2 = 2.0 − 0.01 × (−4.0) = 2.04
b2 = 0.0 − 0.01 × (−4.0) = 0.04

Run the forward pass again with the updated values: h_pre = 0.66 × 2.0 + 0.08 = 1.40, h = 1.40, y = 2.04 × 1.40 + 0.04 = 2.896, and L = (2.896 − 4.0)² ≈ 1.22. The loss fell from 4.0 to about 1.22 in one step. That check is not decoration — if a hand-written backward pass does not reduce the loss on a tiny example, or disagrees with a numerical gradient, the implementation is wrong.

A standard check is the centred finite difference: compare (L(w+ε) − L(w−ε)) / 2ε with the analytic gradient. They should agree closely in double precision — essential when writing a custom layer.

The general pattern in real networks

Real layers work on vectors and matrices, but the logic is unchanged. Each layer receives an upstream gradient and returns gradients for its weights and input. For a linear layer, the weight gradient uses the upstream gradient and cached input, the bias gradient is the upstream gradient summed over the batch, and the input gradient is the upstream gradient times the transposed weight matrix. Frameworks call this a vector-Jacobian product: they never build the full Jacobian, only multiply it by the incoming vector.

Activations contribute a local factor. The sigmoid is instructive: if s = σ(z), then ds/dz = s(1 − s), which is at most 0.25, attained at s = 0.5. A gradient passing through many saturated sigmoid layers is multiplied by a number well below 0.25 again and again, so it shrinks toward zero. That is the classical vanishing-gradient problem. ReLU helps because its derivative on the active side is exactly 1. Residual connections, careful initialisation, normalisation and gradient clipping address the related vanishing and exploding cases in deeper networks. When local factors are consistently larger than 1, the opposite failure occurs and gradients can grow exponentially until updates destabilise training.

Branching follows a clean rule too: if one value feeds two downstream paths, its gradient is the sum from each path. That is why PyTorch accumulates into .grad — and why training loops must clear gradients before each step.

Why the backward direction is the efficient one

A network has many inputs in the mathematical sense — every weight is an input to the loss function — but only one output, the scalar loss. Reverse-mode automatic differentiation exploits that asymmetry. One forward pass records the computation graph; one backward pass returns the gradient with respect to every parameter at once. Forward-mode differentiation, by contrast, would need roughly one pass per parameter. For a model with millions or billions of parameters, that difference is the difference between feasible and impossible.

That is what loss.backward() does in PyTorch. Tensors with requires_grad=True record operations; backward() traverses that graph in reverse, applies the chain rule at each node, and stores results in .grad. An optimiser such as SGD or AdamW then updates the parameters. The hand calculation and the framework call are the same algorithm.

The mental model to carry away is short: forward, cache everything; backward, multiply upstream by local and pass it left; at fan-out, sum; then step opposite the gradient. If you can do the four-parameter example above without looking, you understand what every training loop is doing underneath, and you have a reliable way to debug the moments when a loss refuses to move.