“Explain backpropagation” is the deep learning equivalent of “reverse a linked list”. Almost every machine learning interview asks it in some form: walk me through how the gradient reaches the first layer, derive the gradient of softmax with cross-entropy, or why does training need to store activations. Interviewers are not checking whether you memorised a formula. They want to see that you understand a network as a chain of simple functions, that you can push a number through it by hand, and that you know what an autograd framework is doing for you when you call loss.backward().
Before you start
You should be comfortable with derivatives of simple functions (x**2, tanh, exp, log) and with the idea of a partial derivative. Matrix-vector products help, but every example below uses plain Python lists so you can follow each multiplication. The code runs on Python 3.12 or newer with the standard library only. The one PyTorch snippet is illustrative (PyTorch 2.x) and is not needed to follow the article.
The short answer
A neural network computes the loss as a composition of simple operations. Backpropagation applies the chain rule to that composition in reverse order: starting from the loss, each operation receives the gradient of the loss with respect to its output, multiplies it by its own local derivative, and passes the result to its inputs. Because each intermediate gradient is computed once and reused, one backward pass produces the gradient for every weight at a cost comparable to the forward pass. An optimizer such as SGD then moves each weight a small step against its gradient.
How it works
The chain rule says that if L depends on a, and a depends on z, then dL/dz = dL/da * da/dz. A network is just a long chain (more precisely, a directed acyclic graph) of such dependencies. The forward pass computes and remembers every intermediate value. The backward pass walks the graph from the loss to the inputs and multiplies local derivatives along the way.
Start with one neuron: z = w*x + b, a = tanh(z), and a squared-error loss L = 0.5 * (a - y)**2.
import math
x, w, b, y = 2.0, 0.5, 0.1, 1.0
z = w * x + b # forward pass, keep every intermediate
a = math.tanh(z)
loss = 0.5 * (a - y) ** 2
dloss_da = a - y # backward pass, one local derivative at a time
da_dz = 1 - a ** 2 # derivative of tanh, expressed with the saved output
dw = dloss_da * da_dz * x
db = dloss_da * da_dz
print(f"z={z:.4f} a={a:.4f} loss={loss:.6f}")
print(f"dL/dw={dw:.6f} dL/db={db:.6f}")
# z=1.1000 a=0.8005 loss=0.019900
# dL/dw=-0.143322 dL/db=-0.071661Two details matter. First, dloss_da * da_dz is shared by dw and db; in a deep network that shared prefix is the “error signal” each layer passes to the one before it. Second, the derivative of tanh is computed from the saved output a. That is why frameworks keep activations in memory during training: the backward pass needs them. It is also why training uses much more memory than inference.
For a layer z = W h + b, the same rule in vector form gives three results you should be able to write from memory:
dL/dW = dL/dz(outer product)h, one entry per weightdL/db = dL/dzdL/dh = W^T dL/dz, the signal sent to the previous layer
An elementwise activation multiplies the incoming gradient by its local derivative. For ReLU that derivative is 1 where the input was positive and 0 elsewhere, so ReLU acts as a gate that either passes the gradient through unchanged or blocks it.
Step-by-step walkthrough
The network below has two inputs, two ReLU hidden units and three output classes, trained with softmax and cross-entropy. It is small enough to check by hand.
Step 1: Run the forward pass and keep the cache
import math
def matvec(W, v):
return [sum(wij * vj for wij, vj in zip(row, v)) for row in W]
def softmax(z):
m = max(z) # subtract the max for stability
e = [math.exp(v - m) for v in z]
s = sum(e)
return [v / s for v in e]
def forward(params, x, target):
W1, b1, W2, b2 = params
z1 = [s + b for s, b in zip(matvec(W1, x), b1)]
h = [max(0.0, v) for v in z1] # ReLU
z2 = [s + b for s, b in zip(matvec(W2, h), b2)]
p = softmax(z2)
return -math.log(p[target]), (x, z1, h, p)
W1 = [[0.2, -0.4], [0.7, 0.1]]; b1 = [0.0, -0.1]
W2 = [[0.5, -0.3], [-0.2, 0.8], [0.1, 0.1]]; b2 = [0.0, 0.0, 0.0]
params = (W1, b1, W2, b2)
x, target = [1.0, 2.0], 1
loss, cache = forward(params, x, target)
print("z1", [round(v, 4) for v in cache[1]]) # z1 [-0.6, 0.8]
print("h ", [round(v, 4) for v in cache[2]]) # h [0.0, 0.8]
print("p ", [round(v, 4) for v in cache[3]]) # p [0.2089, 0.5035, 0.2876]
print("loss", round(loss, 4)) # loss 0.6861The correct class gets probability 0.5035, so the loss is -log(0.5035) = 0.6861. Notice that the first hidden unit has a negative pre-activation, so ReLU outputs 0. Keep an eye on it in the backward pass.
Step 2: Start the backward pass at the logits
For softmax followed by cross-entropy, the gradient of the loss with respect to the logits collapses to p - y, where y is the one-hot target. You can derive it by differentiating -log(softmax(z)[t]): the log cancels the exp, and what remains is p_i for every class minus 1 for the target class. This is the reason frameworks fuse the two operations into one loss function: the fused gradient is simpler and numerically safer than differentiating a separate softmax.
def backward(params, cache, target):
W1, b1, W2, b2 = params
x, z1, h, p = cache
dz2 = [pi - (1.0 if i == target else 0.0) for i, pi in enumerate(p)] # p - y
dW2 = [[dz2[i] * h[j] for j in range(len(h))] for i in range(len(dz2))]
db2 = dz2[:]
dh = [sum(W2[i][j] * dz2[i] for i in range(len(dz2))) for j in range(len(h))] # W2^T dz2
dz1 = [dh[j] * (1.0 if z1[j] > 0 else 0.0) for j in range(len(h))] # ReLU gate
dW1 = [[dz1[j] * x[k] for k in range(len(x))] for j in range(len(dz1))]
db1 = dz1[:]
return dW1, db1, dW2, db2
dW1, db1, dW2, db2 = backward(params, cache, target)
print("dz2", [round(v, 4) for v in db2]) # dz2 [0.2089, -0.4965, 0.2876]The sign pattern is the intuition to say out loud: the gradient is negative for the correct class (so gradient descent raises its logit) and positive for the wrong classes (so it lowers theirs), in proportion to how much probability they took.
Step 3: Push the signal through the hidden layer
print("dW2", [[round(v, 4) + 0.0 for v in r] for r in dW2])
# dW2 [[0.0, 0.1671], [0.0, -0.3972], [0.0, 0.2301]]
print("dW1", [[round(v, 4) + 0.0 for v in r] for r in dW1])
# dW1 [[0.0, 0.0], [-0.4311, -0.8621]]Every weight leaving the first hidden unit has zero gradient because that unit output 0, and every weight entering it also has zero gradient because the ReLU gate was closed. For this example, that unit learns nothing. If a unit’s pre-activation is negative for every input in the dataset, it never recovers: that is the “dying ReLU” problem, and it is one reason for leaky ReLU, GELU and careful initialization.
Step 4: Take one optimizer step and confirm the loss falls
lr = 0.5
step = lambda M, G: [[w - lr * g for w, g in zip(r, gr)] for r, gr in zip(M, G)]
new = (step(W1, dW1), [b - lr * g for b, g in zip(b1, db1)],
step(W2, dW2), [b - lr * g for b, g in zip(b2, db2)])
print("loss after one step", round(forward(new, x, target)[0], 4)) # 0.1132The loss drops from 0.6861 to 0.1132 for this single example. On a real dataset the gradient is averaged over a mini-batch, but the loop is identical: forward, backward, update, repeat.
Worked scenario
A teammate writes a custom activation layer with a hand-written backward pass because the framework’s version does not support a feature they need. Training runs without errors, but the loss plateaus far above the baseline. Nothing crashes, so nothing points to the bug.
The tool for this is a gradient check: nudge each parameter by a tiny eps in both directions, measure the change in loss, and compare the central difference with the analytic gradient. Simulate the teammate’s bug by forgetting the ReLU gate:
import copy
def grad_check(backward_fn, params, x, target, eps=1e-5):
loss, cache = forward(params, x, target)
analytic = backward_fn(params, cache, target)
worst = 0.0
for which in (0, 1, 2, 3):
g = analytic[which]
flat = isinstance(g[0], float)
for i, row in enumerate([g] if flat else g):
for j, gij in enumerate(row):
plus, minus = copy.deepcopy(list(params)), copy.deepcopy(list(params))
if flat:
plus[which][j] += eps; minus[which][j] -= eps
else:
plus[which][i][j] += eps; minus[which][i][j] -= eps
n = (forward(plus, x, target)[0] - forward(minus, x, target)[0]) / (2 * eps)
worst = max(worst, abs(n - gij) / max(1e-12, abs(n) + abs(gij)))
return worst
def backward_buggy(params, cache, target):
W1, b1, W2, b2 = params
x, z1, h, p = cache
dz2 = [pi - (1.0 if i == target else 0.0) for i, pi in enumerate(p)]
dW2 = [[dz2[i] * h[j] for j in range(len(h))] for i in range(len(dz2))]
dh = [sum(W2[i][j] * dz2[i] for i in range(len(dz2))) for j in range(len(h))]
dz1 = dh # BUG: the ReLU gate is missing
dW1 = [[dz1[j] * x[k] for k in range(len(x))] for j in range(len(dz1))]
return dW1, dz1[:], dW2, dz2[:]
print(f"correct: {grad_check(backward, params, x, target):.1e}") # correct: 1.8e-11
print(f"buggy: {grad_check(backward_buggy, params, x, target):.1e}") # buggy: 1.0e+00A relative error around 1e-7 or smaller (in double precision) means the backward pass is right; anything near 1 means a term is missing or has the wrong sign. The fix is to multiply by the activation’s local derivative, which the check confirms. In PyTorch, torch.autograd.gradcheck does the same job for custom autograd.Function code and expects double-precision inputs for the same reason this check uses Python floats.
Common mistake
- “Backprop is an optimization algorithm.” It only computes gradients. SGD, Adam or another optimizer decides how to use them.
- “Backward is free.” It needs the saved activations, so training memory grows with batch size, sequence length and depth. Activation checkpointing trades extra recomputation for less memory.
- Forgetting that gradients accumulate. In PyTorch,
.gradis summed acrossbackward()calls. Withoutoptimizer.zero_grad()each step uses the sum of all previous gradients, and training diverges or wanders. Accumulation is intentional only when you are simulating a larger batch. - Applying softmax before a loss that expects logits.
nn.CrossEntropyLossapplies log-softmax itself; feeding it probabilities squashes the gradients and slows learning without raising any error.
Verify the behavior
Turn the key facts into assertions that run with python3 alone:
def test_logit_gradient_is_p_minus_y():
_, cache = forward(params, x, target)
p = cache[3]
_, _, _, db2 = backward(params, cache, target)
assert all(abs(g - (pi - (i == target))) < 1e-12 for i, (g, pi) in enumerate(zip(db2, p)))
assert abs(sum(db2)) < 1e-12 # logit gradients sum to zero
def test_backward_matches_finite_differences():
assert grad_check(backward, params, x, target) < 1e-7
test_logit_gradient_is_p_minus_y(); test_backward_matches_finite_differences()
print("ok")The zero-sum assertion holds because both the probabilities and the one-hot target sum to 1.
The illustrative PyTorch equivalent (PyTorch 2.x) shows that autograd computes the same numbers:
import torch
z2 = torch.tensor([[0.0, 0.0, 0.0]], requires_grad=True)
loss = torch.nn.functional.cross_entropy(z2, torch.tensor([1]))
loss.backward()
print(z2.grad) # tensor([[ 0.3333, -0.6667, 0.3333]]), which is p - yFollow-up questions
Why reverse mode and not forward mode? A network maps millions of parameters to one scalar loss. Reverse mode gets all of those partial derivatives in one backward pass; forward mode would need one pass per parameter. Forward mode wins only when there are few inputs and many outputs.
What happens to gradients in a very deep network? The gradient for an early layer is a product of many local derivatives. If those factors are mostly below 1 it vanishes; if they are above 1 it explodes. Initialization, normalization, residual connections and gradient clipping all exist to keep that product near 1.
How does backprop work through a branch or a skip connection? When a value feeds several operations, the gradients from each use are added. A residual connection y = x + f(x) therefore passes the incoming gradient straight to x in addition to the path through f, which is why deep residual networks train well.
What is backpropagation through time? The same algorithm applied to a recurrent network unrolled over time steps. The shared weights receive the sum of gradients from every step.
Interview exercise
A three-class classifier outputs logits [2.0, 1.0, 0.0] and the true class is 0. Compute the loss and the gradient of the loss with respect to each logit, and describe what one gradient descent step does to the logits.
Answer and reasoning
Softmax of [2, 1, 0] is about [0.6652, 0.2447, 0.0900], so the cross-entropy loss is -log(0.6652) = 0.4076. The gradient with respect to the logits is p - y = [0.6652 - 1, 0.2447, 0.0900] = [-0.3348, 0.2447, 0.0900]. A gradient descent step subtracts the learning rate times this vector, so the correct logit rises by 0.3348 * lr, and the two wrong logits fall in proportion to the probability they were given. Good answers also mention that the components sum to zero, that the gradient shrinks as the model becomes confident and correct, and that a confidently wrong prediction produces a gradient near 1 for the wrong class, which is why cross-entropy learns fast from bad mistakes where squared error on probabilities would not.