Vanishing Gradients

A network is a chain of layers, each one a function of the layer before it: $h_1 \to h_2 \to \dots \to h_L$, with the loss $\mathcal{L}$ computed from $h_L$. Training needs $\partial \mathcal{L} / \partial h_1$ — how much the loss would change if the very first hidden representation changed slightly — so that the first layer's weights know which direction to move. The chain rule is the only tool calculus gives you for this, and it has a specific shape: to get from $h_L$ back to $h_1$, you multiply together one local derivative per layer in between.

Here is that product written out for an $L$-layer network. $\partial \mathcal{L}/\partial h_L$ is the gradient at the output — how the loss responds to the last layer — and each $\partial h_{l+1}/\partial h_l$ is the local Jacobian of one layer, how that layer's output responds to its input:

A product of $L-1$ numbers, each smaller than 1, shrinks exponentially with depth — and exponential decay is brutal even when each individual factor looks harmless. Take a factor of $0.62$ per layer, which is roughly what an unfavourably-initialised ReLU network gives you (this is the exact attenuation used in the hero clip above). After 20 layers:

Now make the per-layer factor the sigmoid activation's derivative ceiling, $0.25$ (derived in §3 below), and stack just 12 of them — a network depth that was completely ordinary by the late 2000s:

The mirror case: exploding gradients. The same product runs the other way if the per-layer factors are larger than 1 — the gradient grows exponentially instead of shrinking, and a few dozen layers later it overflows to inf or NaN and training diverges in a single step. Exploding gradients are easier to spot (the loss goes to NaN, impossible to miss) and easier to patch: gradient clipping rescales the gradient vector if its norm exceeds a threshold, capping the size of any one update without changing its direction. Clipping is a band-aid, not a cure — it does not fix the underlying Jacobian, it just stops one bad batch from destroying the weights. Vanishing gradients have no equivalent quick fix, which is why the rest of this page is about why the factors end up below 1 in the first place, and the architectural change that addresses the cause rather than the symptom.

Every layer's local Jacobian $\partial h_{l+1}/\partial h_l$ contains the derivative of that layer's activation function, applied element-wise. For decades, the default activation was the logistic sigmoid, $\sigma(x) = 1/(1+e^{-x})$, which squashes any input into $(0, 1)$ — a natural choice when a layer's output was interpreted as a probability or a soft gate. Differentiate it and a clean identity falls out:

Think of $\sigma'(x)$ as a gate that every unit of gradient must pass through on its way backward. At best — when the unit's pre-activation $x$ is exactly 0 — the gate lets through a quarter of the gradient and blocks the rest. Move $x$ away from 0 in either direction and the gate closes further: at $x = \pm 4$, $\sigma'(x) \approx 0.018$, and at $|x| > 6$ it is effectively zero. A unit whose pre-activation sits in that flat region is called saturated — its output barely changes no matter how its input changes, so its gradient gate is all but closed. The figure below plots $\sigma(x)$ alongside $\sigma'(x)$: the derivative's peak is the $0.25$ ceiling at $x=0$, and the shaded saturation zones at the edges are where the gate is essentially shut.

tanh is the same shape stretched to range $(-1, 1)$, and its derivative $1 - \tanh^2(x)$ peaks at $1.0$ instead of $0.25$ — a real improvement, four times less attenuation per layer — but it is still a saturating function: $\tanh'(\pm 3) \approx 0.01$, so a deep tanh network eventually hits the same wall, just a few layers later than sigmoid.

ReLU, $f(x) = \max(0, x)$, breaks the pattern entirely because its derivative is not a shrinking gate — it is a binary one: $f'(x) = 1$ for $x > 0$ and $f'(x) = 0$ for $x \le 0$. A gradient passing through an active ReLU unit ($x>0$) is multiplied by exactly $1$ — passed through unchanged, no attenuation at all. A gradient hitting an inactive unit ($x \le 0$) is multiplied by $0$ — killed outright, but for that one unit only; the gradient continues to flow through every other unit in the layer. This is why stacking ReLU layers does not produce the same exponential decay as stacking sigmoids: most factors in the product are exactly $1$, not $0.62$ or $0.25$.

Footnote — dying ReLU. The failure mode ReLU trades in for saturation is units that get stuck permanently in the $x \le 0$ region during training. Once a unit's weights and bias drift so that its pre-activation is negative for every input in the dataset, its gradient is $0$ forever — it never updates again, and it is effectively deleted from the network. A small fraction of dead units is harmless (the layer has redundancy), but a large fraction is a real failure that careful initialization and a sensible learning rate avoid. Leaky-ReLU and its relatives (a small non-zero slope for $x \le 0$) exist specifically to give dead units a way back.

Saturation is not only a property of the activation function — it is a property of where the inputs to that function land, and that depends on how the weights were initialized. Picture the very first forward pass, before any training has happened. Each layer computes a weighted sum of its inputs: $n_{\text{in}}$ terms, each the product of an input value and a weight. If the weights are drawn from a distribution with variance $\mathrm{Var}(W)$, and the inputs have variance $\mathrm{Var}(x)$, then — treating the sum as $n_{\text{in}}$ roughly independent terms — the variance of the pre-activation adds up across the sum:

The fix is to choose $\mathrm{Var}(W)$ so that the product $n_{\text{in}} \cdot \mathrm{Var}(W)$ equals $1$ — the variance of the activations neither grows nor shrinks as the signal moves forward, layer after layer, and (by the same chain-rule argument run backward) the gradient's variance is preserved on the way back too. Xavier/Glorot initialization (2010) sets $\mathrm{Var}(W) = 1/n_{\text{in}}$, which makes $n_{\text{in}} \cdot \mathrm{Var}(W) = 1$ exactly — the right answer for a linear or tanh-like layer, whose output variance roughly equals its input variance.

He initialization (2015) doubles the variance specifically to compensate for ReLU. A ReLU zeroes out roughly half of its inputs — every pre-activation that lands at or below 0 becomes exactly 0 — so the variance of a ReLU layer's output is roughly half the variance of its input. Xavier's factor of $1/n_{\text{in}}$ would let the signal's variance halve at every layer, which is itself a slow-motion vanishing-gradient problem stacked across dozens of layers. He's factor of $2/n_{\text{in}}$ exactly cancels that halving: $n_{\text{in}} \cdot (2/n_{\text{in}}) \cdot \tfrac{1}{2} = 1$. One sentence each, but the difference is the standard initialization for every ReLU-based CNN in this gallery — get it wrong (use Xavier on a ReLU network, say) and the activation spread silently drifts across dozens of layers, with no error message, until the network at the far end is either dead or saturated.

The figure below sketches that drift schematically across six layers under three regimes: weights drawn too small (the spread collapses toward zero — every layer looks more and more like a constant), a well-scaled He/Xavier init (the spread holds roughly steady), and weights drawn too large (the spread balloons until activations saturate). None of this is visible by inspecting any single layer in isolation — it is a compounding effect, the same exponential-product structure as §2, just acting on activations forward instead of gradients backward.

Every idea above reduces to the same arithmetic: a per-layer factor, raised to a power equal to the depth. The simulator below makes that arithmetic interactive. Pick a depth from 2 to 50 layers and a regime — sigmoid (factor $0.25$, from §3), ReLU with a naive initialization (factor $0.62$, the hero clip's plain network), ReLU with He initialization (factor $0.97$ — variance approximately preserved, so the gradient decays only slowly), or a residual network (factor $1.00$ — the identity shortcut from §6 guarantees a path that never shrinks at all). The chart plots $\text{norm}(l) = \text{factor}^{l}$, the surviving fraction of the gradient $l$ layers back from the loss, and a toggle switches the y-axis between linear and logarithmic.

Try this: leave the regime on ReLU, plain init and drag the depth slider to 20. On the linear scale the curve looks like it has already hit zero by depth 10 — which is true to four decimal places, but hides exactly how small "zero" is. Flip to the log scale and the curve keeps falling in a straight line all the way to depth 20, where it reads roughly $7 \times 10^{-5}$ — four orders of magnitude down, the same number from §2's gut-punch. Now switch to Residual (skip): the line is flat at $1.0$ regardless of depth. That flat line is the entire reason §6 exists.

Every factor discussed so far has been a property of one layer's transformation — its activation function, its weight scale. The residual connection (He et al., 2015) changes the shape of the layer itself. Instead of a layer computing $h_{l+1} = F(h_l)$ — the output is whatever the weight layers compute — a residual block computes:

Differentiate that sum and the chain rule does something it has not done anywhere else on this page — it produces an addition instead of a product term:

Substitute this back into §2's product. Every factor in the chain is now $I + \partial F(h_l)/\partial h_l$ rather than just $\partial F(h_l)/\partial h_l$ — and a product of terms that are each "identity plus something" does not collapse toward zero the way a product of small numbers does. Concretely: even in the worst case, where every $\partial F(h_l)/\partial h_l \approx 0$ (every residual branch has gone to sleep), the product of $L-1$ copies of $I$ is still $I$ — the gradient arrives at layer 1 completely undiminished. The identity term is a gradient highway running underneath the whole network; the $F$ branches are side streets that add detail on top, but the highway is always there.

Veit, Wilber & Belongie (2016) gave this a memorable reading: a ResNet behaves like an ensemble of many paths of different lengths through the network. Every subset of residual blocks that get "skipped" (their $F$ contribution near zero for a given input) defines one shorter effective path from input to output, and the network's behaviour is the combination of all $2^{L}$ such paths at once. Most of those paths are short — a handful of blocks deep — and short paths have short gradient products, which is the real reason a 50-block ResNet trains at all: it is not really one 50-layer chain that the gradient must traverse in full, it is an ensemble dominated by much shorter ones.

This is what the hero clip at the top of this page shows directly. The skip arcs are not a forward-pass shortcut for efficiency — the forward computation is identical with or without them. They exist purely so that $\partial h_{l+1}/\partial h_l$ contains an $I$, which is what lets the backward pulse reach block 1 at full strength while the plain network's pulse has faded to almost nothing by the same point.

Residual connections did not arrive in a vacuum — they were the last piece of a stack of fixes that, together, are what let networks scale from a handful of layers to hundreds. ReLU (§3) replaced a per-layer factor that was at best $0.25$ with one that is either $0$ or $1$ — no shrinkage on the paths that survive. He initialization (§4) keeps the forward signal's variance from drifting, which keeps units away from whatever saturation a ReLU-adjacent activation still has, and keeps the backward signal's variance close to 1 for the same reason. Batch normalization (Ioffe & Szegedy, 2015 — not covered in depth on this page, but worth naming here) re-centres and re-scales each layer's activations during training, which has the practical effect of keeping every layer's local Jacobian close to a well-conditioned matrix, batch after batch, even as the weights themselves are changing.

Stack all three — ReLU, He init, batch norm — and you can train a plain (non-residual) network to roughly 20 layers before the accumulated product in §2 starts to bite again; this is, not coincidentally, close to where networks like VGG (2014) topped out. Residual connections are the piece that removes the depth limit rather than pushing it back: ResNet's original paper trained variants past 100 layers, and later work pushed past 1,000. The qualitative difference is the one drawn in §6 — ReLU/He/BatchNorm make each per-layer factor closer to 1, while residual connections guarantee a term that is exactly 1, independent of what the rest of the layer is doing.

Historical sidebar. The vanishing-gradient problem was not discovered alongside ResNet — it was identified and largely solved, for a different architecture, 24 years earlier. Hochreiter's 1991 diploma thesis first formalised why simple recurrent networks could not learn long-range dependencies: the same chain-rule product, applied across time steps instead of layers, with the same exponential collapse. The fix — the LSTM (Hochreiter & Schmidhuber, 1997) — introduced gated memory cells whose update rule includes an additive, near-identity path for the cell state across time steps, specifically so that $\partial c_{t+1}/\partial c_t$ could be close to $1$. That is the same structural idea as $I + F'(h_l)$ in §6, arrived at independently, almost two decades before residual connections brought it to feed-forward vision networks.

Everything on this page is a story about $\partial \mathcal{L}/\partial h_l$ for various $l$ — and that quantity is not hypothetical. It is a tensor that exists, momentarily, during every real backward pass, and its size (its norm) can be measured directly. A gradient-norm probe runs one backward pass through a trained network and records $\lVert \partial \mathcal{L}/\partial h_l \rVert$ at the output of every block, from the last block (closest to the loss) back to the first (closest to the input).

The schematic below shows what that probe's output looks like for the two families on this page. The residual profile (teal) is close to flat — the gradient norm at block 1 is within roughly half an order of magnitude of the norm at block 8, exactly the "identity highway" of §6 doing its job across real, trained weights, not just the idealised $I + F'$ algebra. The plain profile (cinnamon) decays sharply toward the early blocks — by block 1 it is several orders of magnitude below block 8, the real-network analogue of the $0.62^{20}$ and $0.25^{12}$ numbers from §2.

CIFAR-100 ResNet-18 in this gallery runs exactly this probe on its own trained weights, live, in the Gradient Flow tab — not a schematic, the actual per-block norms from a real backward pass through the model that reached 76.61% validation accuracy. Open it alongside this page's simulator (§5) and compare: the live profile should look like the teal curve above, not the cinnamon one — and if you switch the simulator to ReLU, plain init at depth 18 (roughly this model's effective depth), you'll see why a plain version of the same architecture would not have trained nearly as well.