ResNet-50

Every model in this gallery is built from the convolution operation, but ResNet-50 is the one that takes the bottleneck variant seriously. A basic block (the kind in CIFAR-100 ResNet-18) is two stacked 3×3 convolutions plus a shortcut. A bottleneck block — ResNet-50's unit, 16 of them stacked across four stages — is three convolutions: a 1×1 that reduces channel count, a 3×3 that does the actual spatial work at that reduced width, and a 1×1 that restores the channel count back up, all wrapped in the same kind of shortcut. The name describes the shape: wide → narrow → wide, like a bottle.

Why squeeze first? Recall the parameter formula for a convolution: $\text{params} = C_{\text{out}}(k \cdot k \cdot C_{\text{in}} + 1)$. The cost of a $3\times3$ layer scales with $C_{\text{in}} \cdot C_{\text{out}}$ — squaring the channel count squares the cost. A 1×1 convolution has $k=1$, so its only job is channel mixing: it reads every input channel at one spatial position and writes a new, smaller (or larger) set of channels, at a cost of just $C_{\text{out}} \cdot C_{\text{in}}$ — no $k^2$ multiplier at all. Put a cheap 1×1 before the expensive 3×3 to shrink $C_{\text{in}}$, and the same 1×1 trick after to restore it, and the expensive op runs at a fraction of the width.

A concrete bottleneck from Layer 1, where the block operates on 64-channel, 56×56 feature maps and outputs 256 channels (matching the stage table below). Compare the bottleneck's parameter count against the naive alternative — a single ordinary 3×3 convolution mapping 64 channels straight to 256:

This is the same factorisation idea as the convolution page's depthwise-separable sidebar, applied differently: instead of splitting spatial from channel-mixing work, the bottleneck keeps the spatial $3\times3$ but runs it at reduced width, and uses cheap $1\times1$ channel-mixing convolutions to do the width changes on either side. Across all 16 bottlenecks the saving compounds — it's a large part of why ResNet-50 (25.6M params) sits so much lighter than its layer count alone would suggest, while ResNet-18's 11M basic-block parameters cover only 18 layers.

Each bottleneck also carries a shortcut: the block's input is added back to the $1\times1{\to}3\times3{\to}1\times1$ output, $h_{l+1} = h_l + F(h_l)$. Where the shapes don't match — the first block of each stage, where channel count or spatial resolution changes — the shortcut itself is a $1\times1$ convolution (with stride 2 where the stage downsamples) that projects $h_l$ into the new shape before the addition. Every other block in a stage uses a pure identity shortcut, no extra parameters at all. The mechanism, and why it exists, is the subject of the Gradient Flow tab.

Zoom out from one block to the whole network and the stage table below is the real architecture. A $224\times224\times3$ image first passes through a $7\times7$, stride-2 stem convolution and a max-pool, both before any bottleneck runs — together they take $224 \to 112 \to 56$, three-quarters of the network's total downsampling, in two non-residual layers. Then four stages of bottlenecks (3, 4, 6, 3 — 16 blocks, 48 convolutions, plus the stem and the final linear layer = 50 weight layers, the '50' in ResNet-50) each halve resolution again while doubling channel width, until Layer 4 outputs $7\times7\times2048$. Global average pooling and a 1000-way linear layer finish the job.

Now read the receptive field story off that same table, stage by stage, using the recurrence $r_l = r_{l-1} + (k_l-1)j_{l-1}$ and $j_l = j_{l-1} s_l$. The stem alone is doing heavy lifting: a $7\times7$ stride-2 convolution takes $r$ from 1 to 7 and $j$ from 1 to 2 in one layer, then the stride-2 max-pool doubles $j$ again to 4 — by the time the first bottleneck runs, $j=4$, meaning every subsequent stride-1 layer's $(k-1)=2$ contribution to $r$ is already worth $2 \times 4 = 8$ input pixels, not 2.

Each stage transition (the first block of Layer 2, 3, and 4) carries a stride-2 $3\times3$ inside the bottleneck, doubling $j$ again — so $j$ climbs $1 \to 2 \to 4 \to 8 \to 16 \to 32$ across stem, pool, and the three stage transitions. By Layer 4, $j=32$: every remaining stride-1 $3\times3$ adds $(3-1)\times32 = 64$ pixels of receptive field in a single layer. That's the accelerators mechanism — stride multiplying the jump for everything downstream — running at full strength, and it's why a $7\times7$ kernel near the output can correspond to a receptive field hundreds of pixels wide on the input. The bars in the figure above show the result: RF reaches roughly 483×483 by the last stage — larger than the 224×224 input itself. Every unit in Layer 4 is structurally capable of being influenced by the entire image, with margin to spare for the padding effects that push theoretical RF past the input size.

That headroom matters because of the effective-vs-theoretical distinction: the effective RF — where the gradient signal actually concentrates — is reliably smaller than the theoretical one, and grows roughly with $\sqrt{\text{depth}}$ rather than additively. A theoretical RF of exactly 224 would risk an effective RF that doesn't quite reach the image corners; 483 gives the Gaussian-shaped effective RF (Luo et al., 2016) room to cover the full frame even after it falls off from its theoretical ceiling. This is the architecture-time RF budgeting the receptive-field page describes in the abstract, made concrete: 50 layers, four stages, and a stem aggressive enough that the budget closes with room to spare.

ResNet-50 is 50 weight layers deep. Per the Vanishing Gradients page, backpropagation computes $\partial \mathcal{L}/\partial h_1$ as a product of $L-1$ local Jacobians — and a plain 50-layer ReLU network, even with He initialization (factor $\approx 0.97$ per layer from that page's simulator), would see $0.97^{49} \approx 0.22$: the first layer's gradient arrives at roughly a fifth of the last layer's strength. That's survivable at 50 layers, but the same arithmetic at 100+ layers (which the original ResNet paper also trained) would not be — and 0.97 is the best case; real per-layer factors vary, and a few unlucky layers compound fast.

Every one of ResNet-50's 16 bottleneck blocks computes $h_{l+1} = h_l + F(h_l)$ rather than $h_{l+1} = F(h_l)$ — the shortcut described in the Architecture tab. Differentiating gives $\partial h_{l+1}/\partial h_l = I + \partial F(h_l)/\partial h_l$: the identity matrix, plus whatever the three-convolution branch contributes.

Two more pieces from the supporting cast compound with the shortcut here. Every activation in ResNet-50 is ReLU — its derivative is either exactly 0 or exactly 1, never a shrinking fraction like sigmoid's 0.25 ceiling — and every convolution is He-initialized, chosen specifically because $\text{Var}(W) = 2/n_{\text{in}}$ compensates for the variance ReLU halves at every layer (the stage table figure in the Architecture tab is labelled HE_NORMAL INIT for exactly this reason). Stack ReLU + He + the residual highway and the Veit, Wilber & Belongie reading applies directly: ResNet-50 behaves like an ensemble of many paths of different effective depths through its 16 blocks, most of them much shorter than 16 — which is the real reason the network trains cleanly at all, not 50 layers of gradient surviving in series.

All of this is testable on this exact model's trained weights — not a simulation. The probe below runs one real backward pass through ResNet-50 and plots $\lVert \partial \mathcal{L}/\partial h_l \rVert$ at the output of each residual block. Upload any image (it doesn't need to be classified correctly — the probe only needs a loss to differentiate) and compare the resulting profile to the simulator: a roughly flat profile across blocks is the $I + \partial F/\partial h_l$ highway doing its job on real weights, the empirical version of the flat Residual (skip) curve at factor 1.00.

One subtlety worth flagging: a flat gradient-norm profile is necessary for trainability, but it doesn't mean every block contributes equally to the prediction. A block whose $F$ branch has settled near zero for a given input still receives a full-strength gradient (via the $I$ term) — it's just that the gradient is telling that branch "you're not needed for this input, stay near zero." The channel-attribution probe in the Interpretability tab is the complementary measurement: it asks not "does gradient reach this block" but "how much did this block's output actually move the final prediction." Both probes run on the same 16 blocks; together they separate trainability (this tab) from contribution (the next one).

ResNet-50 supports the gallery's full interpretability suite — Grad-CAM and per-channel attribution — because its 1000-way ImageNet classifier gives both probes enough signal to be informative. Both run live in the workspace; this tab explains what each one computes and why.

Grad-CAM answers where in the image the network looked. It takes the gradient of the predicted class's logit with respect to the final convolutional feature map — Layer 4's $7\times7\times2048$ output, just before global average pooling — and uses that gradient to weight each of the 2048 channels, then sums the weighted channels into a single $7\times7$ heatmap. Channels whose activations and whose gradient (how much increasing that channel would increase the class score) are both large dominate the sum; the result, upsampled back to $224\times224$ and overlaid on the input, is a coarse map of which regions of the image pushed the prediction toward the predicted class. Coarse is the operative word — $7\times7$ is only 49 cells, each one corresponding to a roughly $32\times32$ patch of the input, while the receptive field of every one of those cells (per the Architecture tab) is far larger than that patch — so each cell's heat reflects a wide, overlapping context, not a tight bounding box.

Per-channel attribution is a finer-grained question: not where, but which feature-map channels drove the decision, and it works at any of the four stages, not just the last. For a chosen stage's output — say Layer 4's $7\times7\times2048$ — global-average-pool each channel down to a single number (its mean activation across the $7\times7$ spatial grid), then multiply that number by the classifier's weight connecting that channel to the predicted class:

This formula is literally the forward pass of the classifier head run one channel at a time: global average pooling collapses $A_c$ to a scalar (exactly what the network itself does before the linear layer), and multiplying by $w_{c,\text{class}}$ is exactly the term that channel contributes to the predicted class's logit, before summing across all $C$ channels and adding the bias. A channel with a large positive contribution is a large positive vote for the predicted class — there's no approximation or surrogate model involved, unlike Grad-CAM's gradient-weighted heuristic.

What turns this from a number into an explanation is which channel ranks highest, and that's where the feature hierarchy comes back in. Early-stage channels (Layer 1, RF still small per the architecture tab's table) respond to generic edges, colour blobs, and textures — a high-ranking Layer-1 channel for a photo of a dog might just mean "lots of fur-like texture," true of many classes. By Layer 4, where the receptive field exceeds the input and channels have had three stages of composition to work with, a high-ranking channel is far more likely to correspond to something object-specific — an eye, a wheel, a characteristic silhouette. Running the attribution probe at each of the four stages on the same image and watching the top-ranked channels shift from generic to specific is the feature hierarchy, measured on this network's real trained weights instead of the schematic clip on the convolution page.

Both probes are interactive, not illustrative — open the workspace to upload your own image, run Grad-CAM to see where ResNet-50 looked, then run channel attribution at each stage to see which channels, and what those channels generically respond to, drove the call. Together they're the closest this gallery gets to reading a 25.6M-parameter network's mind on a single forward pass.

Accuracy on a clean validation set is one number; what happens when the input isn't clean is a different question entirely, and ResNet-50 supports all three of this gallery's robustness probes — adversarial attack, common corruptions, and out-of-distribution detection — making it the model where the full picture is visible.

FGSM (Fast Gradient Sign Method, Goodfellow et al. 2014) is the simplest adversarial attack and the clearest demonstration that high accuracy and robustness are different properties. It computes the gradient of the loss with respect to the input image — the same kind of quantity the Gradient Flow tab measures with respect to hidden activations, just one layer further back — and nudges every pixel by a tiny fixed amount $\varepsilon$ in whichever direction increases the loss:

The unsettling result, reproducible in the workspace, is how little $\varepsilon$ it typically takes to flip a confident prediction on a 25.6M-parameter, 76%-accurate network — a perturbation small enough to be invisible at normal viewing size can move the top-1 class to something unrelated, often with higher confidence than the original correct prediction had. FGSM is a single gradient step, the crudest member of a family of attacks (PGD iterates it; this gallery's workspace exposes the one-step version with an $\varepsilon$ slider), and the network was never trained against it — nothing about 76.01% top-1 accuracy implies any resistance to a worst-case, gradient-directed pixel nudge. Accuracy measures performance on the data distribution the model saw; adversarial robustness measures performance against an adversary who can see the model's gradients, which is a fundamentally different and much harder bar.

Common corruptions (in the style of Hendrycks & Dietterich's ImageNet-C benchmark) ask a gentler but more realistic question: not "what happens under an adversary," but "what happens under ordinary camera and environment noise?" The workspace applies corruptions — Gaussian noise, blur, JPEG compression artifacts, brightness shifts — at a severity you control, and reports how top-1 accuracy and confidence degrade as severity increases. Unlike FGSM, these perturbations aren't computed from the model at all; they're generic image-processing operations that any camera, any compression pipeline, or any weather condition could introduce. A model that holds up well here is one that's likely to keep working when deployed, where inputs are never as clean as a curated validation set.

Out-of-distribution (OOD) detection asks a third question: does the network know what it doesn't know? Feed ResNet-50 an image of something outside ImageNet-1k's 1000 classes entirely, and softmax — by construction — still outputs a full probability distribution that sums to 1. The workspace's OOD signal looks at two derived quantities: confidence (the top-1 softmax probability — how much probability mass landed on a single class) and entropy ($-\sum_i p_i \log p_i$ — how spread out the distribution is across all 1000 classes). A well-calibrated response to an unfamiliar input is low confidence and high entropy — the network "hedging" across many classes because nothing matches well. A network that instead reports high confidence on an out-of-distribution input is making a confidently wrong prediction with no internal signal that anything is amiss, which is the actual failure mode this probe is designed to surface.

All three probes share a theme with the Gradient Flow and Interpretability tabs: they turn an abstract property of the network — sensitivity to its input gradient, sensitivity to pixel-level noise, calibration of its own output distribution — into something you can watch change on a real image, in real time, on the same 76.01%-top-1 trained weights used everywhere else on this page.