Receptive Fields

Pick any single value in any feature map, at any depth in a network. Trace backward through every convolution that produced it: which input pixels could possibly change that value if you edited them? That set of pixels is the neuron's receptive field (RF) — the region of the original image it is a function of. Everything outside that region is invisible to it, structurally, before a single weight is trained.

In the first convolutional layer this is trivial: a 3×3 kernel produces an output value from a 3×3 patch, so $\text{RF} = 3$. The interesting part is what happens one layer later. A second-layer neuron reads a 3×3 patch of layer-1 outputs — but each of those layer-1 outputs already summarised its own 3×3 patch of the input. The layer-2 neuron's dependency doesn't stay at 3×3; it widens to cover every input pixel that fed any of the nine layer-1 cells it reads, which is a 5×5 patch. Stack a third layer and it widens again, to 7×7. This is the cone in the clip above: depth doesn't just add layers, it compounds the patch each neuron answers for.

Two receptive fields are worth keeping distinct. The theoretical RF is the geometric region this page computes — every pixel that could influence the output through some path of nonzero weights. The effective RF (§5) is the smaller region that influence is actually concentrated in once you account for how much each path contributes. The theoretical RF is a hard ceiling; the effective RF is where the signal-to-noise ratio is high enough to matter. Both grow with depth, but not at the same rate, and the gap between them is itself something architects budget for.

The theoretical RF gives a hard design law, and it is worth stating plainly because no amount of training escapes it: if the object you need to recognise is larger than the network's final receptive field, the network is structurally blind to it. A neuron that has never seen a pixel cannot use it — not with more epochs, not with more data, not with a better loss function. Receptive field is decided at architecture time, by the sequence of kernel sizes, strides, and dilations a designer writes down before training starts. Get the RF budget wrong and no amount of optimisation fixes it; get it right and the rest of the network's job is to learn what to do with what it can already see.

Computing the RF layer by layer needs exactly two running numbers. $r_l$ is the receptive field at layer $l$ — the side length of the input patch one output neuron depends on, exactly the quantity the cone traces. $j_l$ is the jump: the spacing, in input pixels, between the windows of two adjacent output neurons at layer $l$. At the input, $r_0 = 1$ (one pixel depends on itself) and $j_0 = 1$ (adjacent input pixels are one apart, trivially). Every layer updates both numbers from the layer below:

Read the first equation as: the new patch equals the old patch, plus $(k_l - 1)$ extra steps, each step worth $j_{l-1}$ input pixels — because a $k_l \times k_l$ window reaches $k_l - 1$ positions beyond its centre, and on the input grid each of those positions is $j_{l-1}$ pixels apart, not 1. The second equation says the jump itself only changes when a layer subsamples: a stride-$s_l$ convolution makes adjacent output neurons $s_l$ times further apart on the input than adjacent neurons were at the previous layer. A stride-1 layer leaves $j$ untouched — it widens the patch ($r$ grows) without changing how finely the output samples the input.

Here is a 4-layer stack worked row by row, all stride 1 except layer 2 (stride 2), all $3\times3$ kernels. Watch $j$ stay at 1 through layer 1, double at layer 2, and then make every subsequent step in $r$ count for twice as many input pixels:

Dilation folds in as a change to $k$, not a new formula. A dilated kernel with rate $d$ spaces its $k$ taps $d$ pixels apart instead of 1, so it spans the same area as a larger ordinary kernel of size $k_{\text{eff}} = k + (k-1)(d-1)$ — but with the same $k^2$ weights. Substitute $k_{\text{eff}}$ for $k$ in the recurrence above and dilation drops straight in: a $3\times3$ kernel at $d=2$ behaves, for RF purposes, like a $5\times5$ kernel; at $d=4$, like $9\times9$; at $d=16$, like $33\times33$. The weight count never changes — only the reach does.

The calculator below runs exactly this recurrence. Add layers, edit $k$ / $s$ / $d$, or toggle a row to pool (pooling follows the same recurrence with its own window size and stride), and watch $r$ and $j$ update live. The preset buttons load the real layer stacks of four models in this site's gallery — including the 65-pixel stack from §6.

Stacking plain $3\times3$, stride-1 convolutions grows the RF by exactly 2 per layer — $1, 3, 5, 7, 9, \ldots$ — because each layer adds $(k-1) \cdot j = 2 \cdot 1$. To cover a $224\times224$ image this way would take well over a hundred layers. Three mechanisms accelerate the climb, and the recurrence says exactly how much each one buys — and what it costs.

Stride multiplies the jump $j$ for every layer that follows. A single stride-2 layer doubles $j$ permanently downstream, so every later layer's RF contribution is worth twice as many input pixels — conv 4 in the table above gained 4 pixels of RF from the same $3\times3$ kernel that gave conv 1 only 2. Pooling (max- or average-pool with stride $s_p$) does the same arithmetic — it has its own $k_p$ and $s_p$ and obeys the identical recurrence — while adding zero parameters. Both buy RF by literally throwing away resolution: a stride-2 or pool-2 layer halves the spatial size of every feature map after it, permanently. Dilation is the odd one out — it widens $k_{\text{eff}}$ without touching $j$ or the output resolution at all, buying RF “for free” in both parameters and resolution.

Free is not quite free, though. A dilated kernel only samples $k^2$ of the $k_{\text{eff}}^2$ positions it spans — at $d=4$ a $3\times3$ kernel reads 9 of the 81 cells in its $9\times9$ footprint, and the other 72 are gaps. Stack dilated layers at the same rate and those gaps line up: a checkerboard of input pixels never gets read by anything, a failure mode called gridding. The CIFAR-10 case study in §6 avoids it by using a different dilation rate at every layer (2, 4, 8, 16), so each layer's gaps fall on positions a previous layer already covered.

Everything above computes the theoretical RF: the full set of input pixels connected to an output by some path of nonzero weights. Luo et al. (2016) asked a sharper question — if you measure how much the output actually changes per unit change in each input pixel (the gradient magnitude, averaged over many inputs), how is that influence distributed across the theoretical RF?

The answer is not “uniformly.” Influence is approximately Gaussian: pixels near the centre of the theoretical RF dominate, and influence falls off — often steeply — toward the edges. Two consequences follow directly. First, the effective receptive field (ERF) — the region carrying most of the influence — is reliably smaller than the theoretical RF, sometimes drastically so in very deep networks where the theoretical RF has grown to cover the entire input but the ERF covers only a fraction of it. Second, the ERF grows roughly with $\sqrt{\text{depth}}$ rather than linearly — so simply stacking more layers gives diminishing returns on effective reach, even though the theoretical RF keeps growing additively or multiplicatively per §3-4.

This is precisely why very deep networks still reach for stride, pooling, and dilation rather than relying on depth alone: each of those mechanisms widens the theoretical RF and helps counteract the $\sqrt{\text{depth}}$ falloff by changing how quickly the Gaussian itself spreads, instead of just adding more thin layers whose marginal contribution to the effective footprint keeps shrinking.

The CIFAR-10 dilated model in this site's gallery is the cleanest demonstration of this page's arithmetic, because every one of its layers is stride 1 — $j$ stays at 1 throughout, so the entire RF story is the $k_{\text{eff}}$ column. The assignment constraint was RF $> 44$ on a $32\times32$ input with no max-pooling and no stride-2 convolution anywhere — every pixel of resolution had to be preserved end to end. The model hits RF 65 with six layers, comfortably covering the full $32\times32$ input with margin to spare:

Two design choices make this stack work. First, why dilation suits a $32\times32$ input: on a small image, stride or pooling would shrink the feature map to almost nothing after a few layers — three stride-2 steps already takes $32\to16\to8\to4$, leaving little spatial structure for later layers to refine. Dilation grows the reach of each layer without shrinking it at all, so all six layers operate on full-resolution $32\times32$ feature maps. The escalating rates $2, 4, 8, 16$ are deliberate, not arbitrary: each rate is double the last, so each layer's sampling gaps fall where an earlier layer's taps already landed — the gridding failure mode from §4, designed around by construction.

Second, the depthwise-separable assist. Layer 2 is a depthwise-separable block: a $3\times3$ filter applied per-channel (the spatial RF contribution, $+2$ to $r$, identical to a plain $3\times3$) followed by a $1\times1$ pointwise convolution that mixes channels and contributes nothing to the RF ($k_{\text{eff}}=1$). Splitting the spatial and channel-mixing work this way is what keeps the whole six-layer stack under the parameter budget — the convolution page's factorisation sidebar covers the parameter saving in full. Here, the only point that matters is that it doesn't cost anything in the RF column: depthwise-separable convolutions grow the RF exactly like the spatial filter they contain, for free on the channel-mixing side.

The growth chart below plots this table. Toggle between the actual stack (dilated, $j=1$ throughout) and a plain-equivalent stack (six ordinary $3\times3$, stride-1 layers, which would only reach RF 13) to see the gap dilation buys — without spending a single stride step.