Convolution

A kernel is a template — a small grid of numbers that encodes one visual pattern. The kernel in the clip above encodes “dark on the left, bright on the right.” To ask whether a patch of the image contains that pattern, you take the dot product: multiply each kernel weight by the pixel under it and add everything up. Patches that match the template produce a large sum; patches that don't produce a sum near zero. A convolution simply asks that one question at every position, and the feature map it produces is a map of where the pattern occurs.

Here is the whole operation in symbols. $I$ is the input image, $K$ is a $k \times k$ kernel, and the output at position $(i, j)$ is the dot product between the kernel and the patch whose top-left corner sits at $(i, j)$:

Plug numbers in once and the formula stops being abstract. Take a 3×3 patch whose columns read $0, 1, 1$ — a vertical edge — and the vertical-edge kernel with columns $-1, 0, +1$. Each row contributes $0 \cdot (-1) + 1 \cdot 0 + 1 \cdot (+1) = 1$, and three rows sum to 3: a strong response, because the patch is exactly the pattern the kernel encodes. Slide the same kernel onto a flat patch (all ones) and the row sums become $-1 + 0 + 1 = 0$ — no edge, no response. Every multiplication is laid out below.

A naming footnote worth knowing: the sum above is technically cross-correlation. True mathematical convolution flips the kernel 180° before sliding it, which makes the algebra tidier (it turns convolution into multiplication in the frequency domain). Deep-learning libraries skip the flip and call it convolution anyway — and nobody minds, because the kernel weights are learned. If the optimizer needs the flipped version of a filter, it simply learns the flipped weights. The distinction matters in signal processing; in a CNN it is a historical naming accident.

Now run the operation yourself. The playground below applies your kernel to a 16×16 image entirely in the browser — edit any weight, or load a preset, and watch the feature map react. Teal cells are positive responses, cinnamon cells are negative ones (the pattern matched in reverse).

Four knobs control where the window lands. Kernel size $k$ sets the patch each output value sees. Stride $s$ is how far the window jumps between positions — stride 2 visits every other position and halves the resolution, which is why it is described as “convolve and downsample in one move.” Padding $p$ adds a border of zeros so the window can sit on the edges; without it, corner pixels appear in only one window while centre pixels appear in $k^2$ of them, so edge information gets systematically under-counted and the output shrinks at every layer. Dilation $d$ spreads the kernel's taps apart with gaps, widening the region the kernel spans without adding a single weight.

One formula sizes the output of any combination. For an $n \times n$ input, first widen the kernel by its dilation, then count how many strides fit:

Check it against real layers from this site's model gallery — each row is a convolution that actually runs in the workspaces:

The explorer below puts all four knobs on sliders over an 8×8 input. Watch the gray cells: whenever $n + 2p - k_{\text{eff}}$ does not divide evenly by the stride, the floor in the formula throws those input pixels away — they are read by no window at all.

Here is the part nearly every diagram gets wrong, including the classic textbook GIFs: a “2D” convolution on a colour image is a three-dimensional operation. The input is not a flat grid — it is a volume with $C_{\text{in}}$ channels (3 for RGB, often 64 or 256 deeper in the network). One kernel is therefore a $k \times k \times C_{\text{in}}$ volume of weights that reads every input channel at once, and its sliding dot product still produces exactly one 2-D feature map. Depth in, flat out.

The output gains depth only because a layer learns many kernels: $C_{\text{out}}$ independent kernel volumes produce $C_{\text{out}}$ stacked feature maps, and that stack becomes the next layer's input volume. So the parameter count of a convolutional layer multiplies three things — output channels, kernel area, and input depth (plus one bias per kernel):

Plug in a real layer: the 3×3 convolution inside ResNet-50's first bottleneck block has $C_{\text{in}} = 64$ and $C_{\text{out}} = 64$, so it holds $64 \times (3 \cdot 3 \cdot 64 + 1) = 36{,}928$ parameters. Notice what the formula does not contain: the spatial size of the input. The same 36,928 weights process a 56×56 feature map in ResNet-50 — or a billboard, if you fed it one.

Sidebar: factorising the volume. Because $k \cdot k \cdot C_{\text{in}}$ multiplies, it pays to split it. A depthwise-separable convolution first applies one $k \times k$ filter per input channel (the spatial part, $k^2 C_{\text{in}}$ weights), then mixes channels with a 1×1 convolution ($C_{\text{in}} C_{\text{out}}$ weights). For a 3×3 layer mapping 64 channels to 64, that is $576 + 4{,}096 = 4{,}672$ weights instead of $36{,}864$ — roughly an 8× cut, and the saving approaches $k^2$ as channel counts grow. The CIFAR-10 dilated model in this gallery uses exactly this factorisation to stay under its 200K parameter budget.

The brutal comparison first. Connect a 28×28 image to a 28×28 output with a dense layer — every output pixel free to look at every input pixel — and you need $784 \times 784 = 614{,}656$ weights. A 3×3 convolution producing the same-sized output needs 9. That is a factor of about 68,000, and it comes from two structural bets baked into the operation: each output only needs to look at a small neighbourhood (locality), and the same pattern detector is worth applying at every position (weight sharing).

Weight sharing buys more than efficiency — it buys a guarantee called translation equivariance: shift the input and the feature map shifts with it, exactly. Formally, if $T$ translates an image, then $(T I) \star K = T(I \star K)$. A dense layer offers nothing of the sort; it would have to relearn the digit “7” separately at every position it might appear. A convolution learns the detector once and gets every position free, which is also why convolutions generalise from so much less data: the 614,656-weight dense layer has the capacity to memorise positions, and the 9-weight kernel has no choice but to learn the pattern.

The fine print: equivariance is not invariance. The feature map moves with the object; it is the later global pooling stage that collapses position away and makes the final classification translation-tolerant. And sharing weights is exactly the right bias only because image statistics are roughly the same everywhere — an edge is an edge in any corner of the photo. Break that assumption (say, on data with a fixed layout) and the bet pays less.

Everything above treated the kernel's weights as given — but nobody writes the vertical-edge kernel anymore. The weights are learned: backpropagation nudges every kernel toward whatever pattern reduces the loss. The remarkable, repeatable result is that the first layer of essentially every vision model ever trained — AlexNet made this famous in 2012 — rediscovers the same vocabulary on its own: oriented edge detectors, colour-opponent blobs, and Gabor-like texture filters. Decades of hand-designed computer-vision features, reinvented from pixels by gradient descent in a few epochs.

Depth is what turns that vocabulary into understanding. A first-layer kernel sees a 3×3 patch and can only assert “an edge at 40° is here.” A second-layer kernel reads feature maps, not pixels — its input channels are “edge here, edge there” — so it can detect conjunctions: corners, stripe pairs, textures. Stack again and motifs compose into object parts; again, and parts compose into objects. Each stage also sees a wider patch of the original image than the last (the receptive field grows with depth), which is precisely what gives the later kernels room to represent something as large as a whole object.

This page will not fake the evidence with screenshots — you can generate it live. The ResNet-50 workspace runs channel attribution on any image you upload: it ranks which feature-map channels drove the prediction, stage by stage. Early-stage channels respond to edges and colour patches regardless of the subject; by stage 4 the top-ranked channels fire on object-specific structure — wheels, faces, fur. The hierarchy in the clip above is a cartoon of what that probe shows on real photographs.

Every model in the gallery is built from the operation on this page — the differences are which variant each one bets on: