Attention
Attention lets a model decide which other parts of the input each position should listen to, and by how much. It is the mechanism behind transformers, and it is how Vision Transformers process an image without any convolutions.
Queries, keys, and values
Every position produces three vectors. A query that says what it is looking for, a key that says what it offers, and a value that carries its content. To update a position, the model compares its query against every key with a dot product, turns those scores into weights with a softmax, and returns the weighted average of the values. Positions whose keys match the query contribute the most.
The result is a content-based, data-dependent mixing of information. Unlike a convolution, whose connections are fixed by the kernel, attention recomputes who talks to whom for every input. Self-attention is the case where the queries, keys, and values all come from the same sequence, so each position attends to the rest of the same input.
- check_circleQuery: what this position is looking for.
- check_circleKey: what each position offers for matching.
- check_circleValue: the content that gets mixed in once weights are set.
- check_circleSoftmax over query-key scores produces the attention weights.
Attention on images
A Vision Transformer cuts the image into a grid of small patches, embeds each patch as a vector, and treats the patches as a sequence. Self-attention then lets any patch draw on any other patch from the very first layer, so the model can relate distant parts of the image immediately rather than waiting for the receptive field to grow.
This trades the built-in locality of convolutions for flexibility. Transformers usually need more data to learn the spatial structure that convolutions get for free, but at scale they match or beat CNNs and have become the dominant architecture across vision and language. The Vision Transformer page here is the planned next model to slot into the gallery.