See locally,
share globally.
An image is not a bag of pixels. Nearby pixels belong together, and the same pattern can appear anywhere.
Numbers mode · small matrices you can calculate by hand.
Use the Numbers / Images button above to switch every illustration and lab. Image mode uses a real handwritten 7. Pixel values stay visible; teal and coral distinguish positive and negative filter responses.
Examples use fixed filters, one channel and no bias unless stated. Real CNNs learn filters. Questions use their own stated values.
Same pattern.
Different place.
If a handwritten 7 moves across the page, should the network have to learn it all over again?
Before CNNs became widely used, image recognition often meant hand-designed features plus a classifier, or a fully connected network on flattened pixels. The architectural question: what should the model assume about an image?
Think of a reusable stencil. Local connections let it inspect a small neighbourhood; weight sharing lets the same stencil search the whole page. Stacking these detectors lets strokes become parts, then objects.
People choose edges, textures or descriptors; a classifier learns how to use them. Feature design and classification are separate.
Can learn features, but assigns separate weights to each input/output pair. Locality and translation sharing must be learned from examples.
Learned local detectors are reused across space. Deeper layers combine their responses into larger patterns.
04 / THE FORMULAFormula & notation
Convolution: K² · Cin · CoutWeight counts, excluding biases · compare the same output shape
H, W: input height and width · K: kernel side · C: channel count · “out”: output dimensions.
Go deeperTheory, formula walkthrough & worked examples
A flattened image still contains all its pixels. What a dense layer lacks is a built-in rule for neighbourhoods and shared detectors. A locally connected layer adds neighbourhoods, but still has separate weights at each output location. Convolution adds weight sharing as well. The smaller parameter count is an inductive bias, not a guarantee that every vision problem needs a CNN.
Why locality and sharing are separate decisions
This story predates AlexNet: LeNet demonstrated learned convolutional features for document recognition in the 1990s. AlexNet later scaled CNN training to ImageNet with GPUs, ReLU and regularisation.
For 32×32 → 30×30 with one channel: dense = 921,600 weights; local but unshared = 8,100; convolution = 9. Compute still increases with the number of output locations, even when parameter count does not.
What does one detector cost?
EXPLOREThe small grid shows a sample patch. The counter uses the image size you select.
Scale the image.
One input channel, one output channel, 3×3 kernel, stride 1, no padding. All three architectures produce (H − 2)² outputs.
Double the image side. Which design keeps the same number of weights?
A 32×32 input has 1,024 pixels. A 3×3 valid convolution produces 30×30 = 900 outputs. A dense layer to those 900 outputs needs weights. One shared 3×3 filter needs . Which idea lets a detector work at every location: locality or sharing?
Show answer
1,024 × 900 = 921,600; 3 × 3 = 9; weight sharing. Local connectivity alone uses 9 × 900 = 8,100 separate weights.
A small template.
A sliding inner product.
A pixel is bright. Is it an edge, part of a stroke, or just a flat patch?
A kernel is a small stencil with a score: multiply its weights by the covered pixels, then add. Slide the same stencil to find the same pattern in a new location.
04 / THE FORMULAFormula & notation
Go deeperTheory, formula walkthrough & worked examples
Read the sum. The indices u, v run over the kernel. i, j choose where the top-left of the kernel sits on the input. Every output cell uses the same K. If you had a different K at every location you would have built a locally connected layer with unshared weights, and you would have lost translation sharing. A bias b, when present, is added after the sum and is also shared across space.
Three things to separate
- The arithmetic
Each placement is an inner product. If you can do one by hand, you can do the layer.
- The sharing
The kernel does not change as it moves. Different output cells are different patches, not different detectors.
- The meaning
Learned kernels become texture and part detectors. The presets here are the classical ones, so you can predict the map.
On a 7×7 input with a 3×3 kernel, stride 1 and no padding, there are 5×5 = 25 placements. Each uses the same nine weights. The lab highlights one placement at a time and writes the inner product on the output grid.
Predict before you play: Will the vertical-edge kernel fire in the middle of the left-hand block of 1s?
No. Inside a constant block every 3×3 of 1s gives 1+1+1 − (1+1+1) = 0. The kernel fires at the light-to-dark transition.
In Numbers mode, the edge response is zero inside a constant block and nonzero at its boundaries. In Images mode, compare that response with the strokes of the handwritten 7. The coral frame marks the current patch, regardless of its response.
Weight sharing is not a trick for speed only. It is the claim that the world is stationary: the same pattern is legal everywhere.
“Convolution in deep learning is the same as the mathematical convolution.” Libraries implement cross-correlation: they do not flip the kernel. The name stuck. For learning, an unflipped kernel is simply a different parameterisation.
One patch at a time.
Every cell shows its value. The expanded inspector below keeps more decimal places.
Choose a detector.
Predict the edge-x map, then switch the kernel. Does blur shrink the jump at the boundary?
Use this fixed numerical example for the questions below, in either visual mode.
Row 1 contributes . Row 2 contributes . Row 3 contributes . The output pixel is .
Show answer
1·1 + 2·0 + 0·(−1) = 1. Then 0 + 0 − 2 = −2. Then 3 + 0 − 1 = 2. Total 1 − 2 + 2 = 1. An edge kernel of this shape is a vertical first difference: bright on the left, dark on the right, gives a positive response.
How far the
template hops.
Scanning every possible patch is expensive. What detail do we lose when we take bigger steps?
Imagine reading a tiled floor by stepping on every tile, then every second tile. Stride is the step size of the detector; larger steps produce a smaller output grid.
04 / THE FORMULAFormula & notation
Go deeperTheory, formula walkthrough & worked examples
Read the hop. Increasing S makes the output smaller and makes each output cell look at a patch whose top-left is farther from its neighbours. The receptive-field jump (Part 6) is multiplied by S. A large stride in the first layer, as in AlexNet, is a deliberate decision to throw away fine spatial grid while keeping a large field of view.
Skipped origins are simply not computed
- Start at (0,0)
Then add S to the row or column until the kernel would hang off the (padded) input.
- The formula is not optional
If (H + 2P − K) is not divisible by S, the leftover border is unused. That is what the floor is for.
- Do not confuse with pooling
Stride in a convolution still computes a learned inner product. Pooling (Part 5) summarises a patch with max or mean and has no kernel weights.
AlexNet’s first layer: H = 227, K = 11, S = 4, P = 0. Hout = ⌊(227 − 11)/4⌋ + 1 = 55. A 224-pixel input with P = 2 also gives 55; with P = 0 it gives 54. State the input and padding convention together.
Predict before you play: If we raise stride from 1 to 2 on the 7×7 blob, does any output cell still see the peak?
With the identity kernel, only the centre pixel contributes. Here the peak is at (3,3), and stride 2 still samples that centre. A different alignment could miss it even if the window covers it.
The output grid should shrink as you drag stride. Count the coral visits: 25, then 9, then 4.
Fewer hops, smaller map.
Stride 2 and 3 hop directly between sampled origins; skipped origins are never evaluated. The identity kernel copies each selected centre pixel.
Change the hop.
Set S = 2. How many output cells remain? Check it against the formula.
Same H = 7, K = 3, P = 0. Stride 1 gives Hout = . Stride 2 gives . Stride 3 gives .
Show answer
⌊(7 − 3)/1⌋ + 1 = 5. ⌊4/2⌋ + 1 = 3. ⌊4/3⌋ + 1 = 2. The origins for stride 2 are (0,0), (0,2), (0,4) and the three matching rows 2 and 4 — nine visits, not 25.
Keep the border
in the conversation.
Why does a perfectly ordinary 3×3 filter make the image smaller after every layer?
Add a temporary frame around a photograph so the stencil can reach its edges. Zero padding supplies that frame; it preserves size for an odd kernel at stride 1 when P = (K − 1)/2.
04 / THE FORMULAFormula & notation
Go deeperTheory, formula walkthrough & worked examples
Read the halo. Padding is not a trick to invent data. It is a decision about whether corner pixels get to be window centres. Without it, every layer eats the border and a deep stack silently crops the image. With it, spatial size stays a design choice of stride and pooling, not of kernel size. Zero padding is the default; it can create a faint “border artefact” that later layers sometimes exploit.
Zeros around the picture
- Write the padded array
A 2×2 block with P = 1 becomes 4×4, with the original numbers in the centre and zeros on the rim.
- Run the usual formula on the padded size
H′ = H + 2P, then Hout = ⌊(H′ − K)/S⌋ + 1, which is the same as the compact formula.
- Same is a policy, not a miracle
It holds for stride 1 and odd K. Stride 2 with “same” padding in frameworks is a slightly different rounding convention; always print the tensor shape.
Input [[1,2],[3,4]] with P = 1. The padded grid is a 4×4 of zeros with that block in the centre. A 3×3 identity then returns a 2×2 — the original image — because each original pixel can sit at a window centre.
Predict before you play: With P = 1 on the 7×7 blob, is the output 7×7 or 9×9?
7×7. Padding adds two rows, the 3×3 then removes two, and you are back to 7. You do not keep the extra halo in the output.
Padding cells contain zeros (pale in Numbers mode, black in Images mode). P = 1 restores the original height and width for this stride-1 3×3 layer.
A halo of zeros.
The input drawn here includes the padded zeros when P > 0.
Add a border.
Set P = 1 and compare input and output sizes. Then try P = 2: the output grows.
For H = 5, K = 3, S = 1, the P that keeps Hout = 5 is . For H = 7, K = 5, S = 1, P = . With H = 7, K = 3, S = 1, P = 0, Hout = .
Show answer
P = (3−1)/2 = 1 and (5−1)/2 = 2. Unpadded 7 with 3×3 is ⌊(7−3)/1⌋+1 = 5, which is why the identity lab without padding loses a one-pixel frame.
One kernel,
one channel.
One detector sees vertical strokes. Who notices the horizontal strokes or smooth regions?
Think of a team of specialists: one looks for vertical changes, another horizontal changes, another local brightness. Each writes a map; the next layer reads the whole stack.
04 / THE FORMULAFormula & notation
Go deeperTheory, formula walkthrough & worked examples
Read the volume. For a colour image Cin = 3 at the first layer. After that, Cin is whatever the previous layer emitted. Doubling Cout doubles both compute and parameters. Spatial size (H, W) does not appear in the parameter count — that is the point of sharing — but it does appear in the multiply-adds, which are Hout Wout times the parameter count, before bias.
Maps are stacked, not mixed in the drawing
- One detector, one plane
Edge-x, edge-y and blur are three independent inner products. Early layers of a CNN learn similar low-level detectors, then mix them.
- Next layer sees the stack
Its kernels have thickness Cin. A 3×3 “on 64 maps” is 64 × 3 × 3 = 576 weights per output channel.
- 1×1 convolutions
K = 1 still mixes channels at a single location. That is a cheap learned change of C, used heavily after VGG (Network-in-Network, Inception, ResNet bottlenecks).
In Numbers mode, a 7×7 input and three 3×3 filters give three 5×5 maps at stride 1 with no padding: a 5×5×3 volume. In Images mode the 8×8 digit gives a 6×6×3 volume. The kernels are fixed, so this illustrates a forward pass rather than training.
Predict before you play: Will the blur map look like a softer copy of the edge, or like a softer copy of the image?
A softer copy of the image. Blur averages the 3×3; it is not an edge detector. Edge-x should be quiet inside the white block and loud at the vertical jump.
Compare the three maps side by side. Thickness of the stack is the number of maps, not the spatial size.
Same image, three stories.
Input → selected kernel → its output. Choose a map below to bring that detector into focus.
Inspect a detector.
Compare the same location in all three maps. Why do their responses differ?
16 kernels, 3 input channels, 3×3, no bias: weights. With a bias per kernel: . VGG’s first layer, 64 kernels on RGB 3×3, no bias: .
Show answer
16 × 3 × 9 = 432, plus 16 biases = 448. 64 × 3 × 9 = 1728. A fully connected layer from 224×224×3 to 64 maps of similar size would require hundreds of billions of weights; the convolution stays at 1728.
Summarise,
then move on.
Do we need the exact location of every strong response, or can we keep a smaller summary?
Like choosing one representative for each neighbourhood, max pooling keeps the strongest response. Average pooling keeps the mean. Neither learns weights; both trade spatial detail for a smaller map.
04 / THE FORMULAFormula & notation
Go deeperTheory, formula walkthrough & worked examples
Read the summarise. Max pooling keeps the strongest detector response in the window and forgets where inside the window it was, up to the window size. Average pooling keeps mass instead of peaks. Neither replaces a convolution: you still need a learned kernel to decide what “strong” means. Modern nets (ResNet and after) often prefer stride-2 convolutions to explicit pooling in the main path, but the geometry is the same idea.
Downsample without new parameters
- Partition the map
Non-overlapping 2×2 stride 2 is a partition. Overlapping 3×3 stride 2 (AlexNet) lets neighbouring windows share pixels.
- Reduce H and W, keep C
Pooling is applied per feature map. You still have the same number of detectors, on a coarser grid.
- Invariance is local
A 1-pixel shift that stays inside the same window can leave the max unchanged. A shift that crosses a window boundary does not. True invariance is built by stacking this with later layers, not by one pool.
In the fixed 4×4 example in the questions below, the bottom-right window is [[8,1],[1,3]]. Max pooling gives 8, while average pooling gives 13/4 = 3.25. In Images mode the same operations act on the digit’s 8×8 pixel matrix.
Predict before you play: If we change the 8 to a 1, which max-pool cell changes?
Only the bottom-right, which becomes max(1,1,1,3)=3. The other three windows do not contain that entry.
Toggle max versus average. Max tracks peaks; average tracks mass. Neither redraws the 4×4 as a convolution would.
Four cells become one.
Pooling has no kernel grid to show — only the window and the summary.
Max or mean.
Compare max and average on the highlighted window. Which preserves the strongest stroke response?
Use this fixed numerical example for the questions below, in either visual mode.
| 1 | 3 | 2 | 0 |
| 4 | 1 | 0 | 2 |
| 2 | 2 | 8 | 1 |
| 0 | 5 | 1 | 3 |
2×2 max, stride 2. Top-left , top-right , bottom-left , bottom-right . Top-left average is .
Show answer
max(1,3,4,1)=4, max(2,0,0,2)=2, max(2,2,0,5)=5, max(8,1,1,3)=8. Average of the first window (1+3+4+1)/4 = 2.25.
What can this
unit actually see?
A 3×3 detector sees a tiny patch. How can a deeper unit see a whole digit?
Imagine looking through overlapping windows: a window onto the previous layer covers several windows onto the original image. The receptive field tracks how far that view reaches.
04 / THE FORMULAFormula & notation
Go deeperTheory, formula walkthrough & worked examples
Read the growth. Jump J is the spacing, in input pixels, between adjacent units at the current layer. A stride-2 layer doubles that spacing. Padding changes which pixels exist, not the theoretical field size of a unit in the bulk. A 1×1 convolution does not grow RF at all. Pooling with K = 2, S = 2 grows RF by 1 × J and then doubles J — the same recurrence, with no learned weights.
Field of view is designed, not accidental
- One 7×7 versus three 3×3
Both have RF = 7 at stride 1. The stack uses 27 weights with one channel throughout instead of 49, and it inserts two extra nonlinearities.
- Stride multiplies jump
After a downsample, further 3×3s grow the field faster, in input pixels, because each step is a larger jump.
- Effective field can be smaller
The theoretical RF is the support. If weights in the corners are tiny, the practical field is concentrated in the centre. Dilated convolutions grow RF without a stride, by inserting holes in the kernel.
Three stride-1 3×3 layers grow RF from 1 to 3 to 5 to 7. Set stride to 2 and the recurrence gives RF 3, 7, 15 and jumps 2, 4, 8. The lab clips a field larger than the displayed image; the table still reports the full theoretical size.
Predict before you play: Two 3×3 layers at stride 2 — is RF 5 or something larger?
Layer 1: RF = 3, J = 2. Layer 2: RF = 3 + 2×2 = 7, J = 4. Larger than two stride-1 layers (RF 5), because the second kernel steps in input pixels of size 2.
Read the table under the grid. RF and jump should match the recurrence, not the picture’s crop.
Grow the square.
Coral outlines the theoretical receptive field. Values are input pixels. When the field exceeds this image, the growth bars below show its full size.
| Layer | K / S | RF | Jump |
|---|
Stack layers.
Match three 3×3 stride-1 to one 7×7. Then give them stride 2 and watch jump take over.
Three stride-1 3×3 layers: RF = . AlexNet’s first layer, K = 11, S = 4: RF = and jump = . After one more 3×3 stride-1: RF = .
Show answer
1+2+2+2 = 7. First layer: RF = 11, J = 4. Then RF = 11 + 2×4 = 19, J stays 4. That is why an early large stride buys field of view quickly and spends spatial precision.
Move the object.
Move the features?
Move the digit one pixel. Should its feature map stay still, or move with it?
A tracking spotlight moves with its subject: that is equivariance. A label that still says “7” is invariant. Stride-1 convolution provides the first property away from boundary effects; the whole classifier must learn the second.
04 / THE FORMULAFormula & notation
Go deeperTheory, formula walkthrough & worked examples
Read the commuting diagram. T_Δ slides the array. If you convolve then shift, you should get the same array as shift then convolve. Weight sharing is exactly what makes that true: the detector does not know it has moved. Fully connected layers do not commute with T_Δ, because each position has its own weights. Classification wants invariance at the end, not at the first layer — you still need to know that the ear is next to the eye.
Shift the input, compare the whole maps
- Equivariance first
The identity convolution with P = 1 copies the image. Shifting the copy is the same as copying the shift. That is the cleanest picture of the property.
- Boundaries still matter
Finite-image boundaries can break exact equivariance even with same padding. This lab leaves enough zero margin for every displayed shift, so its vertical-edge responses shift exactly with the input. Cropping the object or its surrounding context can break that equality.
- Then pool
Max-pool after the shift is not always a shift of the pooled map. Pooling can leave some within-window shifts unchanged, but it does not guarantee invariance.
In Numbers mode, slide the blob; in Images mode, slide the digit inside a zero border. With the vertical-edge filter and room to shift inside the zero margin, convolving the shifted input gives the shifted edge map. Compare all cells in the pooled maps before claiming that they stayed the same.
Predict before you play: If we shift by 2 pixels, does 2×2 max-pool stay identical?
Not necessarily. Two pixels is a whole pooling window, so the peaks move into a different output cell. A 1-pixel shift is the interesting case for local invariance.
The vertical-edge responses travel with the input. The zero margin keeps the demonstrated shifts inside the frame; cropping or a larger stride can change this relationship.
The object moves. The detector follows.
Input shows pixel brightness. Convolution shows signed vertical-edge responses. Max pooling summarises those responses. Each pair uses the same colour scale.
Slide the input.
Set the shift to 1, then 2. Equivariance of conv should hold at both; pooling’s “sameness” should not.
Away from boundary effects, stride-1 convolution is translation . After a 1-pixel shift of the input, the feature map shifts by pixel. A fully connected layer on flattened pixels is, in general, .
Show answer
Equivariant; 1; neither (not equivariant, and not invariant). Pooling is a step toward invariance, locally, as in 5.1.
The network that
made the field look up.
The building blocks exist. How do we turn them into a network that can recognise complex images?
Think of an assembly line: convolution and ReLU build features, pooling reduces their spatial size, and a classifier combines the evidence. AlexNet paired this hierarchy with a training recipe that scaled in 2012.
04 / THE FORMULAFormula & notation
Go deeperTheory, formula walkthrough & worked examples
Read the first layer. An 11×11 kernel on RGB is 11×11×3 = 363 weights per map, times 96 maps — still tiny next to the later fully connected layers, which hold most of AlexNet’s 60 million parameters. The large stride is an aggressive downsample: jump 4 already, RF 11. Everything after that reasons on a 55×55 (then smaller) grid, not on the raw 227.
AlexNet’s lessons that lasted: ReLU instead of saturating sigmoids, dropout on the large fully connected layers, heavy augmentation, and GPU training. Local response normalisation did not last. The fully connected tail is the parameter hog; VGG kept it, later nets replaced it with global average pooling.
“AlexNet is 8 learned layers, so it is small.” Depth is 8, but the last three layers are dense. Parameter count is not the same as convolutional depth. VGG-16 is deeper in convolution and even heavier in FC.
At the board What 2012 changed
Before AlexNet, the ILSVRC winner was typically a carefully engineered vision pipeline. After it, the default became: learn the features, use more data, use ReLU, regularise the dense layers. The architecture details were a snapshot. The training recipe was the revolution.
# First AlexNet-style conv in PyTorch (simplified)
nn.Conv2d(3, 96, kernel_size=11, stride=4, padding=0)
nn.ReLU(inplace=True)
nn.MaxPool2d(kernel_size=3, stride=2)Follow the feature pipeline.
EXPLOREA small, computed teaching pipeline. These fixed filters are not trained AlexNet activations.
Inspect a stage.
Look for negative responses after convolution. Then select ReLU: which values survive? What changes after pooling?
First-layer Hout with 227, K = 11, S = 4, P = 0: . Weights in one 11×11 RGB kernel: . After that pool 3×3 stride 2, spatial size ≈ .
Show answer
55. 11×11×3 = 363. Pool: ⌊(55 − 3)/2⌋ + 1 = 27. (Some implementations use 27×27 after rounding conventions on 224-input variants.)
Only 3×3.
Just go deeper.
We need a 7×7 view. Must we pay for one large 7×7 filter?
Build a wider view one small window at a time. Three 3×3 layers at stride 1 reach a 7×7 region. With one channel throughout they use 27 weights versus 49; ReLUs between layers also change what functions can be represented.
04 / THE FORMULAFormula & notation
Go deeperTheory, formula walkthrough & worked examples
Read the bargain. A 5×5 is two 3×3 (RF 5, 18 vs 25 weights). A 7×7 is three 3×3. VGG uses that substitution everywhere, and uses channel widths (64-128-256-512-512). The spatial size of 224 is kept by P = 1 until a pool halves it. The design is a template: later residual nets kept 3×3 as the default atomic convolution.
Deeper, narrower kernels
- Match the field
n layers of 3×3 at stride 1 give RF = 1 + 2n, so n = 3 matches 7×7.
- Count the weights
n × 9 versus (1+2n)². For n > 1 the stack is cheaper, and the gap widens.
- Pay in depth
More layers are harder to train (vanishing signal — see the initialization sheet). VGG still trained with the 2014 recipe. Residual connections later made this bargain even better.
With n = 3, three 3×3 filters use 27 weights and cover a 7×7 region. One 7×7 filter uses 49 weights. These counts assume one channel throughout and no biases. With a fixed width C at every layer, both counts scale by C².
Predict before you play: At n = 4, is the large kernel 9×9?
Yes. RF = 1 + 8 = 9, weights 36 vs 81. The stack is less than half the price.
The two coral regions should match. For n > 1, the stack’s weight count is lower. Equal receptive fields do not mean equal functions: the intermediate ReLUs matter.
| Idea | What it changes | Typical default | Watch for |
|---|---|---|---|
| Kernel | What is detected | 3×3 | Unflipped cross-correlation in code |
| Stride | Output grid and jump | 1, or 2 to downsample | Floor leftovers |
| Padding | Whether borders are centres | P = 1 with 3×3 | Framework “same” with stride 2 |
| Feature maps | How many detectors | Double after a downsample | Params ignore H, W |
| Pooling | Summary, cheap downsample | 2×2 max, stride 2 | No learned weights |
| Receptive field | Support on the input | Grows with depth | Theoretical vs effective |
| Translation | Conv equivariant, not invariant | — | Pooling can help; no general guarantee |
| AlexNet | ReLU, dropout, GPUs | 11×11 S=4 first | FC parameter hog |
| VGG | Deep 3×3 stacks | P = 1, pool halves | Very heavy FC tail |
Same field, fewer weights.
Compare receptive-field support on the same input. Weight counts assume one channel in every layer and no biases.
How many 3×3s.
Set n = 3 and compare the highlighted regions and weight counts. At n = 1 both designs coincide.
Weights in three 3×3 vs one 7×7, one in-channel and one out-channel: vs . Two 3×3 vs one 5×5: vs . VGG same-padding P for K = 3: .
Show answer
27 vs 49, 18 vs 25, P = 1. The extra ReLUs are not in the weight count and are a further reason the stack can represent a more refined function on the same field.
Eight questions,
then three sentences.
Expected 55. Got 54. What changed?
You expect 55 output positions from an 11×11 filter at stride 4. Compare a 224-pixel input with padding 0 and padding 2.
Unfold the shape calculation
P = 0: ⌊(224 − 11)/4⌋ + 1 = 54. P = 2: ⌊(224 + 4 − 11)/4⌋ + 1 = 55. A 227-pixel input with P = 0 also gives 55. Check input size and padding together.
- A 5×5 kernel, stride 1, no pad, on 32×32. Hout? Type 28, 32, 27 or 16.
- Why share a kernel across space? (a) GPUs cannot store more weights (b) The same pattern is legal everywhere (c) It makes the network equivariant to scaling (d) It removes the need for padding.
- Max-pool 2×2 stride 2 is (a) a learned 2×2 kernel (b) a parameter-free downsample (c) the same as stride-2 convolution (d) translation equivariant with no qualification.
- Three 3×3 stride-1 layers, RF?
- Convolution is (a) invariant (b) equivariant (c) neither (d) only equivariant after pooling. Type the letter.
- VGG prefers 3×3 stacks because (a) they have a larger field than 7×7 (b) they match the field with fewer weights and more nonlinearities (c) they need no padding (d) AlexNet already used only 3×3.
- 16 filters, 32 input channels, 3×3, with bias. Parameters?
- AlexNet first-layer stride 4 mainly (a) increases parameters (b) downsamples and sets jump = 4 (c) is same-padding (d) replaces dropout.
Show answers
1: 28 = 32 − 5 + 1. 2: (b). 3: (b) — a stride-2 convolution still has learned weights. 4: 7. 5: (b) for the conv itself. 6: (b). 7: 16×32×9 + 16 = 4608 + 16 = 4624. 8: (b).
After you have a sentence, open the model answers. Nothing here is auto-marked.
- Why does a valid 3×3 stack shrink the image, and how does VGG stop that?
- Why is max-pooling not a substitute for learning kernels?
- Why can three 3×3 layers represent a more refined function than one 7×7 on the same field?
Show model answers
1. Each valid 3×3 removes a one-pixel border (K − 1 = 2). VGG sets P = 1 so every pixel can be a centre, and uses pooling (or later, stride) when it actually wants to downsample. 2. Pooling only summarises responses that already exist. Without a kernel, there is no detector to summarise. 3. The stack composes three linear templates with ReLU between them, so the effective function on the 7×7 patch is piecewise linear and deeper, not a single inner product.
You can see the patch, and the network.
You can now size a map, count the weights, grow a receptive field, and say what AlexNet and VGG each contributed.