The Evolution of CNNs: LeNet, AlexNet, VGG, ResNet, and EfficientNet

Convolutional neural networks (CNNs) went from reading handwritten zip codes to beating humans at ImageNet classification in under 25 years. Each leap came from one clear idea: weight sharing, depth, simplicity, skip connections, and compound scaling. This guide traces those ideas, shows the numbers, and gives you code to run each architecture yourself.

Quick Takeaways

• LeNet-5 (1998) proved that convolutions plus pooling learn visual features from raw pixels with only ~60K parameters.
• AlexNet (2012) won ImageNet with a 15.3% top-5 error, using ReLU, dropout, and GPU training.
• ResNet (2015) solved the degradation problem with residual connections, enabling 152-layer networks.
• EfficientNet (2019) scales depth, width, and resolution together, matching ResNet-level accuracy with up to 8.4x fewer parameters.

Figures are approximate and vary by training recipe. Torchvision weights often score higher than the original papers.

Why CNNs Work: The Three Core Ideas

Before comparing architectures, you need the three principles every CNN shares.

1. Local receptive fields. Each neuron looks at a small patch (for example, 3×3 or 5×5 pixels), not the full image.
2. Weight sharing. One filter slides across the whole image. This slashes parameters versus a fully connected layer.
3. Hierarchical features. Early layers detect edges. Middle layers detect textures. Deep layers detect object parts.

The output size of a convolution follows a simple formula:

Output = ⌊(W − K + 2P) / S⌋ + 1

Here W is input width, K is kernel size, P is padding, and S is stride. A 224×224 input with a 7×7 kernel, stride 2, and padding 3 yields 112×112. ResNet uses exactly this in its first layer.

LeNet-5 (1998): The Blueprint

Yann LeCun’s team built LeNet-5 to read handwritten digits on bank checks. It processed 32×32 grayscale inputs.

Architecture

What It Got Right

• Convolution then pooling as a repeating pattern.
• End-to-end training with backpropagation.
• Tiny footprint: about 60,000 parameters.

What Held It Back

LeNet used tanh/sigmoid activations, which saturate and slow learning. Compute was limited, and datasets were small. Support vector machines dominated vision for the next decade.

AlexNet (2012): The Deep Learning Breakout

AlexNet, by Krizhevsky, Sutskever, and Hinton, won ILSVRC 2012 with a 15.3% top-5 error. The runner-up scored about 26.2%. That 10-point gap ended the debate about hand-crafted features.

Architecture Highlights

• 8 learned layers: 5 convolutional and 3 fully connected.
• ~61M parameters, mostly in the fully connected layers.
• Input: 224x224x3 (commonly cited; the paper’s crop math implies 227).
• Training: two GTX 580 GPUs (3GB each), about 5-6 days.

Four Innovations

Code: Load a Pretrained AlexNet

import torch
from torchvision import models, transforms
from PIL import Image

# Load pretrained weights (ImageNet-1K)
model = models.alexnet(weights=models.AlexNet_Weights.DEFAULT)
model.eval()

# Standard ImageNet preprocessing
preprocess = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225]),
])

img = preprocess(Image.open("cat.jpg")).unsqueeze(0)
with torch.no_grad():
    probs = torch.softmax(model(img), dim=1)
print(probs.topk(5))

VGG (2014): Depth Through Simplicity

Oxford’s Visual Geometry Group asked one question: what happens if you only use 3×3 convolutions and go deeper? VGG-16 and VGG-19 answered it.

Why Stacked 3×3 Filters Win

Two stacked 3×3 convs cover a 5×5 receptive field. Three cover 7×7. The stacked version has fewer weights and more non-linearities.

That is a 45% parameter reduction with deeper feature learning.

VGG-16 Configuration

• 13 conv layers + 3 FC layers, all convs 3×3, stride 1, padding 1.
• Max pooling 2×2 after each block (five blocks total).
• Channel progression: 64 → 128 → 256 → 512 → 512.
• ~138M parameters, roughly 15.5 GFLOPs per image.

The Cost

VGG’s three FC layers hold about 120M of those parameters. It is slow, memory-hungry, and expensive to deploy. Still, its clean design makes it a favorite backbone for feature extraction and neural style transfer.

ResNet (2015): Breaking the Depth Barrier

Researchers noticed a paradox. A 56-layer plain network had higher training error than a 20-layer one. This was not overfitting. It was an optimization failure called the degradation problem.

The Residual Block

Instead of learning a mapping H(x) directly, ResNet learns the residual:

y = F(x) + x

The identity shortcut gives gradients a clean path backward. If a layer has nothing useful to add, F(x) can drift to zero.

Block Types

The bottleneck keeps compute manageable. ResNet-152 has fewer FLOPs (~11.3 GFLOPs) than VGG-16, despite being about 10x deeper.

Results

ResNet-152 ensembles won ILSVRC 2015 with a 3.57% top-5 error, below the commonly cited ~5.1% human benchmark.

Code: Build a Residual Block

import torch.nn as nn

class BasicBlock(nn.Module):
    def __init__(self, channels):
        super().__init__()
        self.conv1 = nn.Conv2d(channels, channels, 3, padding=1, bias=False)
        self.bn1 = nn.BatchNorm2d(channels)
        self.conv2 = nn.Conv2d(channels, channels, 3, padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(channels)
        self.relu = nn.ReLU(inplace=True)

    def forward(self, x):
        out = self.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        return self.relu(out + x)  # the skip connection

Why It Still Matters

Residual connections now sit inside Transformers too. Every attention and feed-forward sublayer in a GPT-style model wraps its output with an identity skip.

EfficientNet (2019): Scaling With Math, Not Guesswork

Before EfficientNet, engineers scaled models ad hoc: deeper (ResNet-200), wider (Wide ResNet), or higher resolution. Tan and Le showed that balancing all three dimensions works better.

Compound Scaling

depth = α^φ, width = β^φ, resolution = γ^φ

Subject to α · β² · γ² ≈ 2, with α ≥ 1, β ≥ 1, γ ≥ 1. The coefficient φ controls total compute. The grid search on the B0 baseline found α = 1.2, β = 1.1, γ = 1.15.

Building Blocks

• MBConv: inverted residual with depthwise separable convolutions (from MobileNetV2).
• Squeeze-and-Excitation (SE): channel-wise attention.
• Swish activation: x · sigmoid(x).
• Baseline B0 found via neural architecture search.

EfficientNet Family

EfficientNet-B7 matched the best prior model’s accuracy with about 8.4x fewer parameters and 6.1x faster inference on CPU, per the paper.

Code: Fine-Tune EfficientNet-B0

import torch.nn as nn
from torchvision import models

model = models.efficientnet_b0(weights=models.EfficientNet_B0_Weights.DEFAULT)

# Freeze the backbone
for p in model.features.parameters():
    p.requires_grad = False

# Replace classifier head for 10 classes
model.classifier[1] = nn.Linear(model.classifier[1].in_features, 10)

Train the head with AdamW, a learning rate near 1e-3, and label smoothing 0.1. Then unfreeze and drop the rate to 1e-4.

Head-to-Head Comparison

Practical Workflows and Real-World Use Cases

Choosing a Backbone

1. Edge device or mobile app? Start with EfficientNet-B0 or MobileNet. Quantize to INT8 for 2-4x speedups.
2. Need a reliable baseline? Use ResNet-50. It has the widest tooling support and well-understood behavior.
3. Building a perceptual loss or style transfer? Use VGG-16 features.
4. Teaching or debugging a pipeline? Use LeNet on MNIST for fast iteration.

Transfer Learning Template

Industry Examples

• Medical imaging: ResNet and EfficientNet variants classify chest X-rays and retinal scans, often with heavy augmentation and class-weighted loss.
• Manufacturing: lightweight EfficientNet models flag defects on-device at the production line.
• Retail: ResNet embeddings power visual search by comparing feature vectors.

Common Pitfalls

• Forgetting ImageNet normalization when using pretrained weights.
• Training with BatchNorm and tiny batches (under 8). Freeze BN statistics instead.
• Comparing models at different input resolutions.

Where CNNs Stand Against Vision Transformers

Vision Transformers (ViT) now lead many large-scale benchmarks, but CNNs remain strong when data is limited or latency matters. ConvNeXt modernized the ResNet design with Transformer-era tricks (7×7 depthwise kernels, LayerNorm, GELU) and closed much of the gap. Hybrid models combine both.

FAQ

What is the main difference between AlexNet and LeNet?

AlexNet is far deeper (8 layers versus 7 for LeNet-5’s layer count, but with ~61M versus ~60K parameters). It uses ReLU, dropout, and GPU training on ImageNet’s 1.2M images. LeNet used tanh and trained on small digit datasets.

Why did VGG use only 3×3 filters?

Stacked 3×3 convolutions match the receptive field of larger kernels with fewer parameters and more non-linear layers. Three 3×3 layers replace one 7×7 and cut weights by about 45%.

What problem do ResNet skip connections solve?

They solve the degradation problem, where deeper plain networks train worse. The shortcut lets gradients flow directly, so layers learn a residual F(x) instead of a full mapping.

Is EfficientNet better than ResNet?

Per parameter and per FLOP, yes. EfficientNet-B0 reaches ~77% top-1 with 5.3M parameters, while ResNet-50 needs 25.6M for similar accuracy. ResNet trains faster and has broader deployment support.