ResNet (He et al., 2015) and EfficientNet (Tan and Le, 2019) are two of the most influential convolutional network families. ResNet solved the optimization problem of depth with residual (skip) connections, making networks of 50, 101 and 152 layers trainable. EfficientNet solved the scaling problem: given more compute, how should you grow depth, width and input resolution together? Both remain widely used backbones for classification, detection and transfer learning.
Before ResNet, adding layers to a plain stack of convolutions eventually made training error worse, not just test error. That is surprising: a deeper network could copy the shallow one and set the extra layers to the identity, so its optimum is at least as good. The solver simply could not find it — learning an identity mapping through several nonlinear layers is hard, and gradients degrade as they pass through many multiplications.
Instead of asking a stack of layers to learn a mapping H(x) directly, a residual block learns F(x) = H(x) − x and outputs F(x) + x. When the shape changes (stride 2 or more channels), the shortcut becomes a 1x1 convolution with matching stride. The identity path makes the block default to "do nothing" and gives gradients an unattenuated route back to early layers.
Residual connections turned out to be far more general than CNNs: every Transformer layer wraps attention and its MLP in exactly this x + F(x) pattern.
Standard depths are ResNet-18 and -34 (basic two-conv blocks) and ResNet-50, -101 and -152 (bottleneck blocks), arranged in four stages with channel widths 64/128/256/512 (times 4 for bottleneck outputs). ResNet-50 has roughly 25 million parameters and is the most common benchmark backbone. Notable descendants:
EfficientNet's base network (B0) was found by neural architecture search optimizing accuracy and FLOPs. Its block is MobileNetV2's inverted residual: expand channels with a 1x1 conv, apply a cheap depthwise convolution (one filter per channel), recalibrate channels with a squeeze-and-excitation module (global pool, tiny MLP, sigmoid gates), and project back down with a linear 1x1 conv. The skip connects the narrow ends. Depthwise separable convolutions cut FLOPs dramatically compared with full 3x3 convolutions at the same width.
Scaling only one dimension saturates quickly: a very deep but narrow network, or a very wide but shallow one, stops improving. EfficientNet scales all three with one coefficient φ: depth d = αφ, width w = βφ, resolution r = γφ, subject to α·β2·γ2 ≈ 2 so FLOPs roughly double per unit of φ. The paper found α=1.2, β=1.1, γ=1.15 by small grid search on B0, then produced B1 through B7 by increasing φ. Width and resolution appear squared because FLOPs grow quadratically in both.
EfficientNetV2 (2021) revisited the design for training speed: it uses fused-MBConv (regular 3x3 conv instead of expand + depthwise) in early stages, where depthwise convolutions are slow on accelerators, and progressive resizing during training.
import torch
import torch.nn as nn
from torchvision import models
# ResNet-50: replace the final fully connected layer
w = models.ResNet50_Weights.IMAGENET1K_V2
resnet = models.resnet50(weights=w)
resnet.fc = nn.Linear(resnet.fc.in_features, 5) # 5 target classes
# EfficientNet-B0: classifier is Sequential(Dropout, Linear)
e = models.EfficientNet_B0_Weights.IMAGENET1K_V1
effnet = models.efficientnet_b0(weights=e)
effnet.classifier[1] = nn.Linear(effnet.classifier[1].in_features, 5)
preprocess = e.transforms() # resize/crop/normalize matching pretraining
# Stage 1: train only the new head
for p in effnet.features.parameters():
p.requires_grad = False
opt = torch.optim.AdamW(filter(lambda p: p.requires_grad, effnet.parameters()), lr=1e-3)
# Stage 2 (after a few epochs): unfreeze and fine-tune with a lower LR
for p in effnet.features.parameters():
p.requires_grad = True
opt = torch.optim.AdamW([
{"params": effnet.features.parameters(), "lr": 1e-5},
{"params": effnet.classifier.parameters(), "lr": 1e-4},
], weight_decay=1e-4)
x = torch.randn(2, 3, 224, 224)
print(resnet(x).shape, effnet(x).shape) # torch.Size([2, 5]) twiceThe timm library offers a much larger catalog (ResNet-RS, EfficientNetV2, ConvNeXt) with consistent APIs via timm.create_model(name, pretrained=True, num_classes=k).
transforms().The design choices in both families are easiest to understand by counting weights (biases and BatchNorm parameters ignored).
2 × 9 × 256 × 256 ≈ 1.18M weights. The bottleneck version (1x1 down to 64, 3x3 at 64, 1x1 back to 256) needs 16,384 + 36,864 + 16,384 = 69,632 — roughly 17 times fewer. That saving is what makes ResNet-50 and deeper practical.9 × 128 × 128 = 147,456 weights. Depthwise (9 × 128 = 1,152) plus pointwise (128 × 128 = 16,384) totals 17,536, about 8.4 times fewer. In general the ratio is 1/Cout + 1/k2, so for 3x3 kernels the saving approaches 9x as channels grow.α = 1.2, β = 1.1, γ = 1.15, one step of φ multiplies FLOPs by about 1.2 × 1.12 × 1.152 ≈ 1.92, close to the intended doubling.The same arithmetic explains why FLOP counts mislead on GPUs: the depthwise layer does very little arithmetic per byte of activation it reads, so it is limited by memory bandwidth, while a dense 3x3 conv keeps tensor cores busy. Parameter count, FLOPs and measured latency are three different numbers; report the one that matches your constraint.
Write a stack of residual blocks as xl+1 = xl + Fl(xl). The Jacobian of each block is I + ∂Fl/∂xl, and the gradient reaching an early layer is a product of such terms. Expanding the product yields a sum that always includes the identity, so some gradient reaches every layer undiminished regardless of depth. With pre-activation blocks the shortcut is a pure identity, and He et al. (2016) showed this cleaner path trains networks of 1,000 layers.
Veit, Wilber and Belongie (2016) offered a complementary view: a residual network with n blocks behaves like an ensemble of 2n paths of different lengths, and most of the gradient during training flows through relatively short paths. Deleting a single block from a trained ResNet degrades accuracy gracefully, whereas deleting a layer from a plain VGG-style network breaks it. Two practical consequences follow: zero-initializing the last BatchNorm scale in each residual branch (so every block starts as the identity) is a cheap stabilizer used in many training recipes, and stochastic depth — randomly dropping whole residual branches during training — works as a regularizer for deep models.
Training only the new head first keeps large, random-initialized gradients from wrecking pretrained features. Unfreezing later with a smaller backbone learning rate (discriminative learning rates) lets the features adapt to the new domain. How much to unfreeze depends on how far the target data is from ImageNet: for natural photos, fine-tuning the last stage or two is often enough; for medical images, satellite tiles or documents, full fine-tuning usually wins given enough labeled data. Augmentation (random resized crops, flips, color jitter, and for longer runs RandAugment or mixup) and label smoothing carry over from the original training recipes.
Evaluate on a split that matches deployment, including resolution, compression and camera differences, and track per-class metrics: a backbone change that raises top-1 accuracy can still hurt a rare but important class. For deployment, both families export to ONNX and TensorRT; fold BatchNorm into the preceding convolutions and benchmark int8 quantization, which tends to be easier on ResNets than on depthwise-heavy EfficientNets.