Image segmentation assigns a label to every pixel. Where detection draws a box around an object, segmentation traces its exact outline — needed for measuring areas, robotic grasping, background removal, autonomous driving (drivable surface), satellite land-use maps, and tumour delineation in medical imaging.
Parent topic: Computer Vision.
Semantic segmentation outputs an H × W class map and does not separate touching objects of the same class. Instance segmentation outputs one binary mask per object (usually only for countable "things"). Panoptic segmentation unifies them: every pixel gets a class and, for things, an instance id.
Fully convolutional networks replaced the classifier's dense layers with convolutions and upsampled the output. U-Net adds skip connections that concatenate high-resolution encoder features into the decoder, recovering fine boundaries — it remains the standard for medical and small-data problems. DeepLab uses dilated (atrous) convolutions and ASPP for multi-scale context. Transformer models such as SegFormer and Mask2Former now lead general benchmarks; Mask2Former handles semantic, instance, and panoptic tasks with one architecture.
Mask R-CNN is the classic instance approach: Faster R-CNN (see Object Detection) plus a small mask head predicting a 28×28 mask per detected box.
Meta's Segment Anything Model was trained on over a billion masks and segments arbitrary objects from point or box prompts. SAM 2 extends it to video with memory across frames. It is class-agnostic, so it is often paired with an open-vocabulary detector (e.g. Grounding DINO) for text-to-mask pipelines, or used to accelerate human annotation.
1 − 2|A∩B| / (|A|+|B|) — robust when the foreground is tiny (lesions, cracks); often combined with cross-entropy.import torch, torch.nn.functional as F
from torchvision.io import read_image
from torchvision.models.segmentation import deeplabv3_resnet50, DeepLabV3_ResNet50_Weights
weights = DeepLabV3_ResNet50_Weights.DEFAULT
model = deeplabv3_resnet50(weights=weights).eval()
img = read_image("street.jpg")
batch = weights.transforms()(img).unsqueeze(0)
with torch.no_grad():
logits = model(batch)["out"] # [1, 21, h, w]
logits = F.interpolate(logits, size=img.shape[-2:], mode="bilinear", align_corners=False)
mask = logits.argmax(1)[0] # [H, W] class ids
classes = weights.meta["categories"]
for cid in mask.unique().tolist():
print(classes[cid], f"{(mask == cid).float().mean():.1%} of pixels")
# Dice loss for binary fine-tuning
def dice_loss(logits, target, eps=1.0):
p = torch.sigmoid(logits).flatten(1); t = target.flatten(1)
inter = (p * t).sum(1)
return 1 - ((2 * inter + eps) / (p.sum(1) + t.sum(1) + eps)).mean()For custom datasets, the segmentation_models_pytorch package provides U-Net, FPN, and DeepLabV3+ with any pretrained encoder in one line.
Intersection over Union (IoU, the Jaccard index) divides the overlap of prediction and ground truth by their union. Dice (the F1 score on pixels) divides twice the overlap by the sum of the two areas. For one mask they are monotonically related, Dice = 2·IoU / (1 + IoU), so they rank single predictions identically; Dice always looks more generous. Datasets aggregate differently, though: mIoU computes IoU per class over the whole dataset (summing intersections and unions across images) and then averages over classes, so a rare class counts as much as road or sky. Averaging per-image Dice instead weights every image equally, which is why medical papers and natural-image benchmarks are not directly comparable.
Instance segmentation uses mask AP: predicted masks are matched to ground truth at IoU thresholds from 0.50 to 0.95 (COCO style), and precision-recall curves are averaged. Panoptic Quality multiplies a segmentation quality term (mean IoU of matched segments) by a recognition quality term (an F1 over segments matched at IoU above 0.5). When thin structures matter, such as cracks, vessels or road lane markings, pixel-overlap metrics hide broken connectivity, so add a boundary F-score or a topology-aware metric like clDice.
Mask R-CNN (He et al., 2017) is two-stage: a region proposal network suggests boxes, RoIAlign extracts a fixed-size feature map for each box with bilinear sampling (fixing the quantisation misalignment of the earlier RoIPool), and parallel heads predict the class, a refined box and a small binary mask. The mask is predicted per class with a sigmoid, which decouples mask shape from classification. Its weakness is resolution: a 28×28 mask upsampled to a large object produces blobby edges, which later work such as PointRend addressed by refining uncertain boundary points.
Query-based models such as MaskFormer and Mask2Former drop anchors and proposals. A transformer decoder holds a fixed set of learned queries; each query predicts a class and a mask embedding, and the mask is the dot product of that embedding with per-pixel features. Training uses bipartite (Hungarian) matching between queries and ground-truth segments, as in DETR. Because "one query, one segment" covers both countable things and amorphous stuff, the same architecture handles semantic, instance and panoptic segmentation. Real-time systems often still use single-stage designs such as YOLO segmentation variants (see YOLO), which add a prototype-mask branch to a detector.
| Situation | Reasonable starting point | Why |
|---|---|---|
| Small dataset, one or few classes, fine boundaries | U-Net with a pretrained encoder | Skip connections preserve detail; trains well on hundreds of images |
| Medical volumes | nnU-Net | Self-configuring, strong baseline across many challenges |
| Many classes, scene parsing | SegFormer or Mask2Former | Global context from attention, strong on Cityscapes / ADE20K-type data |
| Counting or separating touching objects | Mask R-CNN or Mask2Former (instance) | Semantic masks merge touching instances |
| Edge device, real time | Lightweight encoder (MobileNet-class) + simple decoder, or YOLO-seg | Latency budget dominates |
| No labels yet, arbitrary objects | SAM / SAM 2 with prompts, then fine-tune a small model on corrected masks | Turns annotation into click-and-correct |
Practical training notes: segmentation is memory hungry because outputs are full resolution, so batch sizes are small; prefer GroupNorm or frozen BatchNorm statistics when batches drop below about eight images. Train on random crops and run inference on full images or overlapping tiles. Report per-class IoU, not just the mean, because a model can score a respectable mIoU while failing completely on the one rare class the application cares about. Post-processing such as removing tiny connected components or filling holes is often worth a few points and should be tuned on validation data.
Use cases: background removal and portrait mode, agricultural field and crop mapping from satellite imagery, defect inspection on production lines, document layout analysis, autonomous driving free-space estimation, and organ and tumour contouring in medical imaging.
Further reading: Long, Shelhamer & Darrell, "Fully Convolutional Networks for Semantic Segmentation" (2015); Ronneberger et al., "U-Net" (2015); Chen et al., DeepLabv3+ (2018); He et al., "Mask R-CNN" (2017); Kirillov et al., "Panoptic Segmentation" (2019); Cheng et al., "Mask2Former" (2022); Kirillov et al., "Segment Anything" (2023).