Medical imaging AI applies computer vision to radiology (X-ray, CT, MRI, ultrasound), digital pathology, ophthalmology, and dermatology. The core techniques — classification, detection, and segmentation — are the same as in general CV, but data formats, label scarcity, validation standards, and regulation are very different.
Parent topic: Computer Vision.
CT intensities are in Hounsfield units: air is about −1000, water 0, bone several hundred and above. Models usually clip to a task-specific window (for example a lung or soft-tissue window) and normalise. Voxel spacing differs between scanners, so volumes are resampled to a common spacing. MRI intensities have no absolute scale and need per-scan normalisation. Pathology whole-slide images can exceed 100,000 pixels per side and are processed as tiles, often with multiple-instance learning using only slide-level labels.
U-Net and its 3D variants dominate segmentation; nnU-Net automatically configures preprocessing, patch size, and training and is a strong default baseline. For classification, ImageNet-pretrained CNNs still transfer reasonably to X-ray, while self-supervised pretraining on unlabeled in-domain scans (and domain foundation models) helps when labels are scarce. Loss functions combine Dice and cross-entropy because lesions occupy a tiny fraction of voxels.
MONAI is the PyTorch-based framework for medical imaging; its transforms understand spacing, orientation, and intensity windows.
import torch
from monai.networks.nets import UNet
from monai.losses import DiceCELoss
from monai.inferers import sliding_window_inference
from monai.transforms import (Compose, LoadImaged, EnsureChannelFirstd, Orientationd,
Spacingd, ScaleIntensityRanged, RandCropByPosNegLabeld)
train_tf = Compose([
LoadImaged(keys=["image", "label"]), # NIfTI or DICOM series
EnsureChannelFirstd(keys=["image", "label"]),
Orientationd(keys=["image", "label"], axcodes="RAS"),
Spacingd(keys=["image", "label"], pixdim=(1.5, 1.5, 2.0), mode=("bilinear", "nearest")),
ScaleIntensityRanged(keys=["image"], a_min=-175, a_max=250, b_min=0.0, b_max=1.0, clip=True),
RandCropByPosNegLabeld(keys=["image", "label"], label_key="label",
spatial_size=(96, 96, 96), pos=1, neg=1, num_samples=4),
])
model = UNet(spatial_dims=3, in_channels=1, out_channels=2,
channels=(16, 32, 64, 128, 256), strides=(2, 2, 2, 2), num_res_units=2)
loss_fn = DiceCELoss(to_onehot_y=True, softmax=True)
opt = torch.optim.AdamW(model.parameters(), lr=1e-4)
# training step (batch from a monai DataLoader over train_tf)
def step(batch):
logits = model(batch["image"])
loss = loss_fn(logits, batch["label"])
opt.zero_grad(); loss.backward(); opt.step()
return loss.item()
# full-volume inference with overlapping patches
@torch.no_grad()
def predict(volume): # [1, 1, D, H, W]
model.eval()
return sliding_window_inference(volume, (96, 96, 96), 4, model, overlap=0.5).argmax(1)Medical images arrive as DICOM series: one file per slice for CT and MRI, each carrying pixel data plus metadata such as pixel spacing, slice thickness, patient orientation, rescale slope and intercept, and acquisition parameters. Research datasets are often converted to NIfTI volumes, which keep an affine matrix mapping voxel indices to physical millimetres. Losing or ignoring that affine is one of the most common silent bugs: a mask can be flipped left-to-right relative to its image and still train, just badly.
The standard chain is: reorient every volume to a fixed convention, resample to a common voxel spacing (for example 1.0 × 1.0 × 1.5 mm, or the median spacing of the training set, which is what nnU-Net picks), clip intensities to a window and normalise, then sample training patches. For CT, the raw stored values are converted to Hounsfield units with the rescale slope and intercept before windowing; a lung window and a soft-tissue window expose very different structures, and some models take several windows as separate input channels. MRI has no absolute intensity scale, so per-volume z-score normalisation, often restricted to the foreground, is the usual choice, sometimes preceded by bias-field correction.
Because full 3D volumes rarely fit in GPU memory, training uses patches such as 128³ voxels. Uniform random patches would mostly contain background, so samplers deliberately over-sample patches centred on foreground labels (a ratio like one-third foreground is common). At inference, a sliding window with overlapping patches and Gaussian weighting reassembles the full-volume prediction; MONAI's sliding_window_inference implements this.
| Task | Typical example | Common approach | Primary metric |
|---|---|---|---|
| Image classification | Chest X-ray finding present / absent | Pretrained 2D CNN or ViT, multi-label sigmoid head | AUROC, sensitivity at fixed specificity |
| Detection | Lung nodules on CT, fractures on X-ray | 3D or 2D detectors, FROC-oriented training | FROC, sensitivity at N false positives per scan |
| Segmentation | Organs at risk for radiotherapy, tumours | 2D/3D U-Net, nnU-Net, transformer-hybrid U-Nets | Dice, 95th-percentile Hausdorff distance |
| Whole-slide pathology | Metastasis in lymph node slides | Tile encoder plus attention-based multiple-instance pooling | Slide-level AUROC |
| Registration | Align follow-up scan to baseline | Learned deformation fields (e.g. VoxelMorph-style) | Landmark error, Dice of warped labels |
Detection metrics deserve a note: radiology detection tasks often use FROC (free-response ROC), which plots lesion-level sensitivity against the average number of false positives per scan, because a reader cares about how many false marks they must dismiss per study. For segmentation, Dice measures overlap and is insensitive to small boundary errors on large organs, while the Hausdorff distance captures the worst boundary deviation; report both. See image segmentation and object detection for the general versions of these architectures.
A model output is a probability; a clinical tool needs an operating point. Pick it on a held-out tuning set to meet a requirement such as "sensitivity at least 0.95 for triage" and then report specificity there, rather than reporting accuracy at an arbitrary 0.5 threshold. AUROC summarises ranking quality across all thresholds and is useful for comparing models, but two models with the same AUROC can behave very differently at the threshold that matters.
Positive predictive value depends on prevalence. The worked example in the diagram uses Bayes' rule: with 90% sensitivity and 90% specificity, a population with 1% prevalence gives PPV = (0.9 × 0.01) / (0.9 × 0.01 + 0.1 × 0.99) ≈ 0.083, so roughly eleven of every twelve positive flags are false alarms. That is why a model that looked excellent on an enriched research dataset can overwhelm a screening workflow. Also report calibration (does a score of 0.8 mean 80% risk?) and confidence intervals, computed by bootstrapping over patients rather than images.
Software that informs diagnosis is generally regulated as a medical device: in the United States through the FDA (510(k), De Novo or PMA pathways), in the European Union under the Medical Device Regulation, with comparable regimes elsewhere. Regulators expect a defined intended use, documented training and test data, performance on independent data, and a plan for changes after release; the FDA has published guidance on predetermined change control plans for machine-learning devices. Reporting guidelines such as CLAIM (Checklist for AI in Medical Imaging) and TRIPOD+AI help structure studies.
Privacy work starts before training. De-identify DICOM headers according to a profile (DICOM PS3.15 defines confidentiality profiles), scrub burned-in text in the pixels, and remember that facial reconstruction from head CT and MRI is possible, so defacing tools are used for brain imaging. When data cannot leave a hospital, federated learning trains across sites by sharing model updates rather than images; it reduces but does not eliminate privacy risk.
In deployment, the model typically runs behind a DICOM router or PACS integration, receives studies automatically, and writes results back as DICOM secondary captures, structured reports or segmentation objects. Monitor input distributions (scanner model, protocol, slice thickness) and the rate of positive flags, because a new scanner or protocol change is the most common cause of silent degradation. Keep a human in the loop and log disagreements; they are the best source of new training cases.
Further reading: Ronneberger, Fischer & Brox, "U-Net" (2015); Isensee et al., "nnU-Net" (Nature Methods, 2021); Litjens et al., "A survey on deep learning in medical image analysis" (2017); Zech et al., confounding in chest X-ray models across hospitals (PLOS Medicine, 2018); the MONAI documentation.