Vision-language models connect images and text in one representation. CLIP (Contrastive Language-Image Pre-training, OpenAI 2021) trains an image encoder and a text encoder so that matching image-caption pairs have similar embeddings. The result supports zero-shot classification, text-to-image and image-to-image search, and serves as the visual front-end for many multimodal LLMs and text-to-image generators.
Parent topic: Computer Vision.
For a batch of N pairs, both encoders produce L2-normalised embeddings and the model computes the N × N matrix of cosine similarities, multiplied by a learned temperature. The loss is cross-entropy where the correct "class" for image i is text i, averaged with the same loss along columns. Large batches matter because they supply more negatives. SigLIP replaces the softmax with a per-pair sigmoid loss, which removes the need for a global normalisation across the batch. OpenCLIP provides open reproductions trained on LAION data.
Turn each class name into a prompt ("a photo of a {label}"), embed the prompts once, and pick the class whose text embedding is closest to the image embedding. Averaging several prompt templates per class (prompt ensembling) usually helps. Accuracy is strong on everyday categories and weaker on fine-grained or specialised domains (medical, satellite, industrial), where a linear probe or fine-tuning on CLIP features is better.
import torch
from PIL import Image
from transformers import CLIPModel, CLIPProcessor
name = "openai/clip-vit-base-patch32"
model = CLIPModel.from_pretrained(name).eval()
proc = CLIPProcessor.from_pretrained(name)
labels = ["cat", "dog", "car", "pizza"]
prompts = [f"a photo of a {l}" for l in labels]
image = Image.open("photo.jpg").convert("RGB")
inputs = proc(text=prompts, images=image, return_tensors="pt", padding=True)
with torch.no_grad():
probs = model(**inputs).logits_per_image.softmax(dim=-1)[0]
for l, p in sorted(zip(labels, probs.tolist()), key=lambda x: -x[1]):
print(f"{l:6s} {p:.3f}")
# text-to-image search: embed a gallery once, then query with text
@torch.no_grad()
def embed_images(paths):
imgs = [Image.open(p).convert("RGB") for p in paths]
e = model.get_image_features(**proc(images=imgs, return_tensors="pt"))
return e / e.norm(dim=-1, keepdim=True)
@torch.no_grad()
def embed_text(q):
e = model.get_text_features(**proc(text=[q], return_tensors="pt", padding=True))
return e / e.norm(dim=-1, keepdim=True)
gallery = ["a.jpg", "b.jpg", "c.jpg"]
G = embed_images(gallery)
scores = (G @ embed_text("a red bicycle leaning on a wall").T).squeeze(1)
print(gallery[int(scores.argmax())])In production, store the image embeddings in a vector database — the same pattern as RAG embeddings.
Models such as LLaVA connect a CLIP-family vision encoder to a decoder-only LLM: patch embeddings are projected into the LLM's token space and interleaved with text. This enables captioning, visual question answering, chart and document reading. Text-to-image diffusion models use CLIP-style text encoders for conditioning — see Generative Models.
Zero-shot classification is just a softmax over cosine similarities. Suppose an image embedding and three prompt embeddings ("a photo of a cat", "a photo of a dog", "a photo of a car") are all L2-normalised, and the cosine similarities come out as 0.31, 0.26 and 0.12. CLIP multiplies them by its learned logit scale, which in the released OpenAI checkpoints is capped at 100 (the temperature is stored as a log value and clipped during training). The logits become 31, 26 and 12. A softmax over those gives roughly 0.993 for cat, 0.007 for dog and effectively zero for car.
Two lessons follow. First, raw cosine values in CLIP space are small and compressed — a "high" score might be 0.3, not 0.9 — so absolute thresholds chosen by intuition are almost always wrong. Second, the softmax only ranks the labels you supplied. If the image shows a horse, the model will still confidently pick one of cat, dog or car. For open-set problems, add a "something else" class, compare against a calibrated similarity threshold, or keep the raw scores rather than the softmax.
For retrieval the arithmetic is the same without the softmax: embed the query text once, take the dot product with every stored image embedding (they are normalised, so dot product equals cosine), and return the top k. Image-to-image search works by embedding a query image instead of text, which is often more reliable for near-duplicate detection.
Zero-shot performance is a starting point, not a ceiling. The options above trade labelled data and compute for accuracy:
| Method | What is trained | Data needed | Notes |
|---|---|---|---|
| Zero-shot + prompt ensembling | Nothing | None (a small labelled set to evaluate) | Fast baseline; try 5–20 templates per class and average embeddings |
| Linear probe | A logistic-regression head on frozen image embeddings | Tens to thousands of examples per class | Cheap and robust; the CLIP paper reported linear probes as a strong baseline across many datasets |
| Prompt learning / adapters (CoOp, CLIP-Adapter, LoRA) | Learned context vectors or small adapter layers | Few-shot (1–16 per class is common in papers) | Keeps most of the pretrained knowledge; can overfit base classes |
| Full fine-tuning | All encoder weights | Large in-domain sets | Best in-domain accuracy, but can erode zero-shot robustness; weight averaging with the original (WiSE-FT) mitigates this |
A practical rule: start with prompt ensembling, measure on a few hundred labelled images, then fit a logistic regression on the frozen embeddings. Only reach for fine-tuning if the probe plateaus well short of your target. For document-heavy images (invoices, forms) a CLIP model is usually the wrong tool — OCR-based document AI or a multimodal LLM reads text far better.
The two families solve different problems. Dual-encoder models (CLIP, SigLIP) produce one vector per image and per text, so millions of items can be pre-embedded and searched in milliseconds. Generative multimodal LLMs read the image every time a question is asked, which allows open-ended answers but costs far more per query and cannot be indexed ahead of time. Many systems use both: a dual encoder to retrieve candidates, then a multimodal LLM to answer over the few that were retrieved.
Typical real-world uses include photo-library search ("dog on a beach at sunset"), deduplication of user uploads, content moderation triage, auto-tagging product catalogues, and acting as the text-conditioning encoder for diffusion models.