Multimodal models connect text with images, audio and video. This page explains how each modality is turned into vectors, the main fusion strategies, contrastive training as in CLIP, how vision-language models bolt a vision encoder onto an LLM, and where they fail.
A modality is a kind of data: text, images, audio, video, sensor readings, tables. A multimodal model takes in, or produces, more than one modality and relates them — answering a question about a photo, transcribing and summarising a meeting recording, generating an image from a description, or retrieving video clips with a text query.
The central difficulty is that modalities have very different raw forms: a sentence is a short sequence of discrete tokens, an image is a grid of millions of continuous values, audio is a long one-dimensional waveform. Multimodal architectures differ mainly in how they turn each modality into vectors and where they let those vectors interact, a choice called fusion.
Most real information is multimodal: invoices are scanned images containing tables; medical records combine images, notes and lab values; customer support includes screenshots. Text-only models must rely on lossy conversion (OCR, captions) first. Multimodal models can read the original signal, which improves accuracy on charts, layouts and visual details, and they enable new products such as visual search, accessibility descriptions and voice assistants that respond in real time. Most frontier assistant models are now natively multimodal, accepting images and often audio alongside text.
Text is split into tokens and embedded. Images are usually cut into fixed-size patches (for example 14x14 or 16x16 pixels), each flattened and projected into a vector, then processed by a Vision Transformer. Audio is converted to a log-mel spectrogram and encoded with a convolutional front end plus a Transformer, as in Whisper. Video is treated as a sequence of frames, often subsampled. After encoding, every modality becomes a sequence of vectors, which is what lets the Transformer act as a common backbone.
Early fusion combines raw or lightly processed inputs and lets one model learn everything jointly; it captures fine interactions but needs aligned data and lots of compute. Late fusion runs a separate model per modality and merges their predictions, for example by averaging scores; it is simple and robust to a missing modality but cannot learn cross-modal detail. Intermediate fusion encodes each modality separately and then mixes their representations with cross-attention or by interleaving tokens in one Transformer; this is the dominant design for vision-language models.
CLIP (Radford et al., 2021) trains an image encoder and a text encoder together on hundreds of millions of image-caption pairs. For a batch of N pairs it computes all NxN cosine similarities and applies a cross-entropy loss that pushes the N correct pairs (the diagonal) up and all others down. The result is a shared embedding space where "a photo of a dog" lies near dog photos, enabling zero-shot classification and text-to-image retrieval with no task-specific training.
Models such as LLaVA connect a pretrained vision encoder to a pretrained LLM through a small projector that maps image features into the LLM's token-embedding space. The image becomes a few hundred "visual tokens" placed in front of the text prompt, and the LLM generates the answer. Training usually has two stages: align the projector on image-caption data, then instruction-tune on visual question-answer conversations. Flamingo used gated cross-attention layers instead; some newer models are trained natively on interleaved image and text tokens from the start.
Using Hugging Face Transformers (pip install transformers torch pillow requests):
import requests, torch
from PIL import Image
from transformers import CLIPModel, CLIPProcessor
model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
proc = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
url = "http://images.cocodataset.org/val2017/000000039769.jpg" # two cats on a couch
image = Image.open(requests.get(url, stream=True).raw)
labels = ["a photo of a cat", "a photo of a dog", "a photo of a car"]
inputs = proc(text=labels, images=image, return_tensors="pt", padding=True)
with torch.no_grad():
out = model(**inputs)
probs = out.logits_per_image.softmax(dim=1)[0]
for label, p in zip(labels, probs):
print(f"{p:.3f} {label}")No training happened: the class names are the classifier. Prompt wording matters ("a photo of a ..." works better than a bare word), and averaging several templates per class usually helps. For retrieval, store model.get_image_features(...) vectors in a vector index and search with get_text_features(...).
Public benchmarks give a first filter: MMMU for expert-level multi-discipline questions, DocVQA and ChartQA for documents and charts, TextVQA for reading text in scenes, and retrieval recall on COCO or Flickr30k for embedding models. They rarely match your own data, so build a small labelled set from real inputs — your invoices, your product photos, your call recordings — and measure exact-match or rubric scores on it before choosing a model.
Deployment adds its own constraints. Images inflate prompt size and latency, so resize to the smallest resolution that keeps the needed detail and cache embeddings for anything you search repeatedly. For document pipelines, compare a vision-language model reading page images against a classic OCR-plus-LLM pipeline on accuracy, cost and speed; each wins on different document types. Log the inputs that fail so you can grow the evaluation set over time.
| Approach | Example models | Output | Strengths | Limitations |
|---|---|---|---|---|
| Contrastive dual encoder | CLIP, SigLIP | Embeddings | Fast retrieval, zero-shot labels, index once | No generation; weak at counting and spatial relations |
| Encoder + projector + LLM | LLaVA family | Text | Reuses strong LLMs, cheap to train | Fixed image resolution limits fine detail |
| Cross-attention into LM | Flamingo | Text | Interleaved images and text, few-shot | More parameters to train |
| Native multimodal | Frontier assistant models | Text, sometimes audio or images | Best overall quality | Closed or very large |
| Speech models | Whisper | Text from audio | Robust transcription, many languages | Audio only; may hallucinate on silence |
| Text-to-image diffusion | Stable Diffusion | Images | High-quality generation | Text understanding limited by text encoder |