DALL-E is OpenAI's family of text-to-image models. The three generations used quite different architectures — an autoregressive transformer, then a CLIP-guided diffusion system, then a diffusion model trained on heavily recaptioned data — which makes the series a useful map of how the field evolved. This page focuses on what is publicly documented in the papers and system cards; OpenAI has not released weights for DALL-E 2 or 3.
Each generation kept the same product idea — type a description, get an image — while swapping the generative engine. The progression mirrors the broader shift from autoregressive token models to diffusion, and from engineering the model to engineering the training data.
The original DALL-E (Ramesh et al., 2021) compressed each 256×256 image into a 32×32 grid of discrete codes (an 8192-entry vocabulary) using a discrete VAE. It then concatenated up to 256 BPE text tokens with the 1024 image tokens and trained a 12-billion-parameter decoder-only transformer to predict the sequence left to right, exactly like a language model. At inference it sampled many candidates and reranked them with CLIP.
Strengths were compositional creativity ("an armchair in the shape of an avocado"); weaknesses were low resolution, blurriness from the dVAE, and slow token-by-token sampling.
DALL-E 2 (Ramesh et al., 2022, "Hierarchical Text-Conditional Image Generation with CLIP Latents") is built on CLIP. A prior (diffusion or autoregressive; diffusion worked better) maps a CLIP text embedding to a plausible CLIP image embedding. A diffusion decoder, derived from GLIDE, generates a 64×64 image conditioned on that embedding, and two diffusion upsamplers raise it to 1024×1024.
The CLIP image embedding captures semantics and style but discards some details, which made DALL-E 2 good at variations and interpolation but weaker at binding attributes to objects (e.g. "a red cube on a blue sphere") and at rendering text.
The DALL-E 3 paper ("Improving Image Generation with Better Captions") argued that prompt-following was limited mainly by noisy training captions. OpenAI trained a captioner to produce long, descriptive synthetic captions and retrained the image model largely on them. Because users write short prompts, deployment uses an LLM (in ChatGPT) to expand prompts into the detailed style the model was trained on. DALL-E 3 also improved text rendering within images and added safety mitigations around public figures and artist styles.
from openai import OpenAI
import base64
client = OpenAI() # reads OPENAI_API_KEY from the environment
resp = client.images.generate(
model="dall-e-3",
prompt="An isometric cutaway illustration of a small robotics lab, soft lighting",
size="1024x1024", # dall-e-3 also supports 1792x1024 and 1024x1792
quality="standard", # or "hd"
n=1, # dall-e-3 accepts only n=1
response_format="b64_json",
)
print(resp.data[0].revised_prompt) # the automatically expanded prompt
with open("lab.png", "wb") as f:
f.write(base64.b64decode(resp.data[0].b64_json))Check OpenAI's current model list before building on this: newer image models (e.g. gpt-image-1) are exposed through the same endpoint, and older models get deprecated.
revised_prompt may differ from yours; log it for reproducibility.DALL-E 1 treats generation as language modelling over a joint sequence: the transformer attends over text and image tokens with causal masking, and sampling 1024 image tokens means 1024 sequential forward passes. Its discrete VAE downsamples by 8 in each direction, so fine detail is limited by what a 32×32 code grid can express. Reranking many samples with CLIP was a large part of the perceived quality; the raw samples were much more variable.
DALL-E 2 splits the problem. Because CLIP was trained contrastively to place matching images and captions close together, a CLIP image embedding is a compact summary of "what the picture is about". The prior learns to produce such an embedding from text; the decoder learns to invert CLIP, producing an image from its embedding. Two consequences follow. Variations become easy (encode an image with CLIP, decode again with different noise), and interpolation between two embeddings produces smooth blends. But anything CLIP's embedding does not encode well, such as which colour belongs to which object, how many items there are, or the spelling of text, is hard for the decoder to recover.
DALL-E 3's paper discusses the captioning work rather than a full architecture description; it describes a diffusion model with a T5 text encoder and an image latent space, but the emphasis is that descriptive synthetic captions, mixed with a small fraction of original captions to avoid overfitting to the captioner's style, drove the improvement in prompt following.
Both DALL-E 2's decoder and most later text-to-image diffusion models rely on classifier-free guidance (Ho & Salimans, 2021). During training the conditioning is randomly dropped, for example 10% of the time, so the same network learns both a conditional and an unconditional noise prediction. At sampling time both are computed and the result is pushed further in the direction of the prompt: ε = εuncond + s · (εcond − εuncond). A scale s of 1 is plain conditional sampling; larger values give images that match the prompt more closely and look sharper, at the cost of diversity and, at very high scales, oversaturated colours. The GLIDE paper (Nichol et al., 2021) found classifier-free guidance preferred by human raters over CLIP guidance, which is why DALL-E 2's decoder used it. Hosted APIs fix this setting for you; open models such as Stable Diffusion expose it as the "CFG scale".
| Aspect | DALL-E 1 | DALL-E 2 | DALL-E 3 |
|---|---|---|---|
| Generator | 12B autoregressive transformer | CLIP prior + diffusion decoder + upsamplers | Diffusion model trained on recaptioned data |
| Native output | 256×256 | 1024×1024 | 1024×1024 and two wide/tall sizes |
| Strength | Novel compositions | Variations, interpolation, inpainting | Prompt following, legible short text |
| Weakness | Blurry, slow sampling | Attribute binding, text in images | No seed, limited fine control |
| Public access | Research only (no product) | API (edits and variations endpoints) | ChatGPT and API |
Typical uses are illustration for articles and slides, marketing concept images, storyboard and UI mock-up imagery, educational diagrams with simple labels, and quick visual brainstorming inside ChatGPT, where conversation makes iterative refinement natural. For brand-exact assets, consistent characters across a series, or batch generation with tight control, teams generally move to open models with fine-tuning and conditioning, or to dedicated design tools. Midjourney is often chosen when stylised aesthetics matter more than automation.
Text-to-image quality is measured along several axes. FID compares feature statistics of generated and real images and captures realism and diversity but not prompt alignment. CLIP score measures similarity between the prompt and the image in CLIP space, a cheap but imperfect proxy for alignment, and it is biased toward what CLIP itself understands. Compositional benchmarks probe counting, spatial relations and attribute binding with templated prompts. In the end, side-by-side human preference studies remain the main evidence in the DALL-E papers.
OpenAI's system cards for DALL-E 2 and 3 describe safety layers outside the model: filtering of training data, prompt and output classifiers, refusals for requests involving public figures by name, and declining requests in the style of living artists. In 2024 OpenAI began attaching C2PA content credentials metadata to DALL-E 3 images so tools can identify their origin; such metadata is easily stripped, so it supports but does not guarantee provenance. In production, store the original prompt, the revised prompt and the model name with every generated asset.
Further reading: Ramesh et al., "Zero-Shot Text-to-Image Generation" (2021, arXiv:2102.12092); Ramesh et al., "Hierarchical Text-Conditional Image Generation with CLIP Latents" (2022, arXiv:2204.06125); Betker et al., "Improving Image Generation with Better Captions" (OpenAI, 2023); Radford et al., "Learning Transferable Visual Models From Natural Language Supervision" (CLIP, 2021); Nichol et al., "GLIDE" (2021); Ho & Salimans, "Classifier-Free Diffusion Guidance" (2021).