How Diffusion Models Generate Images

Every module before this one assumed the goal was generating text. Stable Diffusion, DALL-E, and Midjourney generate images with a completely different mechanism - one that has nothing to do with predicting the next token. Instead, a diffusion model learns to reverse a process of progressively destroying an image with noise. Understanding that one idea unlocks the entire family.

Training a diffusion model needs images at every stage of being progressively noised. The forward process is a fixed formula - no learning involved - that takes a clean image x0 and a timestep t, and produces a noised version x_t directly, in one step, with no need to simulate t small noise-additions in sequence.

Generation runs the forward process backward: starting from pure noise, and repeatedly removing a little of it until a coherent image emerges. A trained model's actual job is narrow - look at a noisy x_t and predict what noise was added - and that predicted noise plugs directly into a formula that nudges x_t toward x_{t-1}, slightly less noisy. Chain enough of these steps (real models take anywhere from a handful to thousands, depending on the sampler) and static becomes an image.

A text-to-image model can predict noise two ways for the same noisy image: conditioned on the prompt, or completely unconditioned (ignoring it). Classifier-Free Guidance runs both and extrapolates past the conditional prediction, in the direction away from the unconditional one - exaggerating exactly what the prompt contributed. This is the "CFG scale" slider in every Stable Diffusion-style UI.