Skip to content
GenAI Learn/Diffusion Models & Multimodal Generation
Browsing as a guest. Sign in to save your progress and earn XP as you complete chapters.

How Diffusion Models Generate Images

5 min read

You'll learn to

  • -Explain the forward process as a blend of picture and noise, and compute one noising by hand
  • -Show why predicting the noise is the same as predicting the clean picture
  • -Describe one reverse denoising step and why many small steps are used
  • -Explain classifier-free guidance and the CFG scale slider, with numbers

Every module before this one assumed the goal was generating text. Stable Diffusion, DALL-E and Midjourney generate images by a completely different mechanism, one that has nothing to do with predicting the next token. A diffusion model learns to reverse a process of gradually destroying a picture with noise. Understand that one idea and the whole family makes sense.

An Analogy: Static on an Old TV

Tune an old television between channels and you see static, random noise. Imagine a picture slowly being buried in that static, frame by frame, until nothing of it is left. Now imagine a skilled restorer who has watched this happen thousands of times, and who can look at any half-buried frame and say which part of it is static. Run that skill in reverse, again and again, and you can start from pure static and uncover a picture.

  • -The clean picture is x0.
  • -The random static is the noise.
  • -A frame partly buried in static is the noisy image x_t, where t says how far along the burying we are.
  • -The restorer's skill, looking at a noisy frame and saying which part is static, is the model's whole job.
  • -Repeating that skill many times, removing a little static each time, is generation.

The Forward Process: Burying the Picture on a Schedule

Training needs pictures at every stage of burial. The forward process is a fixed recipe, with no learning involved. It blends the clean picture with fresh noise. A number called alpha-bar, between 0 and 1, says how much clean signal is left: 1 means none lost yet, 0 means everything is gone.

DDPM Forward Process
xₜ = √ᾱₜ · x₀ + √(1 − ᾱₜ) · noise
A blend of the clean value and the noise. alpha-bar (ᾱ) near 1 leaves mostly picture. alpha-bar near 0 leaves mostly noise. DDPM stands for Denoising Diffusion Probabilistic Model.

Take one pixel value x0 = 0.8 and noise = -0.5. With alpha-bar = 0.9 the result is 0.949 x 0.8 + 0.316 x (-0.5) = 0.60, mostly still the picture. At 0.5 it is 0.21, an even mix. At 0.01 it is -0.42: almost pure noise. The last column of the output below is the key insight: whatever the stage, if you know the noise you can recover the original exactly.

Forward diffusion on one value, and recovering the clean value when the noise is known

Here is the same idea on a whole picture. Slide the signal down to bury a small face in noise, see it come back when the exact noise is removed, and then see what happens when the model's guess is slightly off.

Burying a Picture in Noise, Then Digging It Out

Diffusion destroys a picture with noise on a fixed schedule, then trains a model to guess the noise so it can be taken away again. Slide the signal down to bury the picture, then make the model's noise guess less accurate.

0.50

1 means none lost yet, 0 means everything is gone.

0%
The clean picture
Buried in noise
Noise guess removed

One pixel: 0.8 plus noise -0.5 becomes

0.21

matches the chapter at alpha-bar 0.9, 0.5 and 0.01

A wrong guess is multiplied by

1.0x

sqrt(1 - alpha-bar) / sqrt(alpha-bar)

With 50% of the signal left, the picture is badly buried, yet subtracting the exact noise recovers it perfectly. The model's job is only to guess that noise.

The Reverse Process: Learning to Undo It

That recovery trick shows why the model's job is narrow and learnable. Given a noisy picture and the stage t, it only has to predict the noise inside it. Subtract the predicted noise and you have an estimate of the clean picture. In practice the prediction is imperfect, so the sampler removes only a small amount at a time and repeats. Real models take anywhere from a handful to thousands of steps, depending on the sampler.

DDPM Reverse Step
xₜ₋₁ = (xₜ − (βₜ/√(1−ᾱₜ))·ε̂) / √αₜ + √βₜ · z
ε̂ is the model's predicted noise. beta and alpha are fixed numbers from the noise schedule. z is a small fresh random nudge, usually set to zero on the final step so the last image is not jittered.

Run one step with z set to zero. The noisy values [1.1179, 1.5821] come from a clean picture [1, 2]. After one reverse step they become [1.0575, 1.7886]: noticeably closer to [1, 2], but not there yet. Chain enough steps and static becomes an image.

One reverse denoising step, given a predicted noise

Classifier-Free Guidance: Making the Model Follow the Prompt

A text-to-image model can predict the noise in two ways for the same noisy picture: reading the prompt, or ignoring it. Think of two advisers, one who has heard your request and one who has not. The difference between their advice is exactly what your request contributed. Classifier-free guidance (CFG) takes the prompt-aware answer and pushes further in that direction, by a multiple you choose.

Classifier-Free Guidance
guided = uncond + scale · (cond − uncond)
scale = 1 gives the plain prompt-aware prediction. scale = 0 ignores the prompt. Typical user interfaces use 7 to 15, far past 1, which is the CFG scale slider in every Stable Diffusion-style tool.
Classifier-free guidance: exaggerating the prompt's effect

Turning the scale up makes the image follow the prompt more literally, at the cost of variety and, at extremes, natural-looking results. That trade-off is why the slider exists.

The demo below uses the same two predictions as the code. Move the guidance scale and watch the guided prediction slide along the line from the no-prompt prediction, through the with-prompt one, and beyond it.

The CFG Scale Slider, With Numbers

A text-to-image model predicts the noise twice: once without the prompt and once with it. Guidance starts from the first prediction and moves away from it, in the direction the prompt pulled, by a multiple you choose.

3.0
no promptwith promptguided

No prompt

[0, 1]

With prompt

[1, 0]

Prompt's effect

[1, -1]

Guided

[3.0, -2.0]

At scale 3.0 the guided prediction is [3.0, -2.0]: the prompt's effect, [1, -1], multiplied by 3.0 and added to the no-prompt prediction. Above 1 the prompt is exaggerated. Everyday interfaces use roughly 7 to 15; higher scales follow the prompt more strongly but at the cost of variety.

AI Lab's Levels 68-70 (Forward Diffusion, Reverse Denoising Step and Classifier-Free Guidance) implement all three formulas in numpy, over full arrays instead of single points. The math is exactly the same, vectorized.

Interview Signal is part of Pro

See a real weak answer next to a real strong one for this exact topic.

Quiz is part of Pro

Test what you just read with a short quiz, and bank the XP.

Ready to Build This?

Implement all three from scratch: Level 68 (Forward Diffusion), Level 69 (Reverse Denoising Step), and Level 70 (Classifier-Free Guidance) in AI Lab.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!