Understanding Stable Diffusion

Posted Nov 21, 2023 Updated Jul 6, 2026

By cornpip

2 min read

This post was migrated from Tistory. You can find the original here.

* Blue items are things to study later

Stable Diffusion

forward diffusion - Gaussian noise is gradually added to an image until only random noise remains. Eventually the image becomes unidentifiable.

reverse diffusion - This gradually undoes forward diffusion. If the model was trained on dogs and cats, the reversal process will drift toward producing a dog or a cat.

Stable Diffusion Architecture

Latent Space

Using sizes like 512x512 or 1024x1024 directly makes computation take far too long. So

the image is compressed into a lower-dimensional latent space before the computation is carried out. (What exactly does “lower-dimensional” mean here?)

Anyway, the image gets compressed, but can we say the latent image doesn’t lose essential information?

This is possible based on the fact that natural images have a high degree of regularity (the manifold hypothesis).

For example, in a typical human body structure, the face has a specific spatial relationship between the eyes, nose, and mouth, and the arms are connected to the torso.

Image resolution is reflected in the size of the latent image tensor.

At 512x512, the latent image size is 4x64x64 (why these particular numbers?)

At 768x512, the latent image size is 4x96x64.

So generating a larger image takes longer and requires more VRAM.

VAE (Variational Auto Encoder)

This is the technique (role) responsible for compressing an image into latent space and reconstructing it.

Even though natural images are said to have high regularity, information is still lost. As a result, fine details are missed, but a fine-tuned VAE decoder can restore those lost details. (What exactly is being fine-tuned in the VAE decoder?)

U-Net

Predicts and removes the added noise.

In this process, the number of steps and how much noise to remove at each step are predetermined; the distribution that determines this is called the noise schedule. (It’s also possible to remove the same amount at each step.)

Once the noise has been removed from the latent image this way, restoring it with the VAE decoder produces the final image.