Real-ESRGAN Architecture: RRDB and High-Frequency Generation
This post was migrated from Tistory. You can find the original here.
Basic ESRGAN Architecture
Real-ESRGAN is a Super-Resolution model that trains an RRDB-based CNN generator using a GAN approach.
It turns a Low-Resolution (low-frequency) image into a High-Resolution (high-frequency) image.
Input (LR Image)
→ Conv
→ RRDB × N
→ Conv
→ Upsampling (x2 / x4)
→ Conv
→ Output (HR Image)
RRDB (Residual-in-Residual Dense Block)
Here’s what RRDB does:
LR image
→ Conv
→ RRDB blocks
→ “There should be an edge/texture like this at this location”
→ Generate high-frequency feature maps
Frequency in Images
Low frequency
Brightness variation
Large structures, contours, shapes
e.g. the overall color tone of a photo, the rough position of eyes/nose/mouth, the large silhouette of a building
High frequency
Edges
Textures
e.g. flyaway hairs, skin texture, letter outlines
In general, reducing resolution mathematically forces the removal of high-frequency components,
so a Low-Resolution image is one whose high-frequency components have already been lost.
Why Is the RRDB Structure Good at Recovering High Frequencies?
Residual, expressed as a formula:
(Residual = a skip-connection structure that adds the input to the output)
Output = Input + Δ
Here, Δ (delta) is the amount of change (= the high-frequency component) that gets added to the existing image.
In other words, RRDB is structurally designed to leave the low-frequency content in the input untouched while learning only the high-frequency content.
And the dense connections preserve
edge information from shallow layers
complex texture information from deep layers
together.
Upsampling
Upsampling isn’t the stage that creates high-frequency content — it’s the stage that spatially unfolds the high-frequency content that’s already been generated.
Feature Map (H, W, C) = RRDB output
→ Conv (C × r²)
→ PixelShuffle
→ (H×r, W×r, C)
→ Conv
PixelShuffle is an operation that rearranges pixel information stored along the channel dimension, which represents the upscale target, into the spatial dimensions (H, W).
(H, W, C×r²) → (rH, rW, C)
Is the Original (Low-Frequency Content) Preserved?
Since this model is trained via a GAN approach, if the adversarial loss is too large, even the original content can end up distorted.
Very low resolution → very large upscale factor
e.g. 64x64 → 1024x1024
In this case, the low-frequency information itself is insufficient, so the original can be altered.
If you need SR (Super-Resolution) but the original must be preserved (medical imaging, measurement, quantitative analysis, etc.),
you should either use it in a well-controlled setting or consider a non-GAN SR model.