Colorful Image Colorization (Oct 2016, ECCV)

Image Colorization Methodology
📌 The problem of colorizing a grayscale image (L channel in the CIELAB color space) into color channels (AB channels) is under-constrained, requiring data-driven solutions, typically using a CNN.
📌 The system formulates colorization as a multinomial classification problem by dividing the output space into discrete bins (size 10), moving away from standard L2 regression which averages multiple plausible modes, leading to desaturated results.
📌 A class rebalancing term is added to the training objective to counteract the natural statistics of images where most pixels are concentrated in desaturated colors, thus promoting rarer, more vibrant colors.
📌 The network architecture is based on VGG, adapted with dilated convolutions to maintain spatial resolution, and the final prediction involves an interpolation between the mean and mode of the predicted distribution for color vibrancy and spatial consistency.

Comparison with Existing Techniques
➡️ Previous nonparametric techniques relied on transferring colors from a reference image, which can fail to generalize or require user intervention.
➡️ Early parametric work used L2 regression; the proposed approach builds upon earlier 2008 classification frameworks but incorporates deeper networks and class rebalancing.
➡️ When compared against L2 regression, the full system (classification + rebalancing) yielded results qualitatively more colorful and achieved 32% in a perceptual realism test, compared to 21% for L2 regression.

Evaluation and Semantic Learning
🎭 Perceptual realism tests using Amazon Mechanical Turk showed that the system can fool participants about 50% of the time in cases where the predicted color is highly plausible, even if different from the ground truth (e.g., predicting green for a green chameleon when the ground truth was blue).
🖼️ The system demonstrates object recognition capabilities when tested on unseen data like the Macbeth chart, indicating it learns semantics rather than relying on low-level cues like chromatic aberration.
🤖 Colorization is framed as a self-supervised learning task, similar to denoising autoencoders, but here it's a cross-channel encoding where two channels are predicted from the remaining grayscale channel.

Feature Representation Transferability
⭐ By stripping colorization-specific elements, the resulting feed-forward feature extractor showed evidence of learning unsupervised semantics, with units maximally activating for categories like sky, trees, and faces.
📊 When fine-tuned on downstream tasks (classification, detection, segmentation on PASCAL VOC), the features learned via colorization were highly competitive, sometimes achieving state-of-the-art results among previous self-supervision methods (outperforming Gaussian initialization and methods like context and inpainting).
📈 Despite strong performance in self-supervision, a large gap remains between these methods and features pre-trained using full ImageNet labels.

Key Points & Insights
➡️ Recast colorization as multinomial classification using binned outputs to handle multimodal possibilities, avoiding the averaging effect of L2 loss.
➡️ Incorporate class rebalancing in the loss function to ensure rarer, saturated colors are adequately represented during training.
➡️ The unsupervised colorization task successfully generates feature representations that transfer well to supervised downstream tasks like detection and segmentation, competing with other self-supervised techniques.
➡️ The model can successfully generate plausible colorizations for legacy black and white photos, including historical images of extinct species like the thylacine.

📸 Video summarized with SummaryTube.com on Feb 28, 2026, 09:56 UTC