[SIGGRAPH 2016] Automatic Image Colorization
1 like2,201 views
The document discusses a novel end-to-end neural network approach for the automatic colorization of black-and-white images, emphasizing the integration of global and local features through a fusion layer. It highlights the model's architecture, which includes low-level, mid-level, and global feature networks, and its effectiveness demonstrated through user studies. Additionally, it addresses limitations related to color accuracy and provides links to the project's resources.
![5
Related Work
Scribble-based [Levin+ 2004; Yatziv+ 2004;
An+ 2009; Xu+ 2013; Endo+ 2016]
Specify colors with scribbles
Require manual inputs
Reference image-based [Chia+ 2011;
Gupta+ 2012]
Transfer colors of reference images
Require very similar images
[Levin+ 2004]
[Gupta+ 2012]
Input Reference Output](https://image.slidesharecdn.com/iizukasiggraph2016slide-170809084052/85/SIGGRAPH-2016-Automatic-Image-Colorization-5-320.jpg)
![6
Related Work
Automatic colorization with hand-crafted features [Cheng+ 2015]
Uses existing multiple image features
Computes chrominance via a shallow neural network
Depends on the performance of semantic segmentation
Only handles simple outdoor scenes
Image
features
Chroma OutputInput Neural Network](https://image.slidesharecdn.com/iizukasiggraph2016slide-170809084052/85/SIGGRAPH-2016-Automatic-Image-Colorization-6-320.jpg)
![16
Training of Colors
Mean Squared Error (MSE) as loss function
Optimization using ADADELTA [Zeiler 2012]
Adaptively sets a learning rate
Model
Forward
Backward
MSE
Input Output Ground truth](https://image.slidesharecdn.com/iizukasiggraph2016slide-170809084052/85/SIGGRAPH-2016-Automatic-Image-Colorization-16-320.jpg)
![18
Dataset
MIT Places Scene Dataset [Zhou+ 2014]
2.3 million training images with 205 scene labels
256 × 256 pixels
Abbey Airport terminal Aquarium Baseball field
Dining room Forest road Gas station Gift shop
⋯
⋯](https://image.slidesharecdn.com/iizukasiggraph2016slide-170809084052/85/SIGGRAPH-2016-Automatic-Image-Colorization-18-320.jpg)
![[SIGGRAPH 2016] Automatic Image Colorization](https://image.slidesharecdn.com/iizukasiggraph2016slide-170809084052/85/SIGGRAPH-2016-Automatic-Image-Colorization-19-320.jpg)
![22
Comparisons
Input [Cheng+ 2015] Ours
(w/ global features)
Ours
(w/o global features)](https://image.slidesharecdn.com/iizukasiggraph2016slide-170809084052/85/SIGGRAPH-2016-Automatic-Image-Colorization-22-320.jpg)
[SIGGRAPH 2016] Automatic Image Colorization
- 1. Satoshi Iizuka* Edgar Simo-Serra* Hiroshi Ishikawa Waseda University (*equal contribution)
- 2. 2 Colorization of Black-and-white Pictures
- 3. 3 Our Goal: Fully-automatic colorization
- 4. 4 Colorization of Old Films
- 5. 5 Related Work Scribble-based [Levin+ 2004; Yatziv+ 2004; An+ 2009; Xu+ 2013; Endo+ 2016] Specify colors with scribbles Require manual inputs Reference image-based [Chia+ 2011; Gupta+ 2012] Transfer colors of reference images Require very similar images [Levin+ 2004] [Gupta+ 2012] Input Reference Output
- 6. 6 Related Work Automatic colorization with hand-crafted features [Cheng+ 2015] Uses existing multiple image features Computes chrominance via a shallow neural network Depends on the performance of semantic segmentation Only handles simple outdoor scenes Image features Chroma OutputInput Neural Network
- 7. 7 Contributions Novel end-to-end network that jointly learns global and local features for automatic image colorization New fusion layer that elegantly merges the global and local features Exploit classification labels for learning
- 8. 8 Layers of Our Model Fully-connected layer All neurons are connected between layers Convolutional layer Takes into account underlying spatial structure No. of feature maps Convolutional layer … … Fully-connected layer Neuron 𝑥 𝑦
- 9. 9 Our Model Two branches: local features and global features Composed of four networks Scaling Chrominance Low-Level Features Network Global Features Network Mid-Level Features Network Colorization Network Upsampling Luminance Fusion Layer
- 10. 10 Low-Level Features Network Extract low-level features such as edges and corners Lower resolution for efficient processing Shared weights Scaling Chrominance Low-Level Features Network Global Features Network Mid-Level Features Network Colorization Network Upsampling Luminance Fusion Layer
- 11. 11 Global Features Network Compute a global 256-dimensional vector representation of the image Shared weights Scaling Chrominance Low-Level Features Network Global Features Network Mid-Level Features Network Colorization Network Upsampling Luminance Fusion Layer
- 12. 12 Mid-Level Features Network Extract mid-level features such as texture Shared weights Scaling Chrominance Low-Level Features Network Global Features Network Mid-Level Features Network Colorization Network Upsampling Luminance Fusion Layer
- 13. 13 Fusion Layer Shared weights Scaling Chrominance Low-Level Features Network Global Features Network Mid-Level Features Network Colorization Network Upsampling Luminance Fusion Layer
- 14. 14 Fusion Layer Combine the global features with the mid-level features The resulting features are independent of any resolution Fusion Layer Mid-Level Features Network Global Features Network 𝐲 𝑢,𝑣 fusion = 𝜎 𝐛 + 𝑊 𝐲global 𝐲 𝑢,𝑣 mid
- 15. 15 Colorization Network Compute chrominance from the fused features Restore the image to the input resolution Shared weights Scaling Chrominance Low-Level Features Network Global Features Network Mid-Level Features Network Colorization Network Upsampling Luminance Fusion Layer
- 16. 16 Training of Colors Mean Squared Error (MSE) as loss function Optimization using ADADELTA [Zeiler 2012] Adaptively sets a learning rate Model Forward Backward MSE Input Output Ground truth
- 17. 17 Joint Training Training for classification jointly with the colorization Classification network connected to the global features Shared weights Scaling Low-Level Features Network Global Features Network Mid-Level Features Network Colorization Network Upsampling Luminance Fusion Layer 20.60% Formal Garden 16.13% Arch 13.50% Abbey 7.07% Botanical Garden 6.53% Golf Course Predicted labels Classification Network Chrominance
- 18. 18 Dataset MIT Places Scene Dataset [Zhou+ 2014] 2.3 million training images with 205 scene labels 256 × 256 pixels Abbey Airport terminal Aquarium Baseball field Dining room Forest road Gas station Gift shop ⋯ ⋯
- 20. 20 Computational Time Colorize within a few seconds 80ms
- 21. 21 Colorization of MIT Places Dataset
- 22. 22 Comparisons Input [Cheng+ 2015] Ours (w/ global features) Ours (w/o global features)
- 23. 23 Effectiveness of Global Features w/ global featuresInput w/o global features
- 24. 24 User Study 10 users participated We show 500 images of each type: total 1,500 images per user 90% of our results are considered “natural” Natural Unnatural
- 25. 25 Colorization of Historical Photographs Mount Moran, 1941 Scott's Run, 1937 Youngsters, 1912 Burns Basement, 1910
- 28. 28 Style Transfer Adapting the colorization of one image to the style of another Inputs Local Global Local Global Local Global Output
- 29. 29 Limitations Difficult to output colorful images Cannot restore exact colors Input Ground truth Output Input Ground truth Output
- 30. 30 Conclusion Novel approach for image colorization by fusing global and local information Fusion layer Joint training of colorization and classification Style transfer Farm Land, 1933 California National Park, 1936 Homes, 1936 Spinners, 1910 Doffer Boys, 1909
- 31. 31 Thank you! Project Page http://hi.cs.waseda.ac.jp/~iizuka/projects/colorization Code on GitHub! https://github.com/satoshiiizuka/siggraph2016_colorization Norris Dam, 1933North Dome, 1936 Miner, 1937 Community Center, 1936