![Sky Segmentation
Input Image
Sky
Segmentation
Literatures
• Sky/non-sky classifier [Tao et al. SIGGRAPH’09]
• Scene parsing [Long et al. CVPR’15]
• Online refinement [Rother et al. SIGGRAPH’04]
Challenges
• Sky appearance varies widely
• skylines/landscapes, clouds, lighting conditions
• Need accurate sky boundaries](https://image.slidesharecdn.com/skyeditsiggraph16-160511121043/85/Semantic-Aware-Sky-Replacement-SIGGRAPH-2016-8-320.jpg)
![Sky Search
Input Image Reference Images
Sky
Search
Literatures
• GIST [Hays and Efros SIGGRAPH’07, Liu et al. CGF’14]
• Only consider global scene layout
• Need a large database
Challenges
• Search compatible images
• Account for image content
Reference Image 1 Reference Image 2 Reference Image 3](https://image.slidesharecdn.com/skyeditsiggraph16-160511121043/85/Semantic-Aware-Sky-Replacement-SIGGRAPH-2016-9-320.jpg)
![Sky Replacement
Input Image
Literatures
• Global transfer [Reinhard et al. 2001, Tao et al. SIGGRAPH’09]
• Image contents are not considered
• Less realistic results
• Local transfer [Wu et al. CGF’13, Laffont et al. SIGGRAPH’14]
• Boundary artifacts
• Rely on filters for smoothing
Challenges
• Transfer foreground appearance
• Account for image content
Sky
Replacement](https://image.slidesharecdn.com/skyeditsiggraph16-160511121043/85/Semantic-Aware-Sky-Replacement-SIGGRAPH-2016-10-320.jpg)
![Fully Convolutional Networks
Scene Parsing
Fg
Road
Building
Sky
Tree
Semantic Response
Sky
. . .
Building Road
Fully Convolutional Networks (FCN)
• End-to-end model
• Pixel-wise segmentation
• Finetune with 11 scene labels
• Semantic response map
[Long et al. CVPR’15]](https://image.slidesharecdn.com/skyeditsiggraph16-160511121043/85/Semantic-Aware-Sky-Replacement-SIGGRAPH-2016-12-320.jpg)
![Results
DeepLab [Chen et al. ICLR’15]](https://image.slidesharecdn.com/skyeditsiggraph16-160511121043/85/Semantic-Aware-Sky-Replacement-SIGGRAPH-2016-18-320.jpg)
![Semantic-aware Transfer
Direct local transfer [Laffont et al. SIGGRAPH’14]
• Match corresponding semantic regions
• Boundary artifacts
Input image Scene parsing](https://image.slidesharecdn.com/skyeditsiggraph16-160511121043/85/Semantic-Aware-Sky-Replacement-SIGGRAPH-2016-30-320.jpg)
![Semantic-aware Transfer
Direct local transfer [Laffont et al. SIGGRAPH’14]
• Match corresponding semantic regions
• Boundary artifacts
T1 (x)
Input image Scene parsing](https://image.slidesharecdn.com/skyeditsiggraph16-160511121043/85/Semantic-Aware-Sky-Replacement-SIGGRAPH-2016-31-320.jpg)
![Semantic-aware Transfer
Direct local transfer [Laffont et al. SIGGRAPH’14]
• Match corresponding semantic regions
• Boundary artifacts
T2 (x)
T1 (x)
Input image Scene parsing](https://image.slidesharecdn.com/skyeditsiggraph16-160511121043/85/Semantic-Aware-Sky-Replacement-SIGGRAPH-2016-32-320.jpg)
![Semantic-aware Transfer
Direct local transfer [Laffont et al. SIGGRAPH’14]
• Match corresponding semantic regions
• Boundary artifacts
Input image Scene parsing Direct local transfer
T2 (x)
T1 (x)](https://image.slidesharecdn.com/skyeditsiggraph16-160511121043/85/Semantic-Aware-Sky-Replacement-SIGGRAPH-2016-33-320.jpg)
![Semantic-aware Transfer
Direct local transfer [Laffont et al. SIGGRAPH’14]
• Match corresponding semantic regions
• Boundary artifacts
Propose a soft mapping method
• Utilize semantic responses as weights
for each category n
Input image Scene parsing Direct local transfer
T1 (x)
T2 (x)](https://image.slidesharecdn.com/skyeditsiggraph16-160511121043/85/Semantic-Aware-Sky-Replacement-SIGGRAPH-2016-34-320.jpg)
![Semantic-aware Transfer
Direct local transfer [Laffont et al. SIGGRAPH’14]
• Match corresponding semantic regions
• Boundary artifacts
Propose a soft mapping method
• Utilize semantic responses as weights
for each category n
Input image Scene parsing Direct local transfer
T1 (x)
T2 (x)](https://image.slidesharecdn.com/skyeditsiggraph16-160511121043/85/Semantic-Aware-Sky-Replacement-SIGGRAPH-2016-35-320.jpg)
![Semantic-aware Transfer
Direct local transfer [Laffont et al. SIGGRAPH’14]
• Match corresponding semantic regions
• Boundary artifacts
Propose a soft mapping method
• Utilize semantic responses as weights
for each category n
Input image Scene parsing Direct local transfer Soft mapping
Wn (x) = 1 or 0
T1 (x)
T2 (x)](https://image.slidesharecdn.com/skyeditsiggraph16-160511121043/85/Semantic-Aware-Sky-Replacement-SIGGRAPH-2016-36-320.jpg)
![Transfer Functions
Transfer Functions Tn (x) for each category n
• Transfer luminance and color
Color
• Matched regions: chrominance
• Histogram matching [Lee et al. CVPR’16]
• Non-matched regions: color temperature
• Consider entire foreground
• More conservative
Not all the semantic regions are matched!
T1 (x)
T2 (x)
?](https://image.slidesharecdn.com/skyeditsiggraph16-160511121043/85/Semantic-Aware-Sky-Replacement-SIGGRAPH-2016-38-320.jpg)
![Joint Object Classification and Segmentation [BMVC’13]
• How do segmentation and classification help each other?
Class-specific Object Segmentation Hypotheses [ICCV’13]
• How to utilize exemplars to gain more information
during learning and inference?
Image Retrieval [ICIP’14]
• Compute label similarities to bridge semantic gaps
Exemplar-based Object Detection [CVPR’15]
• Discover representative exemplars to build models
• Region-based feature extraction and model learning
Image (Object) Recognition
• Classification
• Segmentation
• Retrieval
• Detection](https://image.slidesharecdn.com/skyeditsiggraph16-160511121043/85/Semantic-Aware-Sky-Replacement-SIGGRAPH-2016-59-320.jpg)
![Video Object Recognition
• Object (Co-)segmentation
• Scene (Co-)parsing
Video Segmentation via Object Flow [CVPR’16]
• How do segmentation and optical flow help each other?
• Segmentation: multi-scale, spatio-temporal graphical model
• Optical flow: use segmentation to refine boundaries
• Iteratively solve the joint model
Semantic Co-segmentation in Videos (submitted to ECCV’16)
• Temporal-consistent object tracklets
• Relations between objects from a collection of videos
Ongoing and future work
• Scene Parsing via Deep CNNs
• Attention to small objects
• Label co-occurrence
• Video Scene Co-parsing
• Weakly-supervised: video tags
• Use image-based classifier](https://image.slidesharecdn.com/skyeditsiggraph16-160511121043/85/Semantic-Aware-Sky-Replacement-SIGGRAPH-2016-60-320.jpg)
![Joint Object Classification
and Segmentation [BMVC’13] Object Segmentation [ICCV’13]
Image Retrieval [ICIP’14]
Object Detection [CVPR’15]
Video Object Segmentation
[CVPR’16]
Sky Replacement [SIGGRAPH’16]
Semantic Co-segmentation in Videos
(submitted to ECCV’16)
Video Scene Co-parsing (ongoing)
Image (Object) Recognition via Exemplars
• Classification
• Segmentation
• Retrieval
• Detection
Video Object Recognition: Temporal + CNN
• Object (Co-)segmentation
• Scene (Co-)parsing
Image/Video Editing
• Background/Object Replacement
• Scene Completion
• Re-coloring
Semantic Information
My homepage:
https://sites.google.com/site/yihsuantsai/
Thank you!](https://image.slidesharecdn.com/skyeditsiggraph16-160511121043/85/Semantic-Aware-Sky-Replacement-SIGGRAPH-2016-67-320.jpg)
Semantic-Aware Sky Replacement (SIGGRAPH 2016)
- 1. Sky is Not the Limit: Semantic- Aware Sky Replacement Yi-Hsuan Tsai Xiaohui Shen Zhe Lin Ming-Hsuan YangKalyan Sunkavalli ACM Transactions on Graphics (SIGGRAPH), 2016
- 2. Motivation Goal: automatically segment and replace with different styles of the sky
- 5. Challenges • Manually edit sky using Photoshop 5 mins 30 mins We need a good segmentation algorithm! Input Image Reference
- 6. Challenges • Manually edit sky using Photoshop Input Image Reference We need image harmonization! v.s Professional editingColors are not matched
- 8. Sky Segmentation Input Image Sky Segmentation Literatures • Sky/non-sky classifier [Tao et al. SIGGRAPH’09] • Scene parsing [Long et al. CVPR’15] • Online refinement [Rother et al. SIGGRAPH’04] Challenges • Sky appearance varies widely • skylines/landscapes, clouds, lighting conditions • Need accurate sky boundaries
- 9. Sky Search Input Image Reference Images Sky Search Literatures • GIST [Hays and Efros SIGGRAPH’07, Liu et al. CGF’14] • Only consider global scene layout • Need a large database Challenges • Search compatible images • Account for image content Reference Image 1 Reference Image 2 Reference Image 3
- 10. Sky Replacement Input Image Literatures • Global transfer [Reinhard et al. 2001, Tao et al. SIGGRAPH’09] • Image contents are not considered • Less realistic results • Local transfer [Wu et al. CGF’13, Laffont et al. SIGGRAPH’14] • Boundary artifacts • Rely on filters for smoothing Challenges • Transfer foreground appearance • Account for image content Sky Replacement
- 11. Semantic-Aware System Input Image Sky Segmentation Reference Images Sky Search Sky Replacement Results Fully Convolutional Networks
- 12. Fully Convolutional Networks Scene Parsing Fg Road Building Sky Tree Semantic Response Sky . . . Building Road Fully Convolutional Networks (FCN) • End-to-end model • Pixel-wise segmentation • Finetune with 11 scene labels • Semantic response map [Long et al. CVPR’15]
- 13. Sky Segmentation Input Image Scene Parsing Online Refinement Fully Convolutional Networks
- 14. Sky Segmentation Input Image Scene Parsing Fully Convolutional Networks Online Refinement Conditional Random Field optimization • Online models: color, texture • Semantic response (sky/non-sky) • Pairwise term: magnitude of gradient
- 16. Input Image FCN Results Our Results
- 18. Results DeepLab [Chen et al. ICLR’15]
- 19. Sky Search Input Image Sky Image Database (415 Images) Sky Search
- 20. Sky Search Input Image Reference Images Semantic Layout Descriptor • Account for local layouts • Utilize semantic responses Sky Search Sky Image Database (415 Images)
- 21. Sky Search Input Image Reference Images Semantic Layout Descriptor • Account for local layouts • Utilize semantic responses Sky Search Check Sky Properties • Prevent large distortions • Aspect ratio • Resolution • Ensure sky diversity • Color similarity Sky Image Database (415 Images)
- 22. Semantic Layout Descriptor Input Image . . . Sky Building Road Semantic Responses • Pixel-wise responses • Range from 0 to 1
- 23. Semantic Layout Descriptor Input Image . . . Sky Building Road Semantic Responses • Pixel-wise responses • Range from 0 to 1 Average pooling on spatial pyramids • Global pooling
- 24. Semantic Layout Descriptor Input Image . . . Sky Building Road Semantic Responses • Pixel-wise responses • Range from 0 to 1 Average pooling on spatial pyramids • Global pooling • Local contents (3x3 grids) . . . ...
- 25. Semantic Layout Descriptor Input Image . . . Sky Building Road
- 26. Semantic Layout Descriptor Input Image . . . Sky Building Road . . . . . . . . .
- 27. Semantic Layout Descriptor Input Image . . . Sky Building Road . . . . . . . . . Descriptor . . .
- 28. Sky Replacement Input Image Sky Alignment Sky Alignment • Extract complete sky regions from reference images • Re-scale and paste on the input image Reference Images
- 29. Sky Replacement Input Image Sky Alignment Semantic-aware Transfer Sky Alignment • Extract complete sky regions from reference images • Re-scale and paste on the input image Semantic-aware Transfer • Adjustment foreground appearance • Account for semantic regions Reference Images
- 30. Semantic-aware Transfer Direct local transfer [Laffont et al. SIGGRAPH’14] • Match corresponding semantic regions • Boundary artifacts Input image Scene parsing
- 31. Semantic-aware Transfer Direct local transfer [Laffont et al. SIGGRAPH’14] • Match corresponding semantic regions • Boundary artifacts T1 (x) Input image Scene parsing
- 32. Semantic-aware Transfer Direct local transfer [Laffont et al. SIGGRAPH’14] • Match corresponding semantic regions • Boundary artifacts T2 (x) T1 (x) Input image Scene parsing
- 33. Semantic-aware Transfer Direct local transfer [Laffont et al. SIGGRAPH’14] • Match corresponding semantic regions • Boundary artifacts Input image Scene parsing Direct local transfer T2 (x) T1 (x)
- 34. Semantic-aware Transfer Direct local transfer [Laffont et al. SIGGRAPH’14] • Match corresponding semantic regions • Boundary artifacts Propose a soft mapping method • Utilize semantic responses as weights for each category n Input image Scene parsing Direct local transfer T1 (x) T2 (x)
- 35. Semantic-aware Transfer Direct local transfer [Laffont et al. SIGGRAPH’14] • Match corresponding semantic regions • Boundary artifacts Propose a soft mapping method • Utilize semantic responses as weights for each category n Input image Scene parsing Direct local transfer T1 (x) T2 (x)
- 36. Semantic-aware Transfer Direct local transfer [Laffont et al. SIGGRAPH’14] • Match corresponding semantic regions • Boundary artifacts Propose a soft mapping method • Utilize semantic responses as weights for each category n Input image Scene parsing Direct local transfer Soft mapping Wn (x) = 1 or 0 T1 (x) T2 (x)
- 37. Transfer Functions Transfer Functions Tn (x) for each category n • Transfer luminance and color T1 (x) T2 (x) Luminance • Shift mean
- 38. Transfer Functions Transfer Functions Tn (x) for each category n • Transfer luminance and color Color • Matched regions: chrominance • Histogram matching [Lee et al. CVPR’16] • Non-matched regions: color temperature • Consider entire foreground • More conservative Not all the semantic regions are matched! T1 (x) T2 (x) ?
- 40. Input Image Sky Replacement Results
- 41. Input Image Sky Replacement Results
- 42. Input Image Sky Replacement Results
- 43. Input Image Sky Replacement Results
- 44. Input Image Sky Replacement Results
- 45. Sky Replacement with User Preference
- 46. Input Image Sky Replacement Results
- 47. Input Image Sky Replacement Results
- 48. Input Image Sky Replacement Results
- 50. Sky Replacement ResultsInput Image Preferred Sky
- 52. Comparisons of different search methods
- 54. Comparisons of different transfer methods
- 57. Conclusions • Automatic sky replacement results can be realistic • New sky image database • Semantics helps a lot • Sky segmentation • Sky image search • Appearance transfer • Apply semantics to other tasks • Scene completion • Photo and video re-coloring
- 58. Summary of my Other Projects: Visual Object Recognition
- 59. Joint Object Classification and Segmentation [BMVC’13] • How do segmentation and classification help each other? Class-specific Object Segmentation Hypotheses [ICCV’13] • How to utilize exemplars to gain more information during learning and inference? Image Retrieval [ICIP’14] • Compute label similarities to bridge semantic gaps Exemplar-based Object Detection [CVPR’15] • Discover representative exemplars to build models • Region-based feature extraction and model learning Image (Object) Recognition • Classification • Segmentation • Retrieval • Detection
- 60. Video Object Recognition • Object (Co-)segmentation • Scene (Co-)parsing Video Segmentation via Object Flow [CVPR’16] • How do segmentation and optical flow help each other? • Segmentation: multi-scale, spatio-temporal graphical model • Optical flow: use segmentation to refine boundaries • Iteratively solve the joint model Semantic Co-segmentation in Videos (submitted to ECCV’16) • Temporal-consistent object tracklets • Relations between objects from a collection of videos Ongoing and future work • Scene Parsing via Deep CNNs • Attention to small objects • Label co-occurrence • Video Scene Co-parsing • Weakly-supervised: video tags • Use image-based classifier
- 61. Object Segmentation 96.4 MCL, 74.4 93.3 PMCut, 59.1 94.4 MCL, 53.083.6 PMCut, 47.3
- 62. Object Segmentation 93.5 PMCut, 26.6 89.2 MCL, 65.373.8 PMCut, 58.0 86.9 PMCut, 68.0
- 63. Object Detection
- 64. Object Detection
- 66. Video Object Segmentation Segmentation Updated Optical Flow Initial Optical Flow
- 67. Joint Object Classification and Segmentation [BMVC’13] Object Segmentation [ICCV’13] Image Retrieval [ICIP’14] Object Detection [CVPR’15] Video Object Segmentation [CVPR’16] Sky Replacement [SIGGRAPH’16] Semantic Co-segmentation in Videos (submitted to ECCV’16) Video Scene Co-parsing (ongoing) Image (Object) Recognition via Exemplars • Classification • Segmentation • Retrieval • Detection Video Object Recognition: Temporal + CNN • Object (Co-)segmentation • Scene (Co-)parsing Image/Video Editing • Background/Object Replacement • Scene Completion • Re-coloring Semantic Information My homepage: https://sites.google.com/site/yihsuantsai/ Thank you!