Improving Spatial Codification in Semantic Segmentation
2 likes784 views
This document summarizes an approach to improving spatial codification in semantic segmentation. It proposes using figure-border-ground spatial pooling with object candidates as well as applying a contour-based spatial pyramid, including crown-based and cartesian-based variations. Experiments show these approaches improve accuracy over traditional spatial pooling, especially when using CPMC and MCG object candidates, achieving state-of-the-art performance on PASCAL VOC 2011 and 2012 datasets.
![Related Work
● The Visual Extent of
an Object [1]
5
[1] Uijlings et al, The VIsual Extent of an Object. IJCV’12](https://image.slidesharecdn.com/slides-151001154911-lva1-app6891/85/Improving-Spatial-Codification-in-Semantic-Segmentation-5-320.jpg)
![Related Work
● Beyond bags of features:
Spatial pyramid matching
for recognizing natural
scene categories [1]
7
[1] Lazebnik et al, Beyond bags of features: Spatial pyramid matching for recognizing natural scenes. CVPR’06](https://image.slidesharecdn.com/slides-151001154911-lva1-app6891/85/Improving-Spatial-Codification-in-Semantic-Segmentation-7-320.jpg)
![Related Work
● Variations of SPM
○ Non-arbitrary division
■ Object-centric pooling [1]
■ Object confidence map
partition [2]
○ SPM over bounding boxes [3]
[4]
[1] Russakovky et al, Object-centric spatial pooling for image classification. ECCV’12
[2] Chen et al, Hierarchical Matching with Side Information for Image Classification. CVPR’12
[3] Arbeláez et at, Semantic segmentation using regions and parts. CVPR’12
[4] Gu et al, Multi-component models for object detection. ECCV’12
8](https://image.slidesharecdn.com/slides-151001154911-lva1-app6891/85/Improving-Spatial-Codification-in-Semantic-Segmentation-8-320.jpg)
![Architecture
● Architecture proposed and released in [1]
[1] Carreira et al, Semantic segmentation with second-order pooling. ECCV’12
Train Test
DataBase
Object
Candidates
Feature
Extraction
Test
Model
Prediction
Evaluation
AAC
Ground
Truth
Train
CPMC SIFT-based
features (O2P)
11](https://image.slidesharecdn.com/slides-151001154911-lva1-app6891/85/Improving-Spatial-Codification-in-Semantic-Segmentation-11-320.jpg)
![Experiments
● Experiments with ideal object
candidates
○ Train set: train11
○ Test set: val11
F[1] F-B F-G[1] F-B-G
eSIFT 63.9 66.2 66.4 68.6
eMSIFT 64.8 68.9 67.6 70.8
[1] Carreira et al, Semantic segmentation with second-order pooling. ECCV’12 13](https://image.slidesharecdn.com/slides-151001154911-lva1-app6891/85/Improving-Spatial-Codification-in-Semantic-Segmentation-13-320.jpg)
![Experiments
● Experiments with ideal object
candidates
○ Train set: train11
○ Test set: val11
F F-B F-B-G
non SP 64.8 [1] 68.9 70.8
crown-based SP 68.7 71.1 71.7
Cartesian-based SP 67.7 71.6 72.7
Figure SP(Figure) Border Ground AAC
eSIFT+eMSIFT+eLBP eSIFT 72.98 [1]
eSIFT+eMSIFT eMSIFT+eSIFT eMSIFT+eSIFT 73.84
eSIFT+eMSIFT+eLBP eMSIFT eMSIFT+eSIFT eMSIFT+eSIFT 75.86
[1] Carreira et al, Semantic segmentation with second-order pooling. ECCV’12
14](https://image.slidesharecdn.com/slides-151001154911-lva1-app6891/85/Improving-Spatial-Codification-in-Semantic-Segmentation-14-320.jpg)
![Experiments
● Experiments with CPMC object candidates
○ Train set: train11
○ Test set: val11
[1] Carreira et al, Semantic segmentation with second-order pooling. ECCV’12
Figure SP(Figure) Border Ground AAC
eSIFT eSIFT 28.6 [1]
eSIFT eSIFT eSIFT 34.8
eSIFT+eMSIFT+eLBP eSIFT 37.2 [1]
eSIFT eSIFT eSIFT eSIFT 37.4
eSIFT+eMSIFT+eLBP eSIFT eSIFT eSIFT 39.6
15](https://image.slidesharecdn.com/slides-151001154911-lva1-app6891/85/Improving-Spatial-Codification-in-Semantic-Segmentation-15-320.jpg)
![Experiments
● Experiments with CPMC object
candidates in comp5 challenge
○ Train set: trainval11 /
trainval12
○ Test set: test11 / test12
F-G [1] F-B-G SP(F)-B-G
PASCAL
VOC11
38.8 43.8 40.3
PASCAL
VOC12
39.9 42.2 40.8
[1] Carreira et al, Semantic segmentation with second-order pooling. ECCV’12
16](https://image.slidesharecdn.com/slides-151001154911-lva1-app6891/85/Improving-Spatial-Codification-in-Semantic-Segmentation-16-320.jpg)
![Experiments
● Experiments with MCG object
candidates [1]
○ Train set: train11
○ Test set: val11
F-G[2] F-B-G SP(F)-B-G
CPMC 37.2 38.9 39.6
MCG 30.9 34.1 36.1
[1] Arbeláez et al, Multiscale Combinatorial Grouping. CVPR’14
[2] Carreira et al, Semantic segmentation with second-order pooling. ECCV’12 17](https://image.slidesharecdn.com/slides-151001154911-lva1-app6891/85/Improving-Spatial-Codification-in-Semantic-Segmentation-17-320.jpg)
Improving Spatial Codification in Semantic Segmentation
- 1. C. Ventura X. Giró-i-Nieto V. Vilaplana F. Marqués K. McGuinness N. O’Connor Improving Spatial Codification in Semantic Segmentation
- 2. Outline ● Introduction ● Related Work and Contributions ● Architecture ● Experiments ● Conclusions
- 3. Introduction ● Object segmentation vs Class segmentation Image Object segmentation Class segmentation 3
- 4. Outline ● Introduction ● Related Work and Contributions ● Architecture ● Experiments ● Conclusions
- 5. Related Work ● The Visual Extent of an Object [1] 5 [1] Uijlings et al, The VIsual Extent of an Object. IJCV’12
- 6. 1st Contribution ● Using a Figure-Border-Ground spatial pooling with object candidates 6 Figure-Ground spatial pooling Figure-Border-Ground spatial pooling
- 7. Related Work ● Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories [1] 7 [1] Lazebnik et al, Beyond bags of features: Spatial pyramid matching for recognizing natural scenes. CVPR’06
- 8. Related Work ● Variations of SPM ○ Non-arbitrary division ■ Object-centric pooling [1] ■ Object confidence map partition [2] ○ SPM over bounding boxes [3] [4] [1] Russakovky et al, Object-centric spatial pooling for image classification. ECCV’12 [2] Chen et al, Hierarchical Matching with Side Information for Image Classification. CVPR’12 [3] Arbeláez et at, Semantic segmentation using regions and parts. CVPR’12 [4] Gu et al, Multi-component models for object detection. ECCV’12 8
- 9. 2nd Contribution ● Applying a contour-based spatial pyramid (SP) ○ Crown-based SP ○ Cartesian-based SP 9 Crown-based spatial pyramid Cartesian-based spatial pyramid
- 10. Outline ● Introduction ● Related Work and Contributions ● Architecture ● Experiments ● Conclusions
- 11. Architecture ● Architecture proposed and released in [1] [1] Carreira et al, Semantic segmentation with second-order pooling. ECCV’12 Train Test DataBase Object Candidates Feature Extraction Test Model Prediction Evaluation AAC Ground Truth Train CPMC SIFT-based features (O2P) 11
- 12. Outline ● Introduction ● Related Work and Contributions ● Architecture ● Experiments ● Conclusions
- 13. Experiments ● Experiments with ideal object candidates ○ Train set: train11 ○ Test set: val11 F[1] F-B F-G[1] F-B-G eSIFT 63.9 66.2 66.4 68.6 eMSIFT 64.8 68.9 67.6 70.8 [1] Carreira et al, Semantic segmentation with second-order pooling. ECCV’12 13
- 14. Experiments ● Experiments with ideal object candidates ○ Train set: train11 ○ Test set: val11 F F-B F-B-G non SP 64.8 [1] 68.9 70.8 crown-based SP 68.7 71.1 71.7 Cartesian-based SP 67.7 71.6 72.7 Figure SP(Figure) Border Ground AAC eSIFT+eMSIFT+eLBP eSIFT 72.98 [1] eSIFT+eMSIFT eMSIFT+eSIFT eMSIFT+eSIFT 73.84 eSIFT+eMSIFT+eLBP eMSIFT eMSIFT+eSIFT eMSIFT+eSIFT 75.86 [1] Carreira et al, Semantic segmentation with second-order pooling. ECCV’12 14
- 15. Experiments ● Experiments with CPMC object candidates ○ Train set: train11 ○ Test set: val11 [1] Carreira et al, Semantic segmentation with second-order pooling. ECCV’12 Figure SP(Figure) Border Ground AAC eSIFT eSIFT 28.6 [1] eSIFT eSIFT eSIFT 34.8 eSIFT+eMSIFT+eLBP eSIFT 37.2 [1] eSIFT eSIFT eSIFT eSIFT 37.4 eSIFT+eMSIFT+eLBP eSIFT eSIFT eSIFT 39.6 15
- 16. Experiments ● Experiments with CPMC object candidates in comp5 challenge ○ Train set: trainval11 / trainval12 ○ Test set: test11 / test12 F-G [1] F-B-G SP(F)-B-G PASCAL VOC11 38.8 43.8 40.3 PASCAL VOC12 39.9 42.2 40.8 [1] Carreira et al, Semantic segmentation with second-order pooling. ECCV’12 16
- 17. Experiments ● Experiments with MCG object candidates [1] ○ Train set: train11 ○ Test set: val11 F-G[2] F-B-G SP(F)-B-G CPMC 37.2 38.9 39.6 MCG 30.9 34.1 36.1 [1] Arbeláez et al, Multiscale Combinatorial Grouping. CVPR’14 [2] Carreira et al, Semantic segmentation with second-order pooling. ECCV’12 17
- 18. Experiments ● Qualitative results F-B-G spatial pooling with CPMC 18 F-G F-B-G F-G F-B-G aeroplane bicycle bicycle birdcat motorbike boat bottle bus bus motorbike car chair cat chair chair horse cow bird
- 19. Experiments ● Qualitative results F-B-G spatial pooling with CPMC 19 chair dining table cow dog person horse person motorbike motorbike motorbike person plotted plant bottle sheep sofa dog bus train train tvmonitor F-G F-B-G F-G F-B-G
- 20. Outline ● Introduction ● Related Work and Contributions ● Architecture ● Experiments ● Conclusions
- 21. Conclusions ● 2 proposals beyond the classic Figure-Ground pooling ○ Figure-Border-Ground spatial pooling ■ Extended to realistic scenario with CPMC object candidates ○ A novel contour-based spatial pyramid has been introduced ■ Cartesian-based spatial pyramid ■ Crown-based spatial pyramid ● Validation of both proposals also for MCG object candidates 21
- 23. Related Work ● The Visual Extent of an Object (Uijlings et al, IJCV’12) 23