Generative modeling with Convolutional Neural Networks
2 likes657 views
This document discusses generative modeling using convolutional neural networks (CNNs) and contrasts discriminative and generative models. It covers various techniques, challenges, and applications of generative adversarial networks (GANs), emphasizing their potential for generating photorealistic images and tackling real-world problems. Additionally, it highlights the evolution of CNN architectures and the importance of future improvements in the field.
![CNNs: Architectures
1. Typical architecture of Convolutional Neural Networks
2. Variations
3. Tendencies
A. Fully Convolutional Networks
B. Reduction of the number of parameters and computational
complexity
INPUT→[[CONV→ReLU] * N→POOL?] * M→[FC→ReLU] * K→FC
ResNet / Inception / …](https://image.slidesharecdn.com/presentationforrigameetup2-170810080352/85/Generative-modeling-with-Convolutional-Neural-Networks-9-320.jpg)
![GAN: Designing the D(x)
1. Normalize the input images in [-1.0, 1.0]
2. Should have less parameters than G(x)
3. Use Dropout/Spatial Dropout
4. 2x2 MaxPooling→Convolution2D + Stride = 2
5. ReLU→LeakyReLU/ELU (SELU?)
6. Adaptive L2 regularization
7. Label Smoothing
8. Instance Noise
9. Balancing of the min-max the game
https://github.com/soumith/ganhacks
Ian Goodfellow, Improved Techniques of Training GANs](https://image.slidesharecdn.com/presentationforrigameetup2-170810080352/85/Generative-modeling-with-Convolutional-Neural-Networks-37-320.jpg)
Generative modeling with Convolutional Neural Networks
- 1. Generative Modeling with Convolutional Neural Networks Denis Dus Lead Data Scientist at InData Labs
- 2. What we will discuss 1. Discriminative vs Generative modeling 2. Convolutional Neural Networks 3. How to train a neural network to fool a discriminative deep learning model? 4. How to train a neural network to produce photorealistic images? 5. What difficulties can we face and how to avoid them? 6. How could GAN framework be used to solve real problems?
- 3. Data Modeling Discriminative model Goal - to find a separating surface of classes A posterior distribution is calculated P(y|x; θ) Example: Logistic regression Generative model Goal - to recover the joint distribution P(x, y; θ) P(y|x) = P(x, y; θ) / P(x) = P(y)P(x|y; θ) / P(x) Example: Naive bayes
- 5. CNNs: Motivation Operation of discrete two-dimensional convolution:
- 7. CNNs: Motivation We are aiming: ● to find weights of filters based on data ● to achieve invariance to small distortions Having: ● specified network architecture ● training data (X, Y) ● loss function L We can optimize weights of filters to minimize the error value L by (X,Y).
- 8. CNNs: Pooling Operation of spatial subsampling (pooling): ● doesn’t have trainable parameters ● reduces convolutions activation maps in spatial dimensions ● makes filters more invariant to small distortions ● Alternative: Conv2D + Stride > 1
- 9. CNNs: Architectures 1. Typical architecture of Convolutional Neural Networks 2. Variations 3. Tendencies A. Fully Convolutional Networks B. Reduction of the number of parameters and computational complexity INPUT→[[CONV→ReLU] * N→POOL?] * M→[FC→ReLU] * K→FC ResNet / Inception / …
- 10. CNNs: VGG16
- 11. CNNs: VGG16 Graph 224x224x64 ≈ 3.2 mln activations ≈ 103 mln parameters Convolutional part (feature extraction)
- 15. Generative CNNs: Idea 1. The joint distribution of image pixels is very complex. 2. It is impossible to approximate it with classical methods. 3. We’ll look for a parametric transformation of the following type:
- 16. Generative CNNs: Idea ● We have strong discriminative networks D(x): VGG-16, ResNet-50, … ● Weights of the networks trained on ImageNet dataset are publicly available https://github.com/fchollet/deep-learning-models ● If you have such network, for every image x you can get
- 17. Generative CNNs: Idea Let’s look for G(z|θ) so that D(G(z|θ)) is close to predefined distribution. What should you do to generate images for the class “Cup”?
- 18. Generative CNNs: Practice - VGG -16 Before training the transformation G(z|θ)
- 19. Generative CNNs: Practice - VGG -16 After training the transformation G(z|θ)
- 20. Generative CNNs: Practice - ResNet-50 Before training the transformation G(z|θ)
- 21. Generative CNNs: Practice - ResNet-50 After training the transformation G(z|θ)
- 22. Generative CNNs: Bad News ● How many images of shape 224x224 pixel exists? ● Natural images are only a small part of this number ● In the statement of the problem, we did not require “naturalness” from G(z|θ)! ● Only a predetermined classification was required
- 23. CNNs: Hack’em all! Let’s consider the following optimization problem
- 24. CNNs: Hack’em all - ResNet-50
- 25. CNNs: Hack’em all - GoogLeNet
- 26. CNNs: Adversarial Attack Competition
- 27. Generative Adversarial Networks The statement of the problem lacks the requirement of image “naturalness”. Let’s add it:
- 30. GAN: Generator
- 31. GAN Framework: Real Data
- 32. GAN Framework: Generator Evolution
- 33. GAN: Difficulties 1. Generation of large-sized images (more than 128x128 pixels) 2. The variability of real-world images 3. There is no guarantee that the game converges to an equilibrium state 4. It is difficult to keep D(x) and G(z) “on the same quality level” 5. Initially, D(x) has a very simple problem (in comparison with G(z)) 6. Quality assessment of G(z)
- 34. GAN: State of the Art
- 35. GAN: State of the Art
- 36. GAN: Variability and Size
- 37. GAN: Designing the D(x) 1. Normalize the input images in [-1.0, 1.0] 2. Should have less parameters than G(x) 3. Use Dropout/Spatial Dropout 4. 2x2 MaxPooling→Convolution2D + Stride = 2 5. ReLU→LeakyReLU/ELU (SELU?) 6. Adaptive L2 regularization 7. Label Smoothing 8. Instance Noise 9. Balancing of the min-max the game https://github.com/soumith/ganhacks Ian Goodfellow, Improved Techniques of Training GANs
- 38. GAN: Designing the G(x) 1. Tanh on the last layer 2. min log(1-D(G(z))→max log(D(G(z)) 3. Should have more parameters than D(x) 4. Use Dropout/Spatial Dropout 5. UpSampling2D/Deconvoluniun2D 6. ReLU→LeakyReLU/ELU 7. Batch Normalization (SELU?) https://github.com/soumith/ganhacks Ian Goodfellow, Improved Techniques of Training GANs
- 39. GAN: Image Segmentation ● One of the main Computer Vision tasks ● Optimization uses per-pixel loss functions
- 40. GAN: Image Segmentation ● The optimization uses per-pixel loss functions ● Per-pixel losses do not reflect many aspects of segmentation quality https://github.com/alexgkendall/SegNet-Tutorial
- 43. Many other interesting things 1. Types of GAN: Conditional Gan, Stacked GAN, Cycle GAN, … 2. Alternative models: VAE, Pix2Pix, ... 3. Other areas of application: generation of text/audio/video 4. Other practical tasks: Image Super-Resolution, ...
- 44. Summary 1. Convolutional neural networks perfectly cope with the CV tasks 2. But there is a serious problem with “Adversarial Examples” 3. GAN - is a very interesting idea at the intersection of areas of mathematics 4. GAN is not only fun, but also useful 5. Future improvements in this area are expected
- 45. Thank you! Denis Dus Lead Data Scientist [email protected] facebook: dzianis.dus www.indatalabs.com @indatalabs @indatalabs