Variational Autoencoder
20 likes8,653 views
Youtube: https://www.youtube.com/playlist?list=PLeeHDpwX2Kj55He_jfPojKrZf22HVjAZY Paper review of "Auto-Encoding Variational Bayes"
![Varia%onal Inference
• Approximate by
• Minimize the KL Divergence:
p(z|x) q(z)
DKL[q(z)||p(z|x)] =
Z
q(z)log
q(z)
p(z|x)
dz](https://image.slidesharecdn.com/variationalautoencoder-160616085558/85/Variational-Autoencoder-6-320.jpg)
![Evidence(Varia%onal) Lower Bound
Evidence Lower Bound (ELBO):
= (Eq(z)[logp(z, x)] Eq(z)[logq(z)]) + logp(x)
L[q(z)]
DKL[q(z)||p(z|x)] =
Z
q(z)log
q(z)
p(z|x)
dz
=
Z
q(z)log
q(z)p(x)
p(z, x)
dz
=
Z
q(z)log
q(z)
p(z, x)
dz +
Z
q(z)logp(x)dz
=
Z
q(z)(logq(z) logp(z, x))dz + logp(x)](https://image.slidesharecdn.com/variationalautoencoder-160616085558/85/Variational-Autoencoder-7-320.jpg)
![Evidence Lower Bound
Minimize
is equal to Maximize L[q(z)]
DKL[q(z)||p(z|x)] = L[q(z)] + logp(x)
logp(x) = DKL[q(z)||p(z|x)] + L[q(z)]
DKL[q(z)||p(z|x)]](https://image.slidesharecdn.com/variationalautoencoder-160616085558/85/Variational-Autoencoder-8-320.jpg)
![Mean-Field Varia%onal Inference
• Q can be factorized:
q(z) =
Y
i
q(zi|✓i)
8i,
Z
q(zi|✓i)dzi = 1
Minimize
DKL[q(z)||p(z|x)]
q(z)
p(z|x)
hXp://cpmarkchang.logdown.com/posts/737247-pgm-varia%onal-inference](https://image.slidesharecdn.com/variationalautoencoder-160616085558/85/Variational-Autoencoder-9-320.jpg)
![Varia%onal Autoencoder
q (z|x)
Encoder
Network
Decoder
Network
p✓(x|z)
DKL[q (z|x)||p✓(z|x)]Minimize:
p✓(z|x) =
p✓(x|z)p✓(z)
p✓(x)
Intractable:](https://image.slidesharecdn.com/variationalautoencoder-160616085558/85/Variational-Autoencoder-10-320.jpg)
![Varia%onal Autoencoder
logp✓(x) = DKL[q (z|x)||p✓(z|x)] + L(✓, , x)
L(✓, , x) = Eq (z|x)[logp✓(x, z) logq (z|x)]
= DKL[q (z|x)||p✓(z)] + Eq (z|x)[logp✓(x|z)]
Marginal Likelihood:
Varia%onal Lower Bound:
= Eq (z|x)[logp✓(z) + logp✓(x|z) logq (z|x)]
= Eq (z|x)[log
p✓(z)
q (z|x)
+ p✓(x|z)]](https://image.slidesharecdn.com/variationalautoencoder-160616085558/85/Variational-Autoencoder-11-320.jpg)
![Monte Carlo Gradient Es%mator
Gradient of contains
which is Intractable
L(✓, , x)
Use Monte Carlo Gradient Es%mator :
where
r Eq (z|x)[logp✓(x|z)]
r Eq (z)[f(z)] = r
Z
q (z)f(z)dz
=
Z
q (z)f(z)
r q (z)
q (z)
dz =
Z
q (z)f(z)r logq (z)dz
= Eq (z)[f(z)r logq (z)]
⇡
1
L
LX
l=1
f(z)r logq (z(l)
) z(l)
⇠ q (z)](https://image.slidesharecdn.com/variationalautoencoder-160616085558/85/Variational-Autoencoder-12-320.jpg)
![Objec%ve Func%on
L(✓, , x(i)
) = DKL[q (z|x(i)
)||p✓(z)] + Eq (z|x(i))[logp✓(x(i)
|z)]
Monte Carlo Gradient Es%mator
˜L(✓, , x(i)
) ⇡ L(✓, , x(i)
)
where z(l)
⇠ q (z|x(i,l)
)
˜L(✓, , x(i)
) = DKL[q (z|x(i)
)||p✓(z)] +
1
L
LX
l=1
logp✓(x(i)
|z(i,l)
)](https://image.slidesharecdn.com/variationalautoencoder-160616085558/85/Variational-Autoencoder-13-320.jpg)
![Objec%ve Func%on
˜L(✓, , x(i)
) = DKL[q (z|x(i)
)||p✓(z)] +
1
L
LX
l=1
(logp✓(x(i)
|z(i,l)
)
˜L(✓, , x(i)
) =
1
2
JX
j=1
(1+log((
(i)
j )2
) (µ
(i)
j )2
(
(i)
j )2
)+
1
L
LX
l=1
(logp✓(x(i)
|z(i,l)
)
Regulariza%on Reconstruc%on
Error
p✓(z) = N(z, 0, I)
q (z|x(i)
) = N(z, µ(i)
, 2(i)
I)](https://image.slidesharecdn.com/variationalautoencoder-160616085558/85/Variational-Autoencoder-17-320.jpg)
![ViT (Vision Transformer) Review [CDM]](https://cdn.slidesharecdn.com/ss_thumbnails/vitreviewcdm-201012184226-thumbnail.jpg?width=560&fit=bounds)
More Related Content
What's hot (20)
Viewers also liked (7)
Similar to Variational Autoencoder (10)
More from Mark Chang (20)
Recently uploaded (20)
Variational Autoencoder
- 2. Original Paper • Title: – Auto-Encoding Varia%onal Bayes • Author: – Diederik P. Kingma – Max Welling • Organiza%on: – Machine Learning Group, Universiteit van Amsterdam
- 3. Outlines • Varia%onal Inference • Varia%onal Autoencoder • Experiment • Further Research
- 4. Varia%onal Inference • Problem Defini%on – Observable Data: – Hidden Variable: – Posterior Distribu%on of hidden variable given some data: Intractable to compute z = {z1, z2, ..., zn} x z m n p(z|x) = p(z, x) p(x) = p(x|z)p(z) R p(x|z)p(z)dz x = {x1, x2, ..., xm}
- 5. Varia%onal Inference • Solu%ons for Intractable Posterior – Monte Carlo Sampling • Metropolis Has%ng • Gibbs Sampling – Varia%onal Inference
- 6. Varia%onal Inference • Approximate by • Minimize the KL Divergence: p(z|x) q(z) DKL[q(z)||p(z|x)] = Z q(z)log q(z) p(z|x) dz
- 7. Evidence(Varia%onal) Lower Bound Evidence Lower Bound (ELBO): = (Eq(z)[logp(z, x)] Eq(z)[logq(z)]) + logp(x) L[q(z)] DKL[q(z)||p(z|x)] = Z q(z)log q(z) p(z|x) dz = Z q(z)log q(z)p(x) p(z, x) dz = Z q(z)log q(z) p(z, x) dz + Z q(z)logp(x)dz = Z q(z)(logq(z) logp(z, x))dz + logp(x)
- 8. Evidence Lower Bound Minimize is equal to Maximize L[q(z)] DKL[q(z)||p(z|x)] = L[q(z)] + logp(x) logp(x) = DKL[q(z)||p(z|x)] + L[q(z)] DKL[q(z)||p(z|x)]
- 9. Mean-Field Varia%onal Inference • Q can be factorized: q(z) = Y i q(zi|✓i) 8i, Z q(zi|✓i)dzi = 1 Minimize DKL[q(z)||p(z|x)] q(z) p(z|x) hXp://cpmarkchang.logdown.com/posts/737247-pgm-varia%onal-inference
- 10. Varia%onal Autoencoder q (z|x) Encoder Network Decoder Network p✓(x|z) DKL[q (z|x)||p✓(z|x)]Minimize: p✓(z|x) = p✓(x|z)p✓(z) p✓(x) Intractable:
- 11. Varia%onal Autoencoder logp✓(x) = DKL[q (z|x)||p✓(z|x)] + L(✓, , x) L(✓, , x) = Eq (z|x)[logp✓(x, z) logq (z|x)] = DKL[q (z|x)||p✓(z)] + Eq (z|x)[logp✓(x|z)] Marginal Likelihood: Varia%onal Lower Bound: = Eq (z|x)[logp✓(z) + logp✓(x|z) logq (z|x)] = Eq (z|x)[log p✓(z) q (z|x) + p✓(x|z)]
- 12. Monte Carlo Gradient Es%mator Gradient of contains which is Intractable L(✓, , x) Use Monte Carlo Gradient Es%mator : where r Eq (z|x)[logp✓(x|z)] r Eq (z)[f(z)] = r Z q (z)f(z)dz = Z q (z)f(z) r q (z) q (z) dz = Z q (z)f(z)r logq (z)dz = Eq (z)[f(z)r logq (z)] ⇡ 1 L LX l=1 f(z)r logq (z(l) ) z(l) ⇠ q (z)
- 13. Objec%ve Func%on L(✓, , x(i) ) = DKL[q (z|x(i) )||p✓(z)] + Eq (z|x(i))[logp✓(x(i) |z)] Monte Carlo Gradient Es%mator ˜L(✓, , x(i) ) ⇡ L(✓, , x(i) ) where z(l) ⇠ q (z|x(i,l) ) ˜L(✓, , x(i) ) = DKL[q (z|x(i) )||p✓(z)] + 1 L LX l=1 logp✓(x(i) |z(i,l) )
- 14. Reparameteriza%on Trick ✏ ⇠ p(✏) z ⇠ q (z|x) determinis%c variable auxiliary variable z = µ + ✏ Example: ✏ ⇠ N(0, 1) z ⇠ p(z|x) = N(µ, 2 ) z = g (✏, x)
- 15. Reparameteriza%on Trick z(i,l) = µ(i) + (i) ✏(l) x(i) Encoder Networks q (z|x) logq (z|x(i) ) = logN(z, µ(i) , 2(i) I) ✏ ⇠ N(0, I)
- 16. Reparameteriza%on Trick z(i,l) = µ(i) + (i) ✏(l) x(i) Encoder Networks x(i) q (z|x) p✓(x|z) z(i,l) Decoder Networks ✏ ⇠ N(0, I)
- 17. Objec%ve Func%on ˜L(✓, , x(i) ) = DKL[q (z|x(i) )||p✓(z)] + 1 L LX l=1 (logp✓(x(i) |z(i,l) ) ˜L(✓, , x(i) ) = 1 2 JX j=1 (1+log(( (i) j )2 ) (µ (i) j )2 ( (i) j )2 )+ 1 L LX l=1 (logp✓(x(i) |z(i,l) ) Regulariza%on Reconstruc%on Error p✓(z) = N(z, 0, I) q (z|x(i) ) = N(z, µ(i) , 2(i) I)
- 18. Training
- 20. Experiment