gan

GAN

KL Divergence, Jensen-Shannon Divergence and f-divergence

Given two probability distributions p and q,

KL divergence $D_{\text{KL}}(p\Vert q)$ measures how “far” q is from p without requiring that D is a metric.

$$\begin{array}{rl}{D}_{\text{KL}}(p\Vert q)& =\sum _{k=1}^K{p}_k\log \frac{p_k}{q_k}=\int p(x)\log \frac{p(x)}{q(x)}dx\\ & =\underset{-\mathbb{H}(p)}{\underbrace{\sum _{k=1}^K{p}_k\log p_k}}-\underset{{\mathbb{H}}_{\text{ce}}(p,q)}{\underbrace{\sum _{k=1}^K{p}_k\log q_k}}\end{array}$$

• $D_{\text{KL}}\ge 0$ with equality iff p=q. Can be proven using Jensen’s inequality.
• Asymmetric, i.e. $D_{\text{KL}}(p\Vert q)\ne D_{\text{KL}}(q\Vert p)$
• Minimising KL divergence to the empirical distribution is equivalent to maximising likelihood.

Jensen-Shannon divergence

$$D_{\text{JS}}(p\Vert q)=\frac{1}{2}{D}_{\text{KL}}\left(p\Vert \frac{p+q}{2}\right)+\frac{1}{2}{D}_{\text{KL}}\left(q\Vert \frac{p+q}{2}\right)$$

JS divergence is symmetric and more smooth.

GAN consists of two models:

• Discriminator or critic $D_{\varphi }:\mathcal{X}\to \Delta$ is a comparison model that estimates the probability that the sample $q$ matches the true distribution $p^{\ast }$.
• Generator $G_{\theta }:\mathcal{Z}\to \mathcal{X}$ that outputs samples, given a latent variable $z$, as close to true distribution. Or informally, generator’s job is to trick the critic into offering a high probability for the synthetic output.

Our goal is to play the adversarial game between $D_{\varphi }$ and $G_{\theta }$, where the generator tries to create images that match the true distribution as close as possible, and critic becomes better at detecting any errors generator is making by classifying the generated samples.

We can compare the two distribution by computing the density ratio $r(x)=\frac{p^{\ast }(x)}{q_{\theta }(x)}$, and converting the problem into binary classification: $\frac{p^{\ast }(x)}{q_{\theta }(x)}=\frac{D(x)}{1-D(x)}$. Using cross-entropy loss, objective becomes

$$\begin{array}{rl}V(q_{\theta },p^{\ast })& =\underset{\varphi }{\mathrm{argmax}}{E}_{p(x|y)p(y)}[y\log D_{\varphi }(x)+(1-y)\log (1-D_{\varphi }(x))]\end{array}$$

Optimal discriminator’s maximises the probability of classifying true distribution $E_{x\sim p_{\mathcal{D}}(x)}[\log D_{\varphi }(x)]$, and maximises the probability of detecting a fake sample $E_{z\sim p_z}[\log (1-D_{\varphi }(G_{\theta }(z)))]$. While generator’s objective is to minimise the discriminator’s probability of classifying incorrect samples $E_{z\sim p_z}[\log (1-D_{\varphi }^{\ast }(G_{\theta }(z)))]$.

Together this turns into minimax game:

$$\arg \underset{\theta }{\min }\underset{\varphi }{\max }{E}_{x\sim p_{\mathcal{D}}}[\log D_{\varphi }(x)]+E_{z\sim p_z}[\log (1-D_{\varphi }(G_{\theta }(z)))]$$