Evaluation of Generative Models Reference Lucas Theis Aron

Evaluation of Generative Models

Reference • Lucas Theis, Aäron van den Oord, Matthias Bethge, “A note on the evaluation of generative models”, ar. Xiv preprint, 2015

Likelihood : real data (not observed during training) : generated data Prior Distribution Generator PG Log Likelihood:

https: //stats. stackexchange. com/questions/244012/can-youexplain-parzen-window-kernel-density-estimation-in-laymansterms/244023 Likelihood - Kernel Density Estimation • Estimate the distribution of PG(x) from sampling Generator PG Each sample is the mean of a Gaussian with the same covariance. Then we can compute the likelihood.

https: //arxiv. org/pdf/1511. 01844. pdf Likelihood - Kernel Density Estimation • How many samples? • Weird results? Model DBN GAN True Distribution Likelihood 138 225 K-means 313 243 ?

Likelihood v. s. Quality • Low likelihood, high quality? Considering a model generating good images (small variance) Generator PG Generated data X 0. 99 • High likelihood, low quality? G 2 G 1 100 4. 6 X 0. 01

Evaluate by Other Networks Well-trained CNN Lower entropy means higher visual quality CNN CNN …… Higher entropy means higher variety

Evaluate by Other Networks - Inception Score Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, “Improved Techniques for Training GANs”, ar. Xiv prepring, 2016 Negative entropy Cross entropy

Evaluate by Other Networks - Inception Score • Improved W-GAN

https: //arxiv. org/pdf/1511. 01844. pdf K-Nearest Neighbor • Using k-nearest neighbor to check whether the generator generates new objects

Mode Collapse https: //devblogs. nvidia. com/parallelforall/photo-editing-generative-adversarial-networks-2/

https: //arxiv. org/pdf/1612. 02136. pdf Missing Mode Missing anything? Discriminator always knows it is real with high confidence