Generative Adversarial Networks MLCV 182 Lecturer Oren Freifeld

Generative Adversarial Networks MLCV 182 Lecturer: Oren Freifeld, TA: Ron Shapira Weber

. Generative Models • [Goodfellow, I. (2016). NIPS 2016 tutorial: Generative adversarial networks. ]

. Why use generative models? • Un-supervised / Semi-supervised learning. • Working with multi-modal output – multiple different correct answers. • Sample from a desired distribution (e. g. from the posterior).

Multi-modal output: Next video frame prediction [Lotter el al, . (2015). Unsupervised learning of visual structure using predictive generative networks]

Image-to-image translation [Zhu et al. , (2017). Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks]

Image-to-image translation [Wolterink et al. , (2017). MR-to-CT Synthesis using Cycle-Consistent Generative Adversarial Networks]

. Generative Adversarial Networks “Supervised learning” (real or fake)

. Generative Adversarial Networks • The two players in the game are represented by two functions, each of which is differentiable both with respect to its inputs and with respect to its parameters. • A game between D, G rather than an optimization problem.

. The Generator •

. [Goodfellow, I. (2016). NIPS 2016 tutorial: Generative adversarial networks. ]

. Cost functions (Vanilla version) •

. Training •

. Gradient saturation •

. Cost functions – non-saturating game •

. Cost functions – non-saturating game

. Generative Adversarial Networks Figure copyrights: https: //www. kdnuggets. com/2017/01/generative-adversarial-networks-hot-topic-machine-learning. html

. Generative Adversarial Networks Generator Discriminator Input Parameters (To optimize) Cost function Output Scalar: probability of input being real. Figure copyrights: https: //www. kdnuggets. com/2017/01/generative-adversarial-networks-hot-topic-machine-learning. html

. GANs Architectures • • DCGANs - (Randford et al. , 2015) Conditional GANs - (Mirza, M. , & Osindero, S. 2014) Info. GANs – (Chen et al. , 2016 a) Cycle-Consistent Adversarial Networks – (Zhu et al. , 2017)

. Deep Convolution GAN – “DCGAN” (Randford et al. , 2015) [Figure copyrights Randford et al. , (2015), Unsupervised Representation Learning with Deep Convolutional GAN]

. DCGAN • Learns reusable feature representation from unlabeled datasets. • Train D and G via GAN framework later reusing parts of G and D networks as: • D: Feature extractors for supervised tasks (e. g. image classification). • G: Generates images. [Figure copyrights Randford et al. , (2015), Unsupervised Representation Learning with Deep Convolutional GAN]

. DCGAN •

. DCGAN • [Figure copyrights Randford et al. , (2015), Unsupervised Representation Learning with Deep Convolutional GAN]

. DCGAN • Vector arithmetic for visual concepts [Figure copyrights Randford et al. , (2015), Unsupervised Representation Learning with Deep Convolutional GAN]

. DCGAN • Pixel space arithmetic doesn’t work very well… [Figure copyrights Randford et al. , (2015), Unsupervised Representation Learning with Deep Convolutional GAN]

. Conditional GAN (Mirza, M. , & Osindero, S. (2014))

. Conditional GAN (Mirza, M. , & Osindero, S. (2014)) • In the generator the prior input noise pz(z), and y are combined in joint hidden representation. • In the discriminator x and y are presented as inputs and to a discriminative function. • Original cost function: • Conditional GAN cost function: [Mirza, M. , & Osindero, S. (2014). Conditional Generative Adversarial Nets]

. Conditional GAN [Figure copyrights Mirzaet al. , (2014). Conditional Generative Adversarial Nets]

. Conditional GAN [Figure copyrights Mirzaet al. , (2014). Conditional Generative Adversarial Nets]

. Info. GAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets (Chen et al. , 2016 a)

. Info. GAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets (Chen et al. , 2016 a) •

. Info. GAN: MNIST example • [Figure copyrights Chen, X et al. , (2016). Info. GAN … ]

. Info. GAN: 3 D faces example • [Figure copyrights Chen, X et al. , (2016). Info. GAN … ]

. Unpaired Image-to-Image Translation Using Cycle. Consistent Adversarial Networks (Zhu et al. , 2017)

. Unpaired Image-to-Image Translation Using Cycle. Consistent Adversarial Networks (Zhu et al. , 2017) [Figure copyrights Zhu et al. , (2017). Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. ]

. Unpaired Image-to-Image Translation Using Cycle. Consistent Adversarial Networks [Figure copyrights Zhu et al. , (2017). Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. ]

. Unpaired Image-to-Image Translation Using Cycle. Consistent Adversarial Networks • [Figure copyrights Zhu et al. , (2017). Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. ]

. Unpaired Image-to-Image Translation Using Cycle. Consistent Adversarial Networks • Full objective: Where: [Figure copyrights Zhu et al. , (2017). Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. ]

. Some examples: [Figure copyrights Zhu et al. , (2017). Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. ]

. Some examples: [Figure copyrights Zhu et al. , (2017). Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. ]

. Some examples: [Figure copyrights Zhu et al. , (2017). Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. ]

. Some fails [Figure copyrights Zhu et al. , (2017). Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. ]

. GANs Architectures •

. Thanks!