Arbitrary Style Transfer via MultiAdaptation Network Yingying Deng

Arbitrary Style Transfer via Multi-Adaptation Network Yingying Deng, Fan Tang, Weiming Dong, Wen Sun, Feiyue Huang, Changsheng Xu

Outline 01 INTRODUCTION 02 RELATEDWORK 03 METHODOLOGY • • Multi-adaptation Module Loss Function 04 EXPERIMENTS • • Comparison with Prior Work Applications 05 CONCLUSIONS AND FUTUREWORK

01 & 02 INTRODUCTION and RELATEDWORK

INTRODUCTION - Artistic style transfer Rendering natural images with artistic style patterns and maintaining the content structure of natural images at the same time.

RELATEDWORK image-to-image translation arbitrary style transfer methods Feature Disentanglement • VGG 16 • Second-order statistics • disentanglement loss • feed-forward neural networks • Style attention • • Patch-swap based Methods style and content disentangled GAN • encoder-decoder network • GAN

RELATEDWORK image-to-image translation arbitrary style transfer methods Feature Disentanglement • VGG 16 • Second-order statistics • disentanglement loss • feed-forward neural networks • Patch-swap based Methods • style and content disentangled GAN • Style attention • encoder-decoder network Leon A Gatys, Alexander S Ecker, and Matthias Bethge. 2016. Image style transfer using convolutional neural networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2414– 2423.

RELATEDWORK image-to-image translation arbitrary style transfer methods Feature Disentanglement • VGG 16 • Second-order statistics • disentanglement loss • feed-forward neural networks • Patch-swap based Methods • style and content disentangled GAN • Style attention • encoder-decoder network Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for realtime style transfer and super-resolution. In European Conference on Computer Vision (ECCV). Springer, 694– 711.

RELATEDWORK image-to-image translation arbitrary style transfer methods Feature Disentanglement • VGG 16 • Second-order statistics • disentanglement loss • feed-forward neural networks • Patch-swap based Methods • style and content disentangled GAN • Style attention • encoder-decoder network Dmytro Kotovenko, Artsiom Sanakoyeu, Pingchuan Ma, Sabine Lang, and Bjorn Ommer. 2019. A Content Transformation Block for Image Style Transfer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 10032– 10041.

RELATEDWORK image-to-image translation arbitrary style transfer methods Feature Disentanglement • VGG 16 • Second-order statistics • disentanglement loss • feed-forward neural networks • Style attention • • Patch-swap based Methods style and content disentangled GAN • encoder-decoder network • GAN Xun Huang and Belongie Serge. 2017. Arbitrary style transfer in real-time with adaptive instance normalization. In IEEE International Conference on Computer Vision (ICCV). IEEE, 1501– 1510.

RELATEDWORK image-to-image translation arbitrary style transfer methods Feature Disentanglement • VGG 16 • Second-order statistics • disentanglement loss • feed-forward neural networks • Style attention • • Patch-swap based Methods style and content disentangled GAN • encoder-decoder Yuan Yao, Jianqiang Ren, Xuansong Xie, Weidong Liu, network Yong-Jin Liu, and Jun Wang. • GAN 2019. Attention-aware multi -stroke style transfer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 1467– 1475.

RELATEDWORK image-to-image translation arbitrary style transfer methods Feature Disentanglement • VGG 16 • Second-order statistics • disentanglement loss • feed-forward neural networks • Style attention • • Patch-swap based Methods style and content disentangled GAN • encoder-decoder network • GAN Tian Qi Chen and Mark Schmidt. 2016. Fast patch-based style transfer of arbitrary style. In Constructive Machine Learning Workshop, NIPS.

RELATEDWORK image-to-image translation arbitrary style transfer methods Feature Disentanglement • VGG 16 • Second-order statistics • disentanglement loss • feed-forward neural networks • Style attention • • Patch-swap based Methods style and content disentangled GAN • encoder-decoder network • GAN Dmytro Kotovenko, Artsiom Sanakoyeu, Sabine Lang, and Bjorn Ommer. 2019. Content and style disentanglement for artistic style transfer. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). IEEE, 4422– 4431.

RELATEDWORK image-to-image translation arbitrary style transfer methods Feature Disentanglement • VGG 16 • Second-order statistics • disentanglement loss • feed-forward neural networks • Style attention • • Patch-swap based Methods style and content disentangled GAN • encoder-decoder network • GAN Hadi Kazemi, Seyed Mehdi Iranmanesh, and Nasser Nasrabadi. 2019. Style and content disentanglement in generative adversarial networks. In IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 848– 856.

RELATEDWORK image-to-image translation arbitrary style transfer methods Feature Disentanglement • VGG 16 • Second-order statistics • disentanglement loss • feed-forward neural networks • Style attention • • Patch-swap based Methods style and content disentangled GAN • encoder-decoder network • GAN 康姆士 Xiaoming Yu, Yuanqi Chen, Shan Liu, Thomas Li, and Ge Li. 2019. Multi-mapping Image-to -Image Translation via Learning Disentanglement. In Advances in Neural Information Processing Systems (Neur. IPS). 2990– 2999.

RELATEDWORK image-to-image translation arbitrary style transfer methods Feature Disentanglement • VGG 16 • Second-order statistics • disentanglement loss • feed-forward neural networks • Style attention • • Patch-swap based Methods style and content disentangled GAN • encoder-decoder network • GAN

INTRODUCTION – Purpose & Main Contributions ● propose a flexible and efficient arbitrary style transfer model ○ ● multi-adaptation network that involves two Self-Adaptation (SA) modules and one Co-Adaptation (CA) module disentanglement mechanism ○ Content disentanglement loss ○ Style disentanglement loss

03 METHODOLOGY

OVERVIEW

Encoder-Decoder Encoder pretrained VGG 19 network (use conv 4_1) Decoder mirrored version of the encoder

Multi-adaptation Module

Multi-adaptation Module convolution kernel sizes utilized in the multi-adaptation module are all set to 1 × 1. Content/Style Conv kernel 1× 1 Conv kernel 3× 3

Multi-adaptation Module

Multi-adaptation Module

Position-wise Content Self-adaptation Module

Channel-wise Style Self-adaptation Module. channel-wise inner product between the vectorized feature maps can represent style

Co-adaptation Module

OVERALL MODEL

Loss Function - Perceptual Loss ● content perceptual loss (conv 4_1) ● style perceptual loss (conv 1_1, conv 2_1, conv 3_1, and conv 4_1) ϕi (・) : the features extracted from i-th layer in a pretrained VGG 19 μ(・) : the mean of features σ(・) : variance of features

Loss Function - Identity Loss ● Style Content Result

Loss Function - Disentanglement Loss ● 2

Loss Function - Disentanglement Loss

Loss Function

Implementation Details ● ● ● Use MS-COCO as content dataset and Wiki. Art as style dataset. The style and content images are randomly cropped to 256 × 256 pixels in the training stage. All image sizes are supported in the testing stage

04 EXPERIMENTS

Comparison with Prior Work - Qualitative Evaluation Style Content Ours Ada. IN WCT SANet AAMS

Comparison with Prior. Work - Qualitative Evaluation Different color distributions from style images Style Content Ours Ada. IN WCT SANet AAMS

Comparison with Prior Work - Qualitative Evaluation content distortion Style Content Ours Ada. IN WCT SANet AAMS

Comparison with Prior Work - Qualitative Evaluation content structures in stylized results are unclear Style Content Ours Ada. IN WCT SANet AAMS

Comparison with Prior Work - Qualitative Evaluation main structures of the content images are clear, but the other structures are damaged and the style patterns in the generated image are not evident Style Content Ours Ada. IN WCT SANet AAMS

Comparison with Prior Work - Qualitative Evaluation Style Content Ours Ada. IN WCT SANet AAMS

Comparison with Prior Work - User Study

Comparison with Prior Work - Quantitative Evaluation fine-tune the pretrained VGG 19 model to classify content / style

Applications

05 CONCLUSIONS AND FUTUREWORK

CONCLUSIONS AND FUTUREWORK • CONCLUSIONS • propose a multi-adaptation network • propose a disentanglement loss • • This method can achieve a trade-off between content structure preservation and style pattern rendering. FUTUREWORK • develop a style image selection method to recommend appropriate style images for a given content image

THANKS! CREDITS: This presentation template was created by Slidesgo, including icons by Flaticon, infographics & images by Freepik and illustrations by Stories