Structurepreserving style transfer Santiago Calvo Ana Serrano Diego

Structure-preserving style transfer Santiago Calvo Ana Serrano Diego Gutierrez Belen Masia Universidad de Zaragoza 1

Style transfer Goal: Transfer style from a source image to a target image Source image Style Target image Content 2

Style transfer Goal: Transfer style from a source image to a target image Source image Style Our result Target image Content 3

Related work Target image Result Source image [Gatys 2016] Image Style Transfer Using Convolutional Neural Networks (CVPR 2016) 4

Related work Target image Result Source image [Huang 2017] Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization ( ICCV 2017) 5

Related work Target image Result Source image [Luan 2017] Deep Photo Style Transfer (CVPR 2017) 6

Related work Target image Result Source image https: //prisma-ai. com/ 7

Style – Content trade-off Target image Source image [Gatys 2016] Image Style Transfer Using Convolutional Neural Networks (CVPR 2016) 8

Style – Content trade-off Target image Source image Preserving content [Gatys 2016] Image Style Transfer Using Convolutional Neural Networks (CVPR 2016) 9

Style – Content trade-off Target image Source image Preserving content Transferring style [Gatys 2016] Image Style Transfer Using Convolutional Neural Networks (CVPR 2016) 10

Structure-preserving style transfer Goal: Transfer style from a source image to a target image Source image Style Our result Target image Content 11

Neural style transfer [Gatys 2016] Image Style Transfer Using Convolutional Neural Networks (CVPR 2016) - VGG Network (trained for object recognition) - 16 convolutional layers and 5 pooling layers - No fully connected layers 12

Neural style transfer 13

Neural style transfer conv 1_2 14

Neural style transfer conv 2_2 15

Neural style transfer conv 3_2 16

Neural style transfer conv 4_2 17

Neural style transfer conv 5_2 Content representation: Activations of the filter i at a position j in a layer l 18

Neural style transfer 19

Neural style transfer 20

Neural style transfer conv 1_1 21

Neural style transfer conv 1_1 conv 2_1 22

Neural style transfer conv 1_1 conv 2_1 conv 3_1 23

Neural style transfer conv 1_1 conv 2_1 conv 3_1 conv 4_1 24

Neural style transfer conv 1_1 conv 2_1 conv 3_1 conv 4_1 conv 5_1 25

Neural style transfer Style representation: Inner product between vectorised feature maps Activations of the filter i at a position k in a layer l conv 1_1 conv 2_1 conv 3_1 conv 4_1 conv 5_1 26

Neural style transfer 27

Neural style transfer 28

Neural style transfer 29

Neural style transfer 30

Neural style transfer Intuition: Higher layers in the network capture the highlevel content in terms of objects and their arrangement in the input image 31

Neural style transfer 32

Neural style transfer Intuition: Correlations between multiple layers Common features across scales instead of global arrangement (style) 33

Neural style transfer 34

Neural style transfer - Linear combination between the content and the style loss 35

Neural style transfer - Linear combination between the content and the style loss - Compute derivative w. r. t. the pixel values using error back-propagation 36

Neural style transfer - Linear combination between the content and the style loss - Compute derivative w. r. t. the pixel values using error back-propagation - Use this gradient to iteratively update the input image until it simultaneously matches the style features and the content features 37

Neural style transfer 38

Structure-preserving style transfer Split the image in patches with different amount of detail Apply different style weights Merge back the stylized patches 39

Structure-preserving style transfer Splitting the image - Quadtree decomposition 40

Structure-preserving style transfer Splitting the image - Quadtree decomposition - Select style weight as a function of detail 41

Structure-preserving style transfer Merging back the image 42

Structure-preserving style transfer Merging back the image Expand the boundaries of the stylized patches and compute the mean of overlapped regions 43

Results [Gatys 2016] Ours 44

Results [Gatys 2016] Ours 45

Results [Gatys 2016] Ours 46

Extension to video 47

Extension to video [Gatys 2016] Ours 48

Conclusions and future work Conclusions - Structure-preserving style transfer that introduces an alternative to alleviate the contentstyle trade-off - Can be easily applied to different style transfer approaches - Can be adapted to video style transfer 49

Conclusions and future work Conclusions - Structure-preserving style transfer that introduces an alternative to alleviate the contentstyle trade-off - Can be easily applied to different style transfer approaches - Can be adapted to video style transfer Limitations - Lowering the intensity of the style transfer for some regions can cause that some results will not completely convey the desired style 50

Conclusions and future work Conclusions - Structure-preserving style transfer that introduces an alternative to alleviate the contentstyle trade-off - Can be easily applied to different style transfer approaches - Can be adapted to video style transfer Limitations - Lowering the intensity of the style transfer for some regions can cause that some results will not completely convey the desired style Future work - Take into account perceptual aspects to identify the regions or features of the image that maximize the impression of style 51

Thanks! Structure-preserving style transfer Santiago Calvo Ana Serrano Diego Gutierrez Belen Masia Universidad de Zaragoza 52