A perceptual metric for photo retouching Eric Kee

A perceptual metric for photo retouching Eric Kee and Hany Farid Presented by: C. Alfaro C. Andrade P. Fonseca J. Vargas

Seminar Series MO 447 - Digital Forensics Prof. Dr. Anderson Rocha anderson. rocha@ic. unicamp. br http: //www. ic. unicamp. br/~rocha

Outline

Outline Introduction Materials & Methods Results Discussion Conclusions ‣ ‣ ‣ 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Introduction

[2] [1] source: [1] http: //www. pnas. org/content/early/2011/11/21/1110747108. full. pdf [2] http: //www. theguardian. com/media/2009/dec/16/twiggys-olay-ad-banned-airbrushing 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Before - After retouching Penelope Keira Cruz Knightley Source: http: //www. smosh. com/smosh-pit/photos/10 -celebs-and-after-photoshop-gifs 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Before - After retouching George Niall Horan Clooney Source: http: //www. smosh. com/smosh-pit/photos/10 -celebs-and-after-photoshop-gifs 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

More retouching Photo from a Nordstrom magazine. Source: http: //www. psdisasters. com/2013/05/nordstrom-nice-neck. html 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

More retouching Source: http: //theberry. com/2013/07/11/getting-a-little-photoshop-happy-18 -photos/photoshop-fail-1 -4/ 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

More retouching Source: http: //www. psdisasters. com/2013/08/nail-art-sausage-fingers. html 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

More retouching Before After Source: http: //www. psdisasters. com/2013/08/botox-wrinkle-rewind. html 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Motivation The idealistic (and unreal) beauty concept conveyed through media leads to (severe) problems in public health. Diseases such as anorexia, bulimia, body dysmorphic disorder, and psychological issues arise as problems related to appearance dissatisfaction. ‣ ‣ 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Motivation According to [2]: two thirds of underweight 12 -year-old girls considered themselves to be too fat; by 17, only 3 out of 10 girls have not been on a diet; among women over 18 looking at themselves in the mirror, research indicates that at least 80% are unhappy with what they see. ‣ ‣ 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Motivation In some countries (UK and France), laws are being proposed to require labeling for retouched photos. How to define a label for different levels of retouching? How to measure the amount of modification? ‣ ‣ 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Materials & Methods

Overview 468 original and retouched photos were collected from a variety of online sources and given a grade by human observers. For every before/after pair of photos, it was estimated the geometric and photometric modifications in order to obtain eight summary statistics, which were compared to the human ratings through SVR. ‣ ‣ 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Perceptual Ratings (I) A group of 390 observers was recruited through Amazon’s Mechanical Turk. Observers were initially shown a representative set of 20 images pairs so they would see the variation. Observers were asked to rate 70 pairs of before/after images in a scale from 1 (very similar) to 5 (very different) ‣ ‣ ‣ 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Perceptual Ratings (II) In order to measure the consistency of responses, each observer also rated a random set of five images three separate times, the presentations of which were uniformly distributed throughout the duration of the experiment. Observers are better able to see the modification when toggling rather than viewing side-by-side. ‣ ‣ 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Perceptual Ratings (III) After some filtering, approximately 9. 5% of the observers were excluded due to high inconsistency (usually because of high variance on the repeated trials or because they toggled the pair of photos only once). The final rating of an image pair is the mean of the rating from 50 different observers. Each observer was payed $3 for their participation and a typical session lasted 30 min. ‣ ‣ ‣ 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Geometric Retouchers change the geometry of the subjects Elongating the neck Slimming of legs, Improving posture ‣ ‣ ‣ Source: http: //www. cs. dartmouth. edu/farid/downloads/publications/pnas 11/beforeafter. tar 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Geometric Photoshop liquify tool is used to make geometric changes. ‣ Source: https: //www. youtube. com/watch? v=REll 3 OCahc. I 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Geometric The geometric transformation is modeled with 8 parameters. ‣ after Luminance Transformation: m 7 : contrast m 8 : brightness 2013 Seminar Series – Digital Forensics (MO 447/MC 919) before Affine Model: *

Geometric How to calculate Minimize : ‣ ? ‣ 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Geometric First order truncate Taylor series expansion and expressing in vector form: ‣ where 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Geometric Solution: ‣ 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Geometric Add constraints to avoid degenerate solutions Solution where a pure brightness (m 8) modulation is used ‣ ‣ where ‣ : average of 2013 Seminar Series – Digital Forensics (MO 447/MC 919) in a neighborhood *

Geometric What to do with missing data? ‣ Before After Source: http: //www. cs. dartmouth. edu/farid/downloads/publications/pnas 11/beforeafter. tar 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Geometric Model 1: Pixel in the before and after image are either related through the transformation. Model 2: Otherwise ‣ ‣ 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Geometric Use a weight , that is proportional to the likelihood that each pixel belong to model M 1. ‣ 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Geometric Expectation/Maximization Algorithm (1) E-step: Compute weights (2) M-step: Estimate the model parameters (3) Repeat step 1 and 2 until the difference between successive estimates of is below specified threshold. ‣ 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Geometric Once estimated, the geometric transformation is represented as a dense two-dimensional (2 D) vector field. ‣ source: http: //www. pnas. org/content/early/2011/11/21/1110747108. full. pdf 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Photometric Region Based. The before region is aligned to the after one. The photometric modifications are modelled with a 9× 9 linear filter and a SSIM measure. Targets: basic blurring, sharpening, and special effects afforded by various Photoshop filters. ‣ ‣ 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Filter model The region in the after image is supposed to be the region in the before image convolved with a 9 x 9 Filter. ‣ 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Filter Estimation The 9 x 9 filter is estimated using a conjugate gradient descent with a Tikhonov regularization. ‣ Regularization is used to enforce symmetry in the filter. ‣ 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Filter’s Frequency Response Once the filter is estimated, the measure D is computed as following: Where Fb and H are the fourier transforms of the before image and the filter h. 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Filter’s Frequency Response Patches of the size of the filter are extracted. The Fourier transform is computed for the filter and for the patch. Interpretation: If D>0, h is a blurring filter. If D<0, h is a sharpening filter. ‣ ‣ 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

SSIM measure (I) Photometric modifications that are not captured by the linear filter are measured with SSIM (contrast and structural modifications) Brightness term is excluded because of the SSIM implementation. SSIM is computed only on the luminance channel of the color image. ‣ ‣ ‣ 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

SSIM measure (II) © Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simonelli; Image Quality Assessment: From Error Visibility to Structural Similarity. 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

SSIM measure (III) The following measures comparison functions: Mean Intensity Standard Deviation : Correlation coefficient : ‣ used to make : ‣ ‣ are (Part ) 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

SSIM measure (IV) Comparison functions: Luminance Comparison : ‣ Contrast Comparison ‣ Structure Comparison ‣ 2013 Seminar Series – Digital Forensics (MO 447/MC 919) : : *

SSIM measure (V) Combined the comparison functions to yield and overall similarity measure: ‣ Choosing the multiplication and some parameters to adjust the relative importance as a function of combination f yield the SSIM measure: ‣ 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

SSIM measure (VI) In this paper, the authors only consider the Contrast and Structure comparisons. ‣ Where: 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Perceptual Distortion The amount of photo distortion will be learned. Eight summary statistics from: Geometric model. Photometric model. ‣ ‣ 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Perceptual Distortion Statistics (I) Geometric model (1 -2 Face) the mean and standard deviation of the magnitude of the estimated vector field. (3 -4 Body) the mean and standard deviation of the magnitude of the estimated vector field. ‣ ‣ ‣ 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Perceptual Distortion Statistics (II) Photometric (5 -6 Face) the mean and standard deviation of the SSIM. (7 -8 Face) the mean and standard deviation for measure D of the frequency response of the linear filters h. ‣ ‣ ‣ 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Masks are calculated for the face, hair and torso regions. ‣ 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Masks Before Source: http: //www. cs. dartmouth. edu/farid/downloads/publications/pnas 11 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Masks After Source: http: //www. cs. dartmouth. edu/farid/downloads/publications/pnas 11 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Masks Face mask Source: http: //www. cs. dartmouth. edu/farid/downloads/publications/pnas 11 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Masks Hair mask Source: http: //www. cs. dartmouth. edu/farid/downloads/publications/pnas 11 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Masks Torso mask Source: http: //www. cs. dartmouth. edu/farid/downloads/publications/pnas 11 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Support Vector Regression (SVR) 8 summary statistics scaled between [-1, 1]. Gaussian Radial Basis Function kernel nu-SVR. Leave one out cross-validation performed (Training: 467, Testing: 1) using LIBSVM. Hyperparameters Gamma/Cost were determined by dense 2 D Grid Search. ‣ ‣ ‣ 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Results

Results: Minimum error source: http: //www. pnas. org/content/early/2011/11/21/1110747108. full. pdf 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Results: Maximum error source: http: //www. pnas. org/content/early/2011/11/21/1110747108. full. pdf 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Overall Results (I) For 99% of the images the algorithm won’t miss by more than 1 in the scale! ‣ 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Overall Results (II) R-value = 0. 80 Mean error = 0. 30 Median error = 0. 24 SD = 0. 24 ‣ ‣ Source: [1] 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Range of errors Absolute prediction error is: Less than 0. 50 for 81. 4% of the images Less than 0. 75 for 94. 4% of the images Less than 1. 00 for 99. 1% of the images ‣ ‣ 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Tested SVR Kernels R-value for the linear SVR is 0. 72 R-value for the non-linear SVR is 0. 80 ‣ ‣ 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Discussion

Discussion What about using this “perceptual measures” for phylogeny? Geometric Distortion Photometric Distortion ‣ ‣ ‣ 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Conclusions

Conclusions (I) The authors aimed to develop a system that would automatically rate how much an image was “photoshoped”. They extracted 8 high level descriptors and trained a SVM regressor (however with 6 of them they’ve got the same R-value). The approach succeeded: high correlation with human evaluators. ‣ ‣ ‣ 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Conclusions (II) However, the masks of hair/face/body are extracted manually: it’s not fully automated. Can be seen as a proof-of-concept for web service method available to publisher industry. If made available, a “cat-mouse” game between the system and publishers is possible. ‣ ‣ ‣ 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

References

References 1. Kee, E. and Farid, H. A perceptual metric for photo retouching. , ” Proc. Natl. Acad. Sci. U. S. A. , vol. 108, no. 50, pp. 19907– 12, Dec. 2011. 2. Mirror, mirror: A summary of research findings on body image. Available at: <http: //www. sirc. org/publik/mirror. html>. (Accessed on: 10 October 2013) 3. Photoshop Top 40 #29 - Liquify - https: //www. youtube. com/watch? v=REll 3 OCahc. I Source: Youtube (Accesed on : 19 October 2013) 4. Implementation and analysis of a perceptual metric for photo retouching. http: //white. stanford. edu/teach/index. php/Collins. Zvinakis. Danowitz#Perceptual_distortion_model (Accesed on : 19 October 2013) 5. Zhou Wang; Bovik, A. C. ; Sheikh, H. R. ; Simoncelli, E. P. , "Image quality assessment: from error visibility to structural similarity, " IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600, 612, April 2004. 6. Periaswamy S, Farid H. Elastic registration in the presence of intensity variations. IEEE Transactions on Medical Imaging. 22: 865– 874, 2003 7. Periaswamy S, Farid H. Medical image registration with partial data. Medical Image Analysis. 10: 452– 464. 2006 2013 Seminar Series – Digital Forensics (MO 447/MC 919) *

Thank You! Obrigado!