Image Aesthetic Assessment Based on Pairwise Comparison A

Image Aesthetic Assessment Based on Pairwise Comparison A Unified Approach to Score Regression, Binary Classification, and Personalization ICCV 2019 (Poster @ Applications. Medical, & Robotics) Jun-Tae Lee and Chang-Su Kim Media Communications Lab, Korea University Presenter: Dingquan Li

Outline • Problem • Background & Motivation • Method • Experiments • Summary

Problem

Outline • Problem • Background & Motivation • Method • Experiments • Summary

Background & Motivation • Most works focus on binary aesthetic classification while few works focus on aesthetic score regression and personalized aesthetics. • The latter ones are more challenging than the former one. • The latter ones are needed in some applications, such as image recommendation, image enhancement, and personal album curation. Can these three tasks be unified?

Outline • Problem • Background & Motivation • Method • Experiments • Summary

Method • Aesthetic Comparator for Pairwise Comparison (PC) • Aesthetic Score Regression Based on PC Matrix • Binary Aesthetic Classification by Comparing with Middles • Personalized Image Aesthetics by Extending the Generic One

Aesthetic Comparator

Feature Extractors

Ternary Classifier: Ground Truth Classes Lloyd, Stuart. "Least squares quantization in PCM. " IEEE Transactions on Information Theory 28. 2 (1982): 129 -137.

Ternary Classifier: Loss Function

Method • Aesthetic Comparator for Pairwise Comparison (PC) • Aesthetic Score Regression Based on PC Matrix • Binary Aesthetic Classification by Comparing with Middles • Personalized Image Aesthetics by Extending the Generic One

Constructing PC Matrix

PC Matrix Example

Priority Vector of Aesthetic Scores Saaty, Thomas L. "A scaling method for priorities in hierarchical structures. " Journal of Mathematical Psychology 15. 3 (1977): 234 -281.

Aesthetic Score Vector Priority vector Score vector

Estimate Scale Factor

Aesthetic Score Regression

Reference Image Selection •

Method • Aesthetic Comparator for Pairwise Comparison (PC) • Aesthetic Score Regression Based on PC Matrix • Binary Aesthetic Classification by Comparing with Middles • Personalized Image Aesthetics by Extending the Generic One

Method • Aesthetic Comparator for Pairwise Comparison (PC) • Aesthetic Score Regression Based on PC Matrix • Binary Aesthetic Classification by Comparing with Middles • Personalized Image Aesthetics by Extending the Generic One

Extend the Generic One for Personalization

PC Matric Example

Outline • Problem • Background & Motivation • Method • Experiments • Summary

Experiments • Datasets • Aesthetic Score Regression • Binary Aesthetic Classification • Personalized Image Aesthetics

Datasets • AVA (CVPR 2012): 235599 images for training (choosing 2000 for validation) and 19930 images for testing. Ratings are from 110. • AADB (ECCV 2016): 8500/1000 images for training/validation/testing. Score ranges are [0, 1]. • FLICKER-AES (ICCV 2017 ): 35263 images for training and 4737 images for testing. Score ranges are [0, 5]. • Note: there is no overlap between workers of training image and testing images (173+37=210). As for the test images, each worker rated about 137 images.