Modeling Perspective Effects in Photographic Composition Zihan Zhou

Modeling Perspective Effects in Photographic Composition Zihan Zhou, Siqiong He, Jia Li, and James Z. Wang The Pennsylvania State University

What is Photo Composition? • Everyone wants to take good pictures… In photography, it’s not just what you shoot that counts – how you organize them within the frame is crucial, too. 2

The Use of Perspective Effects in Photographic Composition • In photography, experienced photographers often make use of the linear perspective effect to emphasize the sense of 3 D space in a 2 D photo • Can we teach computer to understand the use of perspective effects in photo composition? Photo credit: Flickr 3

Real World Applications • Image enhancement • Image summarization and retrieval for large-scale database • On-site composition feedback to photographers Does my photo look good? 4

How Should We Model the Perspective Effects? • Our Approach: A geometric image segmentation • Partition an image into regions (approx. planar structures) according to the dominant vanishing point. • Holistic: it encodes accurate, global geometric information about the scene • Compact: it can be efficiently employed in real-world applications 5

Challenge 1 • How to partition the image into photometrically and geometrically consistent regions? • Existing image segmentation methods do NOT respect the perspective effects in photos P. Arbelaez, et al. Contour detection and hierarchical image segmentation. TPAMI, 2011 6

Challenge 2 • How to detect the dominant VP in an arbitrary image? • Clustering line segments? 7

Our Contributions 1. The first work to model the perspective effects in photographic composition via a geometric image segmentation framework • Beyond photometric cues 2. The first work to detect the dominant VP in arbitrary images • Without relying on edges 3. Applications to on-site composition feedback 8

Review of Hierarchical Image Segmentation An oversegmentation of the image Find two regions with minimum distance Merge them and update the distances of relevant pairs Repeat Original Image Initial Over-segmentation Final Result P. Arbelaez, et al. Contour detection and hierarchical image segmentation. TPAMI, 2011 9

Geometric Distance Measure • Intuition: If the boundary between two regions is parallel to the dominant direction, these two regions are likely to lie on different planes. 10

Geometric Distance Measure • Our approach: measures the similarity of angle distribution of each region w. r. t. the dominant VP in a polar coordinate system 11

Combining Photometric and Geometric Cues 12

Quantitative Evaluation: Image Segmentation • 200 images from Flickr • Manually labeled ground truth segmentation • Compared against g. Pb-owt-ucm • P. Arbelaez, et al. Contour detection and hierarchical image segmentation. TPAMI’ 11 13

Detecting the Dominant Vanishing Point • A simple exhaustive search: • For each candidate location on a uniform grid mesh, compute: Correct Hypothesis Incorrect Hypothesis Consensus Scores for All Hypotheses 14

Quantitative Evaluation: VP Detection • 400 images from Flickr • Manually labeled vanishing points • Compared against [Tardiff, 2009] and [Tretiak et al. , 2012] 15

Application: On-Site Feedback via Composition-Sensitive Image Retrieval • Exemplar dataset: 3, 728 images collected from Flickr by querying the keyword “vanishing point”. Does my photo look good? • Similarity measure: 16 Geometric segmentation map VP location 16

Image Retrieval Results Queries 17

Summary • We propose to model the perspective effects in photographic composition via a novel geometric image segmentation framework • We develop a novel method to detect the dominant VP in arbitrary images • We demonstrate the applications of our model to on-site composition feedback • As future work, we plan to extend our method to more complex scenes • More than one VPs • Irrelevant foreground objects 18

Thank you! Photo credit: Flickr 19