ContextAware Saliency Detection Stas Goferman Lihi ZelnikManor Ayellet

Context-Aware Saliency Detection Stas Goferman, Lihi Zelnik-Manor, Ayellet Tal CVPR, 2010

Outline � Introduction � Detection of context-aware saliency � Results � Application � Conclusion

Introduction(1) � Saliency identify fixation points? Original image Saliency map Fixation points This type of saliency is important for understanding human attention as well as for specific applications such as auto focusing.

Introduction(2) � Saliency identify the dominant object ? Original image Saliency map Itti’s By labeler This type of saliency is useful for several high-level tasks, such as object recognition or segmentation.

Introduction(3) � Context -aware For some application , like summarization of a photo collection , and retargeting… maintain the essential regions of the image is important.

Introduction(4) � Here , introducing a new type of saliency algorithm contain not only the prominent objects but also the parts of the background that convey the context. � The algorithm follows four psychological principles of human visual attention. � We demonstrate the applicability of two application, image retargeting and summarization.

Outline � Introduction � Detection of context-aware saliency � Results � Application � Conclusion

Principles of context-aware saliency � 1. Local low-level considerations, including factors such as contrast and color. the distinctive color and other features area should obtain high attention. � 2. Global considerations, which suppress frequently occurring features, while maintaining features that deviate from the norm. redundant information should be suppressed and popping up the novelty part.

Principles of context-aware saliency � 3. Visual organization rules, which state that visual forms may possess one or several centers of gravity about which the form is organized. the salient pixels should be grouped together, and not spread all over the image. � 4. High-level factors, such as human faces. implemented as post-processing.

Principles of context-aware saliency Comparing different approaches to saliency (a)Local[24] (b)Global[7] (c)Local-global[13] (d)Context-aware

Local-global single-scale saliency � 1. We should not, however, look at an isolated pixel, but rather at its surrounding patch. � 2. It suffices to consider the K most similar patches Let dcolor(pi , pj) be the Euclidean distance between the vectorized patches pi and pj in CIE L*a*b color space, dposition(pi , pj) be the Euclidean distance between the positions of patches pi and pj The single-scale saliency value of pixel i at scale r is defined as left. {qk} k=1 to K , K = 64 in our experiments

Multi-scale saliency enhancement � 3. Background pixels (patches) are likely to have similar patches at multiple scales. � 4. The larger Si is, the more salient pixel i is and the larger is its dissimilarity to the other patches. For a patch pi of scale r, we consider as candidate neighbors all the patches in the image whose scales are Rq = {r , 1/2 r , 1/4 r}. Let R denote the set of patch sizes to be considered for pixel i. The saliency at pixel i is taken as the mean of its saliency at different scales

Including the immediate context � 1: A pixel is considered attended if its saliency value exceeds a certain threshold( Si > 0. 8 in the examples shown in this paper). � 2: The saliency of a pixel is redefined as Let dfoci(i) be the Euclidean positional distance between pixel i and the closest focus of attention pixel, normalized to the range [0, 1]

Steps � The steps of our saliency estimation algorithm

High-level factors � Final , face detection algorithm modified Si , if Si > face(i) , otherwise Si = face detection algorithm of [23], which generates 1 for face pixels and 0 otherwise. [23] P. Viola and M. Jones. Rapid Object Detection Using a Boosted Cascade of Simple Features. In CVPR, 2001.

Outline � Introduction � Detection of context-aware saliency � Results � Application � Conclusion

Salient results (1) Compare to other methods with three cases image. (a) a single object over an uninteresting Background. (b) the immediate surroundings of the salient object is also salient. (c) complex scenes. From left to right input , result of [24] , result of [7], our result. [24] D. Walther and C. Koch. Modeling attention to salient proto objects. Neural Networks, 19(9): 1395– 1407, 2006. [7] X. Hou and L. Zhang. Saliency detection: A spectral residual approach. In CVPR, pages 1– 8, 2007.

Salient results (2) Comparing our saliency results with [13] Top: Input images. Middle: The bounding boxes obtained by [13] capture a single main object. Bottom: Our saliency maps convey the story. [13] T. Liu, J. Sun, N. Zheng, X. Tang, and H. Shum. Learning to Detect A Salient Object. In CVPR, 2007.

Outline � Introduction � Detection of context-aware saliency � Results � Application � Conclusion

Image retargeting aims at resizing an image by expanding or shrinking the non-informative regions. Distortions, if and when introduced, will exist only in regions of lower significance. Input Saliency of [19] Our saliency Results of [19] Our result Figure 9. Seam carving of 100 ”vertical“ lines. The salient objects are distorted by [19] in contrast to our results. [19]M. Rubinstein, A. Shamir, and S. Avidan. Improved seam carving for video retargeting. ACM Trans. on Graphics, 27(3), 2008.

Summarization through collage creation � this technique consists of three stages Compute saliency maps of the images Extract regions-of-interest(ROI) by considering both saliency and image edge information Assemble the non-rectangular ROIs , allowing slight overlaps. [4]S. Goferman, A. Tal, and L. Zelnik-Manor. Puzzle-like collage. Computer Graphics Forum (EUROGRAPHICS), 29, 2010.

Summarization through collage creation Summarization of a trip to LA using 14 images. (a)The saliency maps of the input images (b)The collage summarization

Outline � Introduction � Detection of context-aware saliency � Results � Application � Conclusion

Conclusion � This paper proposes a new type of saliency – context aware saliency based on four principles observed in the psychological literature which detects the important parts of the scene. � In the future we intend to learn the benefits of this saliency in more applications, such as image classification and thumbnailing.

Appendix � Global methods : Saliency Detection: A Spectral Residual Approach [7] H(Image) = H(Innovation) + H(Prior Knowledge) R(f) = L(f) − A(f) = hn (f) ∗ L(f)