Spectral Hashing Y Weiss Hebrew U A Torralba

Spectral Hashing Y. Weiss (Hebrew U. ) A. Torralba (MIT) Rob Fergus (NYU)

What does the. Motivation world look like? Objectlevel Recognition for large-scale search High image statistics

Semantic Hashing [Salakhutdinov & Hinton, 2007] Query Image Semantic Hash Function Quite different to a (conventional) randomizing hash Binary code Semantically similar images Address Space Images in database Query address

1. Locality Sensitive Hashing • Gionis, A. & Indyk, P. & Motwani, R. (1999) • Take random projections of data • Quantize each projection with few bits 101 0 Gist descriptor 1 0 1 1 0 No learning involved

Toy Example • 2 D uniform distribution

2. Boosting • Modified form of Boost. SSC [Shaknarovich, Viola & Darrell, 2003] • Positive examples are pairs of similar images • Negative examples are pairs of unrelated images 0 0 1 1 0 1 Learn threshold & dimension for each bit (weak classifier)

Toy Example • 2 D uniform distribution

3. Restricted Boltzmann Machine (RBM) • Type of Deep Belief Network • Hinton & Salakhutdinov, Science 2006 Hidden units Single RBM layer Symmetric weights W Units are binary & stochastic Visible units • Attempts to reconstruct input at visible layer from activation of hidden layer

Multi-Layer RBM: non-linear dimensionality reduction Output binary code (N dimensional) Layer 3 N w 3 256 Layer 2 256 w 2 512 Layer 1 512 w 1 512 Input Gist vector (512 dimensions) Linear units at first layer

Toy Example • 2 D uniform distribution

2 -D Toy example: 3 bits 7 bits Query Point 15 bits Distance from query point Red – 0 bits Green – 1 bit Black Blue – 2 bits – >2 bits

Toy Results Distance Red – 0 bits Green – 1 bit Blue – 2 bits

Semantic Hashing [Salakhutdinov & Hinton, 2007] Query Image Semantic Hash Function Quite different to a (conventional) randomizing hash Binary code Semantically similar images Address Space Images in database Query address

Spectral Hash Query Image Spectral Hash Non-linear dimensionality reduction Binary code Address Space Images in database Real-valued vectors Quite different to a (conventional) randomizing hash Semantically similar images Query address

Spectral Hashing (NIPS ’ 08) • Assume points are embedded in Euclidean space • How to binarize so Hamming distance approximates Euclidean distance? Ham_Dist(10001010, 1110)=3

Spectral Hashing theory • Want to min YT(D-W)Y subject to: – Each bit on 50% of time – Bits are independent – Sadly, this is NP-complete • Relax the problem, by letting Y be continuous. • Now becomes eigenvector problem

Nystrom Approximation • Method for approximating eigenfunctions • Interpolate between existing data points • Requires evaluation of distance to existing data cost grows linearly with #points • Also overfits badly in practice

What about a novel data point? Need a function to map new points into the space Take limit of Eigenvalues as n inf Need to carefully normalize graph Laplacian Analytical form of Eigenfunctions exists for certain distributions (uniform, Gaussian) • Constant time compute/evaluate new point • For uniform: • • Only depends on extent of distribution (b-a)

Eigenfunctions for uniform distribution

The Algorithm Input: Data {xi} of dimensionality d; desired # bits, k

1. Fit Multidimensional Rectangle • Run PCA to align axes • Bound uniform distribution

2. Calculuate Eigenfunctions

3. Pick k smallest Eigenfunctions Eigenvalues e. g. k=3

4. Threshold chosen Eigenfunctions

Back to the 2 -D Toy example 3 bits 7 bits Distance Red – 0 bits Green – 1 bit Blue – 2 bits 15 bits

2 -D Toy Example Comparison

10 -D Toy Example

Experiments on Real Data

Input Image representation: Gist vectors • • Pixels not a convenient representation Use Gist descriptor instead (Oliva & Torralba, 2001) 512 dimensions/image (real-valued 16, 384 bits) L 2 distance btw. Gist vectors not bad substitute for human perceptual distance NO COLOR INFORMATION Oliva & Torralba, IJCV 2001

Label. Me images • 22, 000 images (20, 000 train | 2, 000 test) • Ground truth segmentations for all • Assume L 2 Gist distance is true distance

Label. Me data

Extensions

How to handle non-uniform distributions

Bit allocation between dimensions • Compare value of cuts in original space, i. e. before the pointwise nonlinearity.

Summary • Spectral Hashing – Simple way of computing good binary codes – Forced to make big assumption about data distribution – Use point-wise non-linearities to map distribution to uniform – Need more experiments on real data

Overview • Assume points are embedded in Euclidean space (e. g. output from RBM) • How to binarize the space so that Hamming distance between points approximates L 2 distance?

Semantic Hashing beyond 30 bits

Strategies for Binarization 2. Deliberately add noise during backprop - forces extreme values to overcome noise 0 1