Convolutional Deep Belief Networks for Scalable Unsupervised Learning

Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y. Ng ICML 2009 Presented by: Mingyuan Zhou Duke University, ECE September 18, 2009

Outline • • Motivations Contributions Backgrounds Algorithms Experiment results Deep Vs Shallow Conclusions

Motivations • To Learn hierarchical models which simultaneously represent multiple levels, e. g. , pixel intensities, edges, object parts, objects, and beyond can be represented by layers from low to high. • Combining top-down and bottom-up processing of an image. • Limitations of deep belief networks (DBNs) • Scaling DBNs to realistic-size images remains challenging: images are high-dimentional and objects can appear at arbitrary locations in images.

Contributions • Convolutional RBM: feature detectors are shared among all locations in an image. • Probabilistic max-pooling: in a probabilistic sound way allowing higher-layer units to cover larger areas of the input. • The first translation invariant hierarchical generative model supporting both top-down and bottom-up probabilistic inference and sales to realistic image sizes.

Backgrounds: Restricted Boltzmann Machine (RBM) (binary v) (real-value v) • Giving the visible layer, the hidden units are conditionally independent, and vise versa. • Efficient block Gibbs sampling can be performed by alternately sampling each layer’s units. • Computing the exact gradient of the log-likelihood is intractable, so the contrastive divergence approximation is commonly used.

Backgrounds: Deep belief network (DBN) • In a DBN, two adjacent layers have a full set of connections between them, but no two units in the same layer are connected. • A DBN can be formed by stacking RBMs. • An efficient algorithm for training DBNs (Hinton et al. , 2006): greedily training each layer, from lowest to highest, as an RBM using the previous layer's activations as inputs.

Algorithms: Convolutional RBM (CRBM)

Algorithms: Probabilistic max-pooling

Algorithms: Probabilistic max-pooling • Each unit in a pooling layer computes the maximum activation of the units in a small region of the detection layer. • Shrinking the representation with max-pooling allows higher-layer representations to be invariant to small translations of the input and reduces the computational burden. • Max-pooling was intended only for feed-forward architectures. A generative model of images which supports both top-down and bottom-up inference is of interest.

Algorithms: Sparsity regulations • Only a tiny fraction of the units should be active in relation to a given stimulus. • Regularizing the objective function to encourage each of the hidden units to have a mean activation close to some small constant.

Algorithms: Convolutional DBN (CDBN) • CDBN consists of several max-pooling-CRBMs stacked on top of one another. • Once a given layer is trained, its weights are frozen, and its activations are used as input to the next layer.

Hierarchical probabilistic inference

Experimental Results: natural images

Experimental Results: image classification

Experimental Results: unsupervised learning of object parts

Experimental Results: Hierarchical probabilistic inference

Deep Vs Shallow . From Jason Weston’s slides: DEEP LEARNING VIA SEMI-SUPERVISED EMBEDDING, ICML 2009 WORKSHOP ON LEARNING FEATURE HIERARCHIES From Francis Bach’s slides: Convex sparse methods for feature hierarchies, ICML 2009 WORKSHOP ON LEARNING FEATURE HIERARCHIES

Conclusions Convolutional deep belief network: • A scalable generative model for learning hierarchical representations from unlabeled images. • Performing well in a variety of visual recognition tasks.