On Natural Scenes Analysis Sparsity and Coding Efficiency
- Slides: 59
On Natural Scenes Analysis, Sparsity and Coding Efficiency Vivienne Ming Mind, Brain & Computation Stanford University Redwood Center for Theoretical Neuroscience University of California, Berkeley Adapted by J. Mc. Clelland for PDP class, March 1, 2013
Two Proposals n Natural Scene Analysis ¨ Neural/cognitive computation can only be fully understood in “naturalistic” contexts n Efficient (Sparse) Coding Theory ¨ Neural computation should follow information theoretic principles Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Classical Physiology Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Classical Physiology + Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Classical Physiology + Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Reverse Correlation Jones and Palmer (1987) Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Limits of Classical Physiology n Assumes units (neurons) are linear ¨ so known nonlinearities are "added on" to the models n n n Contrast sensitivity “Non-classical receptive fields” Two-tone inhibition ETC. Assumes that units operate independently ¨ ¨ activity of one cell doesn't depend on the activity of others i. e. , characterizing cell-by-cell equivalent to characterizing the whole population ¨ of evolution and development, drifting gratings and white noise are very "unnatural“ Is it possible that our sensory systems are functionally adapted to the statistics of “natural” (evolutionarily relevant) signals? ¨ Would this adaptation affect our characterization of cells? ¨ Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Response to Natural Movie Classical Receptive Field Response in “Context” Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Limits of Classical Physiology n Assumes units (neurons) are linear ¨ so known nonlinearities are "added on" to the models n n n Assumes that units operate independently ¨ ¨ n Contrast sensitivity “Non-classical receptive fields” Two-tone inhibition ETC. activity of one cell doesn't depend on the activity of others i. e. , characterizing cell-by-cell equivalent to characterizing the whole population Finally, in terms of evolution and development, drifting gratings and white noise seem very "unnatural“ Is it possible that our sensory systems are functionally adapted to the statistics of “natural” (evolutionarily relevant) signals? ¨ Would this adaptation affect our characterization of cells? ¨ How can we test this? ¨ Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Efficient Coding Theory Barlow (1961); Attneave (1954) n Natural images are redundant ¨ Statistical dependencies amongst pixel values in space and time n An efficient visual system should reduce redundancy ¨ Removing Vivienne Ming, Ph. D. statistical dependencies Natural Scenes Analysis Rev jlm 3/5/2010
Information Theory Shannon (1949) Optimally efficient codes reflect the statistics of target signals Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Naïve Models Natural Scenes Analysis: First-Order Statistics Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Natural Scenes Analysis: First-Order Statistics Intensity Histogram Vivienne Ming, Ph. D. Natural Scenes Analysis Equalization Rev jlm 3/5/2010
Natural Scenes Analysis: Second-Order Statistics Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Natural Scenes Analysis: Second-Order Statistics Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Natural Scenes Analysis: Second-Order Statistics Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Spatial Correlations Compare intensity at this pixel To the intensity at this neighbor Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Spatial Correlations Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
The Ubiquitous. Flat (White) Power Spectrum Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Example: synthetic 1/f signals Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Natural Scenes Analysis: Principal Components Analysis PCA Rotation Whitening Information theory says this is an ideal code. No redundancy Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
PCA vs. Center Surround Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Natural Scenes Analysis: Higher-Order Statistics PCA Rotation Whitening Principle dimensions of variation don’t align with data’s intrinsic structure Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Natural Scenes Analysis: Higher-Order Statistics Need a more powerful learning algorithm Independent Component Analysis (ICA) Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Which are the independent components in the scene below? Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
= + +_______ Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
The Model n n n x = s + n Overcomplete: #(s) >> #(x) Factorial: p(s) = i p(si) Sparse: p(si) = exp(g(si)) ¨ Where g(. ) is some non-Gaussian distribution n n Information Theory demands sparseness e. g. , Laplacian: g(s) = −|s| e. g. , Cauchy: g(s) = −log(2 + s 2) The noise is assumed to be additive Gaussian ¨ n ~ N(0, 2 I) Goal: find dictionary of functions, , such that coefficients, s, are as sparse and statistically independent as possible Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Learning n log likelihood L( ) = <log p(x| )> Learning rule: n Basically the delta rule: n D = (x − s)s. T ¨ n Impose constraint to encourage the variances of each s to be approximately equal to prevent trivial solutions Usually whiten the inputs before learning Forces network to find structure beyond second-order ¨ Increases stability ¨ Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Sparsity Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
? Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Efficient Auditory Coding Smith & Lewicki (2006) n Extend Olshausen (2002) to deal with time -varying signals ¨ e. g. , n sounds or movies Train the network on “Natural” sounds ¨ Environmental Transients ¨ Environmental Ambients ¨ Animal Vocalizations Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Cat ANF Revcor Filters Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Efficient Kernels Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Population Coding Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Population Coding Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Population Coding Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Population Coding Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Population Coding Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Speech Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Speech Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Speech Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Efficient Coding Literature n Empirical ¨ ¨ ¨ n Theoretical Weliky, Fiser, Hunt & Wagner (2003) Vinje & Gallant (2002) De. Weese, Wehr & Zador (2003) Laurent (2002) Theunissen (2003) ¨ ¨ ¨ ¨ Vivienne Ming, Ph. D. Natural Scenes Analysis Field (1987) van Hateren (1992) Simoncelli & Olshausen (2001) Olshausen & Field (1996) Bell & Sejnowski (1997) Hyvarinen & Hoyer (2000) Smith & Lewicki (2006) Doi & Lewicki (2006) Rev jlm 3/5/2010
Hierarchical Structure? n Can we identify interesting structure in the world by looking at higher order statistics of the activations of the linear features discovered by the first-order model? ¨ Karklin and Lewicki (2005) looked for patterns at the level of the variances of the linear features. ¨ Karklin and Lewicki (2009) looked for patterns at the level of the covariances of the linear features. Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Looking at Hierarchical Structure Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Looking at Hierarchical Structure Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Looking at Hierarchical Structure Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Looking at Hierarchical Structure Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Looking at Hierarchical Structure Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Generalizing the standard ICA model Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Generalizing the standard ICA model Instead of: we now have units u and v such that Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Independent density components Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Karklin & Lewicki (2009) The model tries to find the values of the yj’s that lead to a combined covariance matrix C that matches the covariance of the data represented by activities across first-level filters. The learning process involves a search for vectors bk and weights wjk that allow the model to fit the data while keeping the yj’s sparse and independent. Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Responses of Cell to Gratings Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Vivienne Ming, Ph. D. Natural Scenes Analysis Rev jlm 3/5/2010
Efficient Coding Summary Algorithm Example Biological Example Statistic Computation 1 st-order Contrast gain Histogram Retina or H 1 control equalization adaptation Reference Fairhall et al. (2001) 2 nd-order Whitening PCA Retinal/ Thalamic coding Higher-order Sparse Coding ICA / Sparsenet V 1 coding Olshausen & Field (1996) Time-varying Shiftinvariance Efficient Spike Coding Cochlear coding Smith & Lewicki 2006 Hierarchical Conditional Independence Hierarchical coding Natural Scenes Analysis ? Karklin & Lewicki Rev ’ 05, ’ 09 jlm 3/5/2010 Vivienne Ming, Ph. D. Atick (1992)