Stochastic Pooling Networks On the interaction between redundancy

Stochastic Pooling Networks: On the interaction between redundancy, noise and lossy compression in biological neurons. Mark D. Mc. Donnell University of South Australia Pierre-Olivier Amblard CNRS, Grenoble, France Nigel G. Stocks University of Warwick, UK 6 July 2008

This talk is about understanding how biological neurons efficiently compress stimuli during coding. What is a Stochastic Pooling Network? What do SPNs Model? Emergent properties of SPNs

Real signals are analog but nearly everything in modern electronics is digital. Why is this? X 1 x R 1 1 2 x 2 R 2 x N RN N CEO X

Information theory underpins fast and accurate communication. But it is also about selecting what information is worth storing and communicating. What mechanisms do biological sensory systems use to compress information during transduction? There is lossy, and lossless compression… and maybe something in between: “loss-least!”

A “Recipe” for Stochastic Pooling Networks Ingredients • 1 information source • N > 1 independent sensors (compressive types) • N > 1 random noise sources Preparation Step 1: For each sensor, measure the information source mixed with one of the noise sources, and then compresses its measurement. Step 2: Ensure the whole network produces a single measurement by pooling each sensor’s.

SPNs can model many sensor network or source coding scenarios • Digital Beamforming Arrays – Sonar, radar, MIMO communications • Digital signal processing – Noise reduction via coherent averaging after digitization in ADCs. • Distributed/Decentralized sensor networks – CEO problem – Multiaccess Communication Channels – Power constrained wireless sensor networks • Biological neurons – Representation of analog stimuli by rate coding action potentials – Quantal release of chemical neurotransmitters at synaptic junctions – Maybe some ion channels?

SPNs can model many sensor network or source coding scenarios • Reliability schemes in nano-electronics – Averaging and redundancy to overcome parameter variations and noise [Ferran Martorell, Spain] • Social networks – Subjective voting on a continuous variable • Quantum optical communication using polarization detection of single photons? • Coupled multistable dynamic systems? • Reconfigurable chaotic logic gates?

The neurons that code sounds immediately after transduction can be modelled as an SPN Ear Sound Cochlea Information Inner Hair Cell Auditory Nerve

The neurons that code sounds immediately after transduction can be modelled as an SPN Outer hair cell Inner hair cell Brain Basilar membrane Slide courtesy of Prof Tony Burkett, Uni. Melbourne

We assume combining of the ingredients is left to physics: POOLING 1 x g 1 (. ) g 2 (. ) y 1 {0, 1, . . , M} y 2 {0, 1, . . , M} 2 P y = h(y 1, . . , y. N) |y| << M N g. N(. ) y. N {0, 1, . . , M} Pooling must occur “naturally, ” without external intervention, e. g. adding, or superposition. We will not have a pooling network otherwise!

Pooling loses no (or negligible) information! Assume the information source, x, is random. The mutual information, I(x, y) loosely measures how well, on average, the SPN output, y, provides a good estimate for x. This is a surprising emergent property of SPNs

SPNs digitize (quantize) their input Mc. Donnell, Stocks et. al. , Fluct. Noise Lett. 5, L 457 -L 468, 2005. Mc. Donnell, “Applying stochastic signal quantization theory to the robust digitization of noisy analog signals", Book chapter in Springer Verlag Complexity Series, In Press, 2008.

SPNs digitize (quantize) their input Mc. Donnell an Abbott, Proc. SPIE, 2006.

SPNs reduce noise via coherent averaging… but not in a linear way! 0. 5 log 2(1+NSNR) N=1, M=511 N=511, M=1 • For small noise, performance is limited by compression: I(x, y) < log(1+MN) • For large noise, performance is limited by “averaging”: I(x, y) < 0. 5 log(1+N SNR)

There a number of differences in the final SNR achieved by linear analog averaging vs SPNs Output SNR Analog SPN Dependency on source distribution no yes Decoding dependency N/A yes Scaling with input SNR linear Scaling with N (no. of times averaged) Scaling with M (bits) -linear for small SNRs, -nonlinear for large SNRs proportional - Approx. proportional for small SNRs -no effect for large SNRs N/A -increases for small SNRs -no effect for large SNRs Mc. Donnell, “Signal Estimation Via Averaging of Coarsely Quantised Signals, ” Proc IEEE Information, Decision and Control, Adelaide, Australia, pp 100 -105, 2005.

Suprathreshold Stochastic Resonance in SPNs SSR measured by mutual information for additive random noise. [Stocks, Phys. Rev. Lett. , 2000] Probability of Error: binary detection. [Zozor, Amblard and Duchene, Fluct. Noise Lett. 7, L 39 -L 60, 2007] Multiplicative Noise at the input to network nodes. [Nikitin, Stocks and Morse, Phys. Rev. E, 2007] Simulation of cochlear implant coding. [Stocks et. al. , Proc SPIE, 2007. ] *Also studies by others, e. g. Rousseau & Chapeau-Blondeau (2003), Hoch et al (2003), Martorell (2005)…

Optimizing the nodes: more noise means more nodes that are identical Mc. Donnell, Stocks et. al. , “Optimal information transmission in nonlinear arrays through suprathreshold stochastic resonance, ” Phys. Lett. A 352, pp. 183 -189, 2006. We have observed similar effects for optimized networks of Poisson neuron models, and binary detection networks.

There are many other surprising emergent properties • Very noisy SPNs behave like analogue Gaussian channels [Mc. Donnell, IEEE Aus. Comm Theory Wkshp, 2008] • Very large SPNs behave like multiplicative noise channels [Mc. Donnell and Stocks, Proc. SPIE. 2007] • Optimal reconstruction depends only on the noise distribution and the number of sensors [Mc. Donnell, Stocks and Abbott, Phys. Rev. E 75, Art. No. 061105. ] • Optimizing the noise distribution is like optimizing a neuron’s stimulus-response curve. • Negative correlation provides improved MI. • The optimal stimulus is actually discrete!

Unsolved problems on SPNs in biology • Do biological senses really use SPN principles? • What mechanisms does the brain use to compress/reduce information • Will the controlled use of random noise in cochlear implants improve the hearing of patients?

If healthy auditory neurons act like SPNs then bionic ears should stimulate them randomly! Electrode array 1 st turn of inner ear Auditory nerve fibres Image courtesy of Cochlear Ltd, 2008

If healthy auditory neurons act like SPNs then bionic ears should stimulate them randomly! “Stochastic beamforming coding strategy”, Morse, Holmes, Shulgin, Nikitin and Stocks, 2007 © Australasian Science, 2008

There are many unsolved theoretical problems on SPNs • Can the clustering of nodes be predicted mathematically? • What further complexities can be added to SPNs without changing the basic ingredients?

In summary, stochastic pooling networks are a versatile and surprising concept for achieving the twin goals of accuracy and efficiency. Redundancy allows noise reduction and simplicity Lossy compression is required for efficiency Random noise improves sub-optimal compression Questions?