Ghent University Pattern recognition with CNNs as reservoirs

Ghent University Pattern recognition with CNNs as reservoirs David Verstraeten 1 – Samuel Xavier de Souza 2 – Benjamin Schrauwen 1 Johan Suykens 2 – Dirk Stroobandt 1 – Joos Vandewalle 2 1 ELIS, Ghent University 2 ESAT , KU Leuven Belgium NOLTA 2008 – September 8, 2008

Outline • Introduction: • Reservoir what is Reservoir Computing? Computing with CNNs • Simulation results using grid-search optimization • On-chip results for optimization using CSA • Conclusions and future challenges Pattern recognition with CNNs as reservoirs – David Verstraeten Ghent University - Faculty of Engineering – Electronics and Information Systems Dept. 2/13

Reservoir Computing • Novel method for training recurrent neural networks • Internal weights and input weights are randomly drawn (gaussian), globally scaled and left untrained • Linear output layer is trained using e. g. ridge regression Input layer Reservoir Output layer Pattern recognition with CNNs as reservoirs – David Verstraeten Ghent University - Faculty of Engineering – Electronics and Information Systems Dept. 3/13

Reservoir Computing • Functionality is similar to a kernel, but works for dynamic signals • Reservoir offers rich pool of nonlinear dynamic transformations of the input to the linear output layer • Applied to: speech recognition, time series prediction, robot control, . . . Input layer Reservoir Output layer Pattern recognition with CNNs as reservoirs – David Verstraeten Ghent University - Faculty of Engineering – Electronics and Information Systems Dept. 4/13

Reservoir Computing with CNNs • ACE 16 k: very fast hybrid a. VLSI implementation. • Limitations compared to traditional reservoirs: • • 2 D topology • Uniform internal weight template • Dynamic node behavior For computational reasons, we only use the central 8 x 8 cells Pattern recognition with CNNs as reservoirs – David Verstraeten Ghent University - Faculty of Engineering – Electronics and Information Systems Dept. 5/13

Grid search template optimization • Search space for traditional reservoirs is huge: N^2 -dimensional for N neurons • Search space for CNN templates is smaller, especially when we restrict it Diagonal Lateral Self Pattern recognition with CNNs as reservoirs – David Verstraeten Ghent University - Faculty of Engineering – Electronics and Information Systems Dept. 6/13

Experiment 1: simple signal classification • Academic task, rather easy: proof of concept signal switches randomly between sawtooth and square wave Input • Input Output • Main difficulty lies at transitions between signals Pattern recognition with CNNs as reservoirs – David Verstraeten Ghent University - Faculty of Engineering – Electronics and Information Systems Dept. 7/13

Experiment 1: simple signal classification Pattern recognition with CNNs as reservoirs – David Verstraeten Ghent University - Faculty of Engineering – Electronics and Information Systems Dept. 8/13

Experiment 2: spoken digit classification • Isolated • Subset • Two spoken digits, ‘zero’ to ‘nine’ of TI 46 different female speakers • Every digit uttered 10 times 200 samples • Preprocessing using biological model of cochlea. Readout + postprocessing Pattern recognition with CNNs as reservoirs – David Verstraeten Ghent University - Faculty of Engineering – Electronics and Information Systems Dept. 9/13

Experiment 2: spoken digit classification Pattern recognition with CNNs as reservoirs – David Verstraeten Ghent University - Faculty of Engineering – Electronics and Information Systems Dept. 10/13

Template optimization using CSA • CSA: Coupled Simulated Annealing • Extension of traditional simulated annealing • Better convergence and less sensitivity to initial conditions Pattern recognition with CNNs as reservoirs – David Verstraeten Ghent University - Faculty of Engineering – Electronics and Information Systems Dept. 11/13

Template optimization using CSA : results Signal classification Digit recognition No reservoir N/A 11% Classic RC 1% 2% Simulated CNN RC 1% 3. 6% On-chip CNN RC 0. 1% 6% Pattern recognition with CNNs as reservoirs – David Verstraeten Ghent University - Faculty of Engineering – Electronics and Information Systems Dept. 12/13

Conclusions and future challenges • Novel use of CNNs for computation: Reservoir Computing • Proof-of-concept • Both : academic and real world task in simulation and on-chip • Future challenges: • On-chip • How • Can readout? to get it trained (large dataset)? we use the dynamic behavior of the nodes? Pattern recognition with CNNs as reservoirs – David Verstraeten Ghent University - Faculty of Engineering – Electronics and Information Systems Dept. 13/13