Introduction to BioInspired Models During the last three

Introduction to Bio-Inspired Models During the last three decades, several efficient machine learning tools have been inspired in biology and nature: üArtificial Neural Networks (ANN) are inspired in the brain to automatically learn and generalize (model) observed data. üEvolutive and genetic algorithms offer a solution to standard optimization problems when no much information about the function to optimize is available. üArtificial ant colonies offer an alternative solution for optimization problems. All these methods share some common properties: üThey are inspired in nature (not in human logical reasoning). üThey are automatic (no human intervention) and nonlinear. üThey provide efficient solutions to some hard NP-problems. 1

Introduction to Artificial Neural Networks are inspired in the structure and functioning of the brain, which is a collection of interconnected neurons (the simplest computing elements performing information processing): üEach neuron consists of a cell body, that contains a cell nucleus. üThere are number of fibers, called dendrites, and a single long fiber called axon branching out from the cell body. üThe axon connects one neuron to others (through the dendrites). üThe connecting junction is called synapse. 2

Functioning of a “Neuron” • • The synapses releases chemical transmitter substances. The chemical substances enter the dendrite, raising or lowering the electrical potential of the cell body. When the potential reaches a threshold, an electric pulse or action potential is sent down to the axon affecting other neurons. (Therefore, there is a nonlinear activation). Excitatory and inhibitory synapses. weights (+ or -, excitatory or inhibitory) neuron potential: mixed input of neighboring neurons nonlinear activation function (threshold) 3

Multilayer perceptron. Backpropagation algorithm Inputs Outputs Gradient descent 1. Init the neural weight with random values 2. Select the input and output data and train it 3. Compute the error associate with the output The neural activity (output) is given by a no linear function. 4. Compute the error associate with the hidden neurons 5. Compute and update the neural weight according to these values 4

Time Series Modeling and Forecast Sometimes the chaotic time series have a stochastic look difficult to predict An example is Henon map 5

Example: Supervised Fitting and Prediction Given a time series with 2000 points (T=20), generated from a Lorenz system (chaotic behavior). To check modeling power different parameters are tested. Three variables (x, y, z) (xn, yn, zn) 3: 6: 3 x n+1 y 1 Continuous System Neural Network 3: k: 3 zn+1 yi 3: k: 3 zi Wik h 1 (xn+1, yn+1, zn+1) yn+1 h 2 hk 3: 15: 3 wkj x 2 xn x 3 yn xj zn 6

Dynamical Behavior A simple model doesn’t capture the complete structure of the system , then the dynamics of the system is not reproduce. A complex system it’s overfitting the problem and the dynamics of the system is not reproduce 3: 6: 3 3: 15: 3 Only a intermediate model with an appropriate amount of parameters can model the functional structure of the system and the dynamics 7

Time series from a infrared laser. Xi Infrared laser intensity is modeled using a neural network. Only time lagged intensities are used. Net 6: 5: 5: 1 The Neural network reproduces laser behavior Xi-1 Xi-2 Xi-3 Xi-j The Neural Network can be synchronized with the time series obtained from the laser. 8

Structural Learning: Modular Neural Networks With the aim of giving some flexibility to the network topology, modular neural networks combine different neural blocks into a global topology. Fully-connected topology (too many parameters). Combining several blocks (parameter reduction). 2*4+4*1+9= 37 weights Assigning different subnets to specific tasks we can simplify the complexity of the model. 2(2*2)+4*1+9= 29 weights In most of the cases, block division is a heuristic task !!! How to obtain an optimal “block division” for a given problem ? 9

Functional Networks Functional networks are a generalization of neural networks which combine both qualitative domain knowledge and data. I x Qualitative knowledge: x 3 = F (x 1, x 2), F F y u z I Initial Topology Theorem. The simplest functional form is: Simplified Topology x f y f z f + f-1 u This is the optimal “block division” for this problem !!! Data: (x 1 i , x 2 i , x 3 i ), i=1, 2, . . . Learning (least squares): {f 1, . . . , fn} {a 1, . . . , an} 10

Some FN Architectures Associative Model: F(x, y) is an associative operator. Separable Model: A simple topology. Sliced-Conditioned Model: where f and y are covenient basis for the x- and y-constant slices. 11

A First Example. Functional vs Neural 100 points of Training Data with Uniform Noise in (-0. 01, 0. 01). 25 x 25 points from the exact surface for Validation. Neural Network 2: 2: 2: 1 MLP 15 parameters RMSE=0. 0074 2: 3: 3: 1 MLP 25 parameters RMSE=0. 0031 Functional Network (separable model) F = {1, x 2 , x 3} 12 parameters RMSE=0. 0024 Knowledge of the network structure (separable). Appropriate family of functions (polynomial). Non-parametric approach to learn the neuron functions !!!! 12

Functional Nets & Modular Neural Nets Advantages and shortcomings of Neural Nets Black-box topology with no problem connection. Efficient non-parametric models for approximating functions. Functional Nets Parametric learning techniques (supply basis functions). Model driven optimal topology. The topology of the network is obtained from the Functional network. The neuron functions are Approximated using MLPs. Hybrid functional-neural networks (Modular networks) 13

Another example. Nonlinear Time Series Time series modeling and forecasting is an important problem with many practical applications. Goal: predicting the future using past values. x 1, x 2, …, xn ¿¿¿ Nonlinear time series modeling is a difficult task because: Sensitivity to initial conditions Trajectories starting at very close initial points split away after a few iterates. xn+1 ? ? ? X 1=0. 8 + 10 -3 Modeling methods: xn+1 = X 1=0. 8 - 10 -3 F(x 1, x 2, …, xn) There are many well-known techniques for linear time series (ARMA, etc. ). Fractal geometry Evolve in a irregular fractal space. Nonlinear time series may exhibit complex seemingly stochastic behavior. Nonlinear Maps (the Lozi model) 14

Functional Models (separation) 500 training points 1000 validation points. Separable Functional Net: ¿which basis family? F={sin(x), …, sin(mx), cos(x), …, cos(mx)} With 4*m parameters Symmetric Modular Functional Neural Net: 1: m: 1 With 6*m parameters Asymmetric Modular Functional Neural Net: 1: 2 m: 1 1: 2: 1 With 2*m-2 parameters m=11 (44 pars) RMSE=5. 3 e-3 FN m=7 (42 pars) RMSE=1. 5 e-3 MFNN 1 m=7 (42 pars) RMSE=4. 0 e-4 MFNN 2 15

Minimum Description Length for a model The Minimum Description Length (MDL) algorithm has proved to be simple and efficient in several problems about Model Selection. 16