Radial Basis Function NetworksRBF Radial Basis Function Networks

Radial Basis Function Networks(RBF) Radial Basis Function Networks Artificial Neural Network- Instructor : Dr. M. Rezaeian 1

Radial Basis Functions (RBFs) DEFINITION: • A function is radially symmetric (or is an RBF) if its output depends on the distance of the input sample (vector) from another stored vector (e. g. Gaussian functions). • Neural networks whose node functions are radially symmetric functions are referred to as RBFnets. • The training of RBF networks is substantially faster than the methods used to train multi-layer perceptron networks A two-class problem with a single cluster, the backpropagation method may implement lines A, B, C, D using four hidden nodes. C D Radial Basis Function Networks B Artificial Neural Network- Instructor : Dr. M. Rezaeian • Radial basis functions are generally used for function approximation problems particularly for interpolation.

Linear Interpolation: 1 -Dimensional Case For function approximation, the desired output for new (untrained) inputs could be estimated by linear interpolation. 3 As a simple example, how do we determine the desired output of a one-dimensional function at a new input x 0 that is located between known data points x 1 and x 2? with distances D 1 and D 2 from x 0 to x 1 and x 2, resp. Radial Basis Function Networks Artificial Neural Network- Instructor : Dr. M. Rezaeian which simplifies to:

Linear Interpolation: Multiple Dimensions Where Dp is the Euclidean distance between x 0 and xp and f (xp) is the desired output value for input xp. Radial Basis Function Networks 4 Artificial Neural Network- Instructor : Dr. M. Rezaeian In the multi-dimensional case, hyperplane segments connect neighboring points so that the desired output for a new input x 0 is determined by the P 0 known samples that surround it:

Linear Interpolation: Multiple Dimensions Radial Basis Function Networks Artificial Neural Network- Instructor : Dr. M. Rezaeian 5

Radial Basis Functions (RBFs) Each commonly used radial basis function is a non-increasing function of a distance measure u which is its only argument, with (u 1) (u 2) whenever u 1< u 2 6 Function is applied to the Euclidean distance: between the "center" or stored vector and the input vector i. for some square matrix A suggested by Poggio and Girosi (1990). Generalized distance norms are useful, because all coordinates of a vector input may not be equally important. But the main difficulty with this measure is in determining an appropriate A matrix. Radial Basis Function Networks 6 Artificial Neural Network- Instructor : Dr. M. Rezaeian Vector norms other than the Euclidean distance may also be used. e. g. , the generalized distance norm:

Exact Interpolation Radial Basis Function Networks Neural networks for pattern recognition – Bishop 1995, Slides from: Dr. J A Bullinaria Artificial Neural Network- Instructor : Dr. M. Rezaeian 7

Determining the Weights Radial Basis Function Networks Neural networks for pattern recognition – Bishop 1995, Slides from: Dr. J A Bullinaria Artificial Neural Network- Instructor : Dr. M. Rezaeian 8

Radial Basis Functions (RBFs) A range of theoretical and empirical studies have indicated that many properties of the interpolating function are relatively insensitive to the precise form of the basis functions (r) 9 where is a parameter whose value controls the smoothness properties of the interpolating function. The Gaussian is a localized basis function with the property that: 0 as |r| It is not, however, necessary for the functions to be localized. Radial Basis Function Networks Neural networks for pattern recognition – Bishop 1995, Slides from: Dr. J A Bullinaria Artificial Neural Network- Instructor : Dr. M. Rezaeian Several forms of basis function have been considered, the most common being the Gaussian:

Radial Basis Functions Network Radial Basis Function Networks Neural networks for pattern recognition – Bishop 1995, Slides from: Dr. J A Bullinaria Artificial Neural Network- Instructor : Dr. M. Rezaeian 10

Exact Interpolation Radial Basis Function Networks Neural networks for pattern recognition – Bishop 1995 Artificial Neural Network- Instructor : Dr. M. Rezaeian 11

Problems with Exact Interpolation Radial Basis Function Networks Neural networks for pattern recognition – Bishop 1995, Slides from: Dr. J A Bullinaria Artificial Neural Network- Instructor : Dr. M. Rezaeian 12

Improving RBF networks Radial Basis Function Networks Neural networks for pattern recognition – Bishop 1995, Slides from: Dr. J A Bullinaria Artificial Neural Network- Instructor : Dr. M. Rezaeian 13

The Improved RBF Network Radial Basis Function Networks Neural networks for pattern recognition – Bishop 1995, Slides from: Dr. J A Bullinaria Artificial Neural Network- Instructor : Dr. M. Rezaeian 14

Radial Basis Functions (RBFs) Radial Basis Function Networks Neural networks for pattern recognition – Bishop 1995, Slides from: Dr. J A Bullinaria Artificial Neural Network- Instructor : Dr. M. Rezaeian 15

Computational Power of RBF Networks Radial Basis Function Networks Neural networks for pattern recognition – Bishop 1995, Slides from: Dr. J A Bullinaria Artificial Neural Network- Instructor : Dr. M. Rezaeian 16

Training RBF Networks Radial Basis Function Networks Neural networks for pattern recognition – Bishop 1995, Slides from: Dr. J A Bullinaria Artificial Neural Network- Instructor : Dr. M. Rezaeian 17

Fixed Centers Selected At Random Radial Basis Function Networks More reading: Haykin 1999, p. p. 299 -305, Slides from: Dr. J A Bullinaria Artificial Neural Network- Instructor : Dr. M. Rezaeian 18

K-Means Clustering Radial Basis Function Networks More reading: Haykin 1999, p. p. 299 -305, Slides from: Dr. J A Bullinaria Artificial Neural Network- Instructor : Dr. M. Rezaeian 19

Training the output layer Radial Basis Function Networks Neural networks for pattern recognition – Bishop 1995, Slides from: Dr. J A Bullinaria Artificial Neural Network- Instructor : Dr. M. Rezaeian 20

Training the output layer Radial Basis Function Networks Slides from: Dr. J A Bullinaria Artificial Neural Network- Instructor : Dr. M. Rezaeian 21

Supervised RBF Network Training Radial Basis Function Networks Slides from: Dr. J A Bullinaria Artificial Neural Network- Instructor : Dr. M. Rezaeian 22

Example: Radial Basis Function Networks Neural networks for pattern recognition – Bishop 1995 Artificial Neural Network- Instructor : Dr. M. Rezaeian 23

Example: Radial Basis Function Networks Neural networks for pattern recognition – Bishop 1995 Artificial Neural Network- Instructor : Dr. M. Rezaeian 24

Example: Radial Basis Function Networks Neural networks for pattern recognition – Bishop 1995 Artificial Neural Network- Instructor : Dr. M. Rezaeian 25

RBF Networks for Classification Radial Basis Function Networks Slides from: Dr. J A Bullinaria Artificial Neural Network- Instructor : Dr. M. Rezaeian 26

Implementing RBF Classification Networks Radial Basis Function Networks Slides from: Dr. J A Bullinaria Artificial Neural Network- Instructor : Dr. M. Rezaeian 27

Example: The XOR Problem Radial Basis Function Networks Slides from: Dr. J A Bullinaria Artificial Neural Network- Instructor : Dr. M. Rezaeian 28

The XOR Problem in RBF Form Radial Basis Function Networks Slides from: Dr. J A Bullinaria Artificial Neural Network- Instructor : Dr. M. Rezaeian 29

The XOR Problem in RBF Form Radial Basis Function Networks Slides from: Dr. J A Bullinaria Artificial Neural Network- Instructor : Dr. M. Rezaeian 30

The XOR Problem in RBF Form Radial Basis Function Networks Slides from: Dr. J A Bullinaria Artificial Neural Network- Instructor : Dr. M. Rezaeian 31

Radial Basis Function Networks Artificial Neural Network- Instructor : Dr. M. Rezaeian RBF Networks 32

Radial Basis Functions (RBFs) Comparison of RBF Networks with MLPs Radial Basis Function Networks Artificial Neural Network- Instructor : Dr. M. Rezaeian 33

Comparison of RBF Networks with MLPs Radial Basis Function Networks Artificial Neural Network- Instructor : Dr. M. Rezaeian 34