Artificial Neural Networks Unsupervised ANNs Artificial Neural Networks

Artificial Neural Networks Unsupervised ANNs Artificial Neural Networks III

Contents • Unsupervised ANNs • Kohonen Self-Organising Map (SOM) – – – Structure Processing units Learning Applications Further Topics: Spiking ANNs Application • Adaptive Resonance Theory (ART) – – – 10/31/2020 Structure Processing units Learning Applications Further Topics: ARTMAP Inteligência Artificial 2

Unsupervised ANNs • Usually 2 -layer ANN • Only input data are given • ANN must self-organise output • Two main models: Kohonen’s SOM and Grossberg’s ART • Clustering applications 10/31/2020 Inteligência Artificial Output layer Feature layer 3

Learning Rules • Instar Learning rule: incoming weights of neuron converge to input pattern (previous layer) – Convergence speed is determined by learning rate – Step size proportional to node output value – Neuron learns association between input vectors and their outputs • Outstar Learning rule: outgoing weights of neuron converge to output pattern (next layer) – Learning is proportional to neuron activation – Step size proportional to node input value – Neuron learns to recall pattern when stimulated 10/31/2020 Inteligência Artificial 4

Self-Organising Map (SOM) • • T. Kohonen (1984) 2 D map of output neurons Input layer and output layer fully connected Lateral inhibitory synapses Model of biological topographic maps, e. g. primary auditory cortex in animal brains (cats and monkeys) Hebbian learning Akin to K-means Data clustering applications 10/31/2020 Inteligência Artificial Output layer Feature layer 5

SOM Clustering • Neuron = prototype for a cluster • Weights = reference vector (protoype features) • Euclidean distance between reference vector and input pattern • Competitive layer (winner take all) • In biological systems winner take all via inhibitory synapses • Neuron with reference vector closest to input wins 10/31/2020 Inteligência Artificial x 1 wi 1 x 2 wi 2 x 3 wi 4 x 4 wi 5 x 5 yi Neuron i 6

SOM Learning Algorithm • Only weights of winning neuron and its neighbours are updated • Weights of winning neuron brought closer to input pattern (instar rule) • Reference vector is usually normalised • Neighbourhood function in biological systems via short range excitatory synapses • Decreasing width of neighbourhood ensures increasingly finer differences are encoded • Global convergence is not guaranteed. • Gradual lowering of learning rate ensures stability (otherwise vectors may oscillate between clusters) • At end neurons are “tagged”, similar ones become sub-clusters of larger cluster N(t) = Neighbourhood function E(t 0) E(t 1) E(t 2) E(t 3) 10/31/2020 Inteligência Artificial 7

SOM Mapping • Adaptive Vector Quantisation • Reference vectors iteratively moved towards centres of (sub)clusters • Best performing on gaussian distributions (distance is radial) 10/31/2020 Inteligência Artificial 8

SOM Topology • Surface of map reflects frequency distribution of input set, i. e. the probability of input class occurring. • More common vector ``types’’ occupy proportionally more of output map. • The more frequent the pattern type, the finer grained the mapping. • Biological correspondence in brain cortex • Map allows dimension reduction and visualisation of input data 10/31/2020 Inteligência Artificial 9

Some Issues about SOM • SOM can be used on-line (adaptation) • Neurons need to be labelled – Manually – Automatic algorithm • • • Sometimes may not converge Precision not optimal Some neurons may be difficult to label Results sensitive to choice of input features Results sensitive to order of presentation of data – Epoch learning 10/31/2020 Inteligência Artificial 10

SOM Applications • Natural language processing – Document clustering – Document retrieval – Automatic query • Image segmentation • Data mining • Fuzzy partitioning • Condition-action association 10/31/2020 Inteligência Artificial 11

Further Topics – Spiking ANNs • Image segmentation task • SOM of spiking units • Lateral connections – Short range excitatory – Long range inhibitory • Train using Hebbian Learning • Train showing one pattern at a time 10/31/2020 Inteligência Artificial 12

Spiking SOM Training • Hebbian Learning • Different learning coefficients – afferent weights la – lateral inhibitory weights li – lateral excitatory weights le • Initially learn long-term correlations for self-organisation • Then learn activity modulation for segmentation N = normalisation factor la li, le t 10/31/2020 Inteligência Artificial 13

Spiking Neuron Dynamics y(t) urest+ (t-tf) 10/31/2020 Inteligência Artificial 14

Spiking SOM Recall • Show different shapes together • Bursts of neuron activity • Each cluster alternatively fires 10/31/2020 Inteligência Artificial 15

Adaptive Resonance Theory (ART) • Carpenter and Grossberg (1976) • Inspired by studies on biological feature detectors • On-line clustering algorithm • Leader-follower algorithm • Recurrent ANN • Competitive output layer • Data clustering applications • Stability-plasticity dilemma 10/31/2020 Inteligência Artificial Output layer Feature layer 16

ART Types • • 10/31/2020 ART 1 binary patterns ART 2 binary or analog patterns ART 3 hierarchical ART structure ARTMAP supervised ART Inteligência Artificial 17

Stability-Plasticity Dilemma • Plasticity: System adapts its behaviour according to significant events • Stability: system behaviour doesn’t change after irrelevant events • Dilemma: how to achieve stability without rigidity and plasticity without chaos? – Ongoing learning capability – Preservation of learned knowledge 10/31/2020 Inteligência Artificial 18

ART Architecture • Bottom-up weights wij – Normalised copy of vij • Top-down weights vij – Store class template • Input nodes – Vigilance test – Input normalisation • Output nodes – Forward matching • Long-term memory – ANN weights • Short-term memory – ANN activation pattern top down bottom up (normalised) 10/31/2020 Inteligência Artificial 19

ART Algorithm new pattern recognition comparison categorisation known Adapt winner node 10/31/2020 unknown Initialise uncommitted node Inteligência Artificial • Incoming pattern matched with stored cluster templates • If close enough to stored template joins best matching cluster, weights adapted according to outstar rule • If not, a new cluster is initialised with pattern as template 20

Recognition Phase • Forward transmission via bottom-up weights • Input pattern matched with bottom-up weights (normalised template) of output nodes • Inner product x • wi • Hypothesis formulation: best matching node fires (winner-take-all layer) • Similar to Kohonen’s SOM algorithm, pattern associated to closest matching template • ART 1: fraction of bits of template also in input pattern Innner product x=input pattern wi=bottom-up weight of neuron I N=input features x q wi 10/31/2020 Inteligência Artificial 21

Comparison Phase • Backward transmission via top-down weights • Vigilance test: class template matched with input pattern • Hypothesis validation: if pattern close enough to template, categorisation was successful and “resonance” achieved • If not close enough reset winner neuron and try next best matching • Repeat until x=input pattern vi=top-down weight of neuron I r=vigilance threshold – Either vigilance test passed – Or hypotheses (committed neurons) exhausted • ART 1: fraction of bits of input pattern also in template 10/31/2020 Inteligência Artificial 22

Vigilance Threshold • Vigilance threshold sets granularity of clustering • It defines basin of attraction of each prototype • Low threshold – Large mismatch accepted – Few large clusters – Misclassifications more likely Small r, imprecise Large r, fragmented • High threshold – Small mismatch accepted – Many small clusters – Higher precision 10/31/2020 Inteligência Artificial 23

Adaptation • Only weights of winner node are updated • ART 1: only features common to all members of cluster are kept • ART 1: prototype is intersection set of members • ART 2: prototype brought closer to last example • ART 2: b determines amount of modification 10/31/2020 Inteligência Artificial ART 1 ART 2 24

Additional Modules Categorisation result Output layer Gain control Reset module Input layer Input pattern 10/31/2020 Inteligência Artificial 25

Reset Module • Fixed connection weights • Implements the vigilance test • Excitatory connection from input lines • Inhibitory connection from input layer • Output of reset module inhibitory to output layer • Disables firing output node if match with pattern is not close enough • Duration of reset signal lasts until pattern is present 10/31/2020 Inteligência Artificial 1. New pattern p is presented 2. Reset module receives excitatory signal E from input lines 3. All active nodes are reset 4. Input layer is activated 5. Reset module receives inhibitory signal I from input layer 6. I>E 7. If p • v<r inhibition weakens and reset signal is sent 26

Gain module • • 10/31/2020 Fixed connection weights Controls activation cycle of input layer Excitatory connection from input lines Inhibitory connection from output layer Output of gain module excitatory to input layer Shuts down system if noise produces oscillations 2/3 rule for input layer 1. New pattern p is presented 2. Gain module receives excitation signal E from input lines 3. Input layer allowed to fire 4. Input layer is activated 5. Output layer is activated 6. Gain module turned down 7. Now is feedback from output layer that keeps input layer active 8. If p • v<r output layer switched off and gain allows input to keep firing for another match Inteligência Artificial 27

2/3 Rule 2 inputs out of 3 are needed for input layer to be active 1. New pattern p is presented 2. Input layer is activated 3. Output layer is activated 4. Reset signal is sent 5. New match 6. Resonance 7. Input off 10/31/2020 Input signal Gain module Output layer Input layer 1 1 1 0 1 2 1 1 0 1 3 1 0 1 1 4 1 1 0 1 5 1 0 1 1 6 1 0 1 1 7 0 0 1 0 Inteligência Artificial 28

Issues about ART • • • Learned knowledge can be retrieved Fast learning algorithm Difficult to tune vigilance threshold Noise tends to lead to category proliferation New noisy patterns tend to “erode” templates ART is sensitive to order of presentation of data Accuracy sometimes not optimal Assumes samples distribution to be Gaussian (see SOM) Only winner neuron is updated, more “point-to-point” mapping than SOM 10/31/2020 Inteligência Artificial 29

SOM Plasticity vs. ART Plasticity SOM mapping ART mapping new pattern Given new pattern, SOM moves previously committed node and rearrange its neighbours, prior learning is partly “forgotten” 10/31/2020 Inteligência Artificial 30

ART Applications • Natural language processing – Document clustering – Document retrieval – Automatic query • Image segmentation • Character recognition • Data mining – Data set partitioning – Detection of emerging clusters • Fuzzy partitioning • Condition-action association 10/31/2020 Inteligência Artificial 31

Further Topics - ARTMAP Desired output • Composed of 2 ART ANNs and a mapping field • Online, supervised, self-organising ANN • Mapping field: connects output nodes of ART 1 to output nodes of ART 2 • Mapping field trained using hebbian learning • ART 1 partitions input space • ART 2 partitions output space • Mapping field learns stimulus-response associations Input layer ART 2 Output layer Mapping field Output layer ART 1 Input layer Input pattern 10/31/2020 Inteligência Artificial 32

Conclusions - ANNs • ANNs can learn where knowledge is not available • ANNs can generalise from learned knowledge • There are several different ANN models with different capabilities • ANNs are robust, flexible and accurate systems • Parallel distributed processing allows fast computations and fault tolerance • ANNs require a set of parameters to be defined – Architecture – Learning rate • Training is crucial to ANN performance • Learned knowledge often not available (black box) 10/31/2020 Inteligência Artificial 33

Further Readings Mitchell, T. (1997), Machine Learning, Mc. Graw Hill. Duda, R. O. , Hart, P. E. , and Stork, D. G. (2000), Pattern Classification, New York: Wiley. 2 nd Edition. ANN Glossary www. rdg. ac. uk/CSC/Topic/Graphics/Gr. GMatl 601/Matlab 6/toolbox/nnet/a_gloss. html 10/31/2020 Inteligência Artificial 34