Chapter 4Part 3 Differential Heb learning Differential Competitive

Chapter 4_Part 3 Differential Heb learning & Differential Competitive learning Tutor : Prof. Gao Wang. Ying Reporter :

Review Ø Signal Heb Learning Law Ø Competitive Learning Law 2006. 10. 30

Part I: Differential Heb Learning law Its simpler version Hebbian correlations promote spurious causal associations among concurrently active units. Differential correlations estimate the concurrent and presumably causal variation among active units. 2006. 10. 30

Differential Heb Learning Ø Fuzzy Cognitive Maps (FCMs) Ø Adaptive Causal Inference Ø Klopf’s Drive Reinforcement Model Ø Concomitant Variation as Statistical Covariance Ø Pulse-Coded Differential Hebbian Learning 2006. 10. 30

Fuzzy Cognitive Maps (模糊认知映射) Fuzzy signed directed graphs with feedback. It model the world as a collection of classes and causal relations between classes. : Sells of computers : Profits The directed edge from causal concept to concept measures how much causes. 2006. 10. 30

Causal Connection Matrix 2006. 10. 30

TAM recall process We start with the foreign investment policy Then The arrow indicates the threshold operation with, say, ½ as the threshold value. So zero causal input produces zero causal output. contains equals 1 because we are testing the foreign-investment policy option. Next So is a fixed point of the FCM dynamical system. 2006. 10. 30

Strengths and weaknesses of FCM Ø Advantages ☺ Experts: 1. represent factual and evaluative concepts in an interactive framework; 2. quickly draw FCM pictures or respond to questionnaires; 3. consent or dissent to the local causal structure and perhaps the global equilibrations. ☺ FCM knowledge representation and inferencing structure: reduces to simple vector-matrix operations, favors integrated-circuit implementation, and allows extension to neural, statistical, or dynamical systems techniques. Ø Disadvantages It equally encodes the expert’s knowledge or ignorance, wisdom or prejudice. Since different experts differ in how they assign causal strengths to edges, and in which concepts they deem causally relevant, the FCM seems merely to encode its designer’s biases, 2006. 10. 30 and may not even encode them accurately.

Combination of FCMs We combined arbitrary FCM connection matrices by adding augmented(增广)FCM matrices. We add the pointwise to yield the combined FCM matrix : Some experts may be more credible than others. We can weight each expert with a nonnegative credibility weight by multiplicatively weighting the expert’s augmented FCM matrix: Adding FCM matrices represents a simple form of causal learning. 2006. 10. 30

Differential Heb Learning Ø Fuzzy Cognitive Maps (FCMs) Ø Adaptive Causal Inference Ø Klopf’s Drive Reinforcement Model Ø Concomitant Variation as Statistical Covariance Ø Pulse-Coded Differential Hebbian Learning 2006. 10. 30

Adaptive Causal Inference We infer causality between variables when we observe concomitant variation or lagged variation between them. If B changes when A changes, we suspect a causal relationship. The more correlated the changes, the more we suspect a causal relationship, or, more accurately. Time derivatives measure changes. Products of derivatives correlate changes. This leads to the simplest differential Hebbian learning law: 2006. 10. 30

Adaptive Causal Inference The passive decay term forces zero causality between unchanging concepts. The concomitant-variation term indicates causal increase or decrease according to joint concept movement. If and both increase or both decrease, the product of derivatives is positive, v. v. The concomitant-variation term provides a simple causal “arrow of time”. 2006. 10. 30

Differential Heb Learning Ø Fuzzy Cognitive Maps (FCMs) Ø Adaptive Causal Inference Ø Klopf’s Drive Reinforcement Model Ø Concomitant Variation as Statistical Covariance Ø Pulse-Coded Differential Hebbian Learning 2006. 10. 30

Klopf’s Drive Reinforcement Model Harry Klopf independently proposed the following discrete variant of differential Hebbian learning: where the synaptic difference updates the current synaptic efficacy in the first-order difference equation 2006. 10. 30

Klopf’s Drive Reinforcement Model The term drive reinforcement arises from variables and their velocities. Klopf defines a neuronal drive as the weighted signal and a neuronal reinforcer as the weighted difference. A differentiable version of the drive-reinforcement model take the form: The synaptic magnitude amplifies the synapse’s plasticity. In particular, suppose the ijth synapse is excitatory: . Then we can derive: Implicitly the passive decay coefficient scales the term. The coefficient will usually be much smaller than unity to prevent rapid forgetting: 2006. 10. 30

Klopf’s Drive Reinforcement Model Ø Drive-reinforcement synapses can rapidly encode neuronal signal information. Moreover, signal velocities or directions tend to be more robust, more noise tolerant. Ø Unfortunately, it tend to zero as they equilibrate, and they equilibrate exponentially quickly. This holds for both excitatory and inhibitory synapses. 2006. 10. 30

Klopf’s Drive Reinforcement Model The equilibrium condition implies that or in general. This would hold equally in a signal Hebbian model if we replaced the signal product with the magnitude -weighted product. Klopf apparently overcomes this tendency in his simulations by forbidding zero synaptic values: . 2006. 10. 30

Klopf’s Drive Reinforcement Model The simple differential Hebbian learning law equilibrates to More generally the differential Hebbian law learns an exponentially weighted average of sampled concomitant variations, since it has the solution in direct analogy to the signal-Hebbian integral equation. 2006. 10. 30

Differential Heb Learning Ø Fuzzy Cognitive Maps (FCMs) Ø Adaptive Causal Inference Ø Klopf’s Drive Reinforcement Model Ø Concomitant Variation as Statistical Covariance Ø Pulse-Coded Differential Hebbian Learning 2006. 10. 30

Concomitant Variation as Statistical Covariance The very term concomitant variation resembles the term covariance. In differential Hebbian learning we interpreted variation as time change, and concomitance as conjunction or product. Alternatively we can interpret variation spatially as a statistical variance or covariance. Sejnowski has cast synaptic modification as a mean-squared optimization problem and derived a covariance-based solution. After some simplifications the optimal solution takes the form of the covariance learning law 2006. 10. 30

Concomitant Variation as Statistical Covariance Since We can derive The stochastic-approximation approach estimates the unknown expectation with the observed realization product So we estimate a random process with its observed time samples 2006. 10. 30

Concomitant Variation as Statistical Covariance Suppose instead that we estimate the unknown jointexpectation term as the observed time samples in the integrand: This leads to the new covariance learning law How should a synapse estimate the unknown averages and at each time t? 2006. 10. 30

Concomitant Variation as Statistical Covariance We can lag slightly the stochastic-approximation estimate in time to make a martingale assumption. A martingale assumption estimates the immediate future as the present, or the present as the immediate past for some time instant s arbitrarily close to t. The assumption increases in accuracy as s approaches t. 2006. 10. 30

Concomitant Variation as Statistical Covariance This approximation assumes that the signal processes are well-behaved: continuous, have finite variance, and are at least approximately wide-sense stationary. In an approximate sense when time averages resemble ensemble averages, differential Hebbian learning and covariance learning coincide. 2006. 10. 30

Differential Heb Learning Ø Fuzzy Cognitive Maps (FCMs) Ø Adaptive Causal Inference Ø Klopf’s Drive Reinforcement Model Ø Concomitant Variation as Statistical Covariance Ø Pulse-Coded Differential Hebbian Learning 2006. 10. 30

Pulse-Coded Differential Hebbian Learning The velocity-difference property for pulse-coded signal functions The pulse-coded differential Hebbian law replaces the signal velocities in the usual differential Hebbian law with the two differences When no pulse are present, the pulse-coded DHL reduces to the random-signal Heb law. 2006. 10. 30

Pulse-Coded Differential Hebbian Learning Replace the binary pulse functions with the bipolar pulse functions, and then suppose the pulses and the expected pulse frequencies, are pairwise independent. Then the average behavior reduces to the ensemble-averages random signal Hebbian learning law or, equivalently, the classical deterministic-signal Hebbian learning law. 2006. 10. 30

Pulse-Coded Differential Hebbian Learning In the language of estimation theory, both randomsignal Heb learning and random pulse-coded differential Heb learning provide unbiased estimators of signal Heb learning. The pulse frequencies and can be interpret ergodically (time averages equaling space averages) as ensemble averages 2006. 10. 30

Pulse-Coded Differential Hebbian Learning Substituting these martingale assumptions into pulse-coded DHL It suggests that random pulse-coded DHL provides a real-time stochastic approximation to covariance learning This show again how differential Heb learning and covariance learning coincide when appropriate time averages resemble ensemble averages. 2006. 10. 30

Part II: Differential Competitive Learning law Learn only if change! The signal velocity is a local reinforcement mechanism. Its sign indicates whether the jth neurons are winning or losing, and its magnitude measures by how much. 2006. 10. 30

Differential Competitive Learning If the velocity-difference property replaces the competitive signal velocity , then the pulse-coded differential competitive learning law is just the difference of nondifferential competitive laws =0 =1 Winning! Losing! 0 Losing 2006. 10. 30

Competitive signal velocity & supervised reinforcement function Ø Both of them use a sign change to punish misclassifying. Ø Both of them tend to rapidly estimate unknown pattern-class centroids. The unsupervised signal velocity dose not depend on unknown class memberships, it estimates this information with instantaneous win-rate information. Even uses less information: DCL will perform comparably to SCL! 2006. 10. 30

Computation of postsynaptic signal velocity Velocity-difference property Nonlinear derivative reduces to the locally available difference lies between , except when The signal velocity at time is estimated by mere presence or absence of the postsynaptic pulse. slower, stabler pattern environments high-speed sensory environments stimulus patterns shift constantly 2006. 10. 30

Differential-competitive synaptic conjecture then states: Synapse Electrochemically Incoming pulse train Detected postsynaptic pulse Synapse can physically detect the presence or absence of pulse as a change in the postsynaptic neuron’s polarization. Synapse can clearly detects the presynaptic pulse train , and thus the pulse-train’s pulse count in the most recent 30 milliseconds or so. 2006. 10. 30

Behavior patterns involved in animal learning Klopf and Gluck suggest that input signal velocities provide pattern information for this. Process signals Ordinar Microscope store, y case recognize, recall patterns Pulse-coded differential Hebbian learning Classical signal Hebbian learning Pulse-coded differential competitive learning Noisy synaptic vectors can locally estimate pattern centroids in real time without supervision. 2006. 10. 30

Differential Competitive Learning as Delta Modulation The discrete differential competitive learning law represents a neural version of adaptive delta modulation. In communication theory, delta-modulation systems transmit consecutive sampled amplitude differences instead of the sampled amplitude values themselves. A delta-modulation system may transmit only signals, indicating local increase or decrease in the underlying sampled waveform. 2006. 10. 30

Differential Competitive Learning as Delta Modulation Signal difference can be approximated as the activation The signum operator sgn(. ) behaves as a modified threshold function It fixes the step size of the delta modulation, and a variable step size will results in adaptive delta modulation. 2006. 10. 30

Consecutive differences, more informative than consecutive samples We define the statistical correlation between random variables x and z, it takes values in the bipolar interval [1, -1], x and z are positively correlated if , v. v. Let denote pulse difference Suppose the wide-sense-stationary random sequence is zero mean, and each of them has the same finite variance. 2006. 10. 30

Consecutive differences, more informative than consecutive samples Random sequence also has zero mean. The above properties simplifies the variance as If consecutive samples are highly positively correlated, if the differences have less variance than the samples In the pulse-coded case, when the jth neuron wins, it emits a dense pulse train, this winning pulse frequency nay be sufficiently high to satisfy the property. 2006. 10. 30

Thanks for your attention!