Boltzmann Machine Chonbuk National University A I Lab

Boltzmann Machine Chonbuk National University A. I. Lab

Features •

Probability of a unit’s state •

Equilibrium state • Run by repeatedly choosing a unit and setting its state according to the formula • After running for long enough at a certain temperature, the prob. of a global state of the network will depend only on the global state energy, and not on the initial state from the process was started • The log prob. of global state is linear in their energies • Thermal equilibrium that the probability distribution of global states has converged

Equilibrium state • Start running the network from a high temperature and gradually decrease it until we reach a thermal equilibrium at a low temperature we may converge to a distribution where the energy level fluctuates around the global minimum <simulated annealing>

Algorithm : pattern completion •

Algorithm : Boltzmann Input/output network • The input vector is clamped at the visible layer and is never updated. All hidden and output units are updated according to the simulated anealing.

Training the Boltzmann Machine •

Training the Boltzmann Machine • Problems • Time the machine must be run in order to collect equilibrium statistics grow exponentially with machine’s site • Connection strengths are more plastic when the units being connected having activation

Training Algorithm Step 1: Raise the temperature of the NN to some high value. Step 2: Anneal the system until equilibrium is reached at some low-temperature value. (Simulated Annealing) Step 3: Adjust the weights of the network so that G is adjusted. (gradient descent) Step 4: Repeat 1~3 until observed Bolzmann distributions.

Expanded Algorithm Step 1: Clamp one training vector to the visible units Step 2: Anneal the network until eq. at the minimum temperature Step 3: Continue for several processing cycles (step 1 and 2) After each cycle, determine the pairs of connected units which are on simultaneously(co-occurrence) Step 4: Average the co-occurrence results.

Expanded Algorithm •