NEURAL NETWORKS Introduction The branch of AI known

NEURAL NETWORKS

Introduction • The branch of AI known as "neural networks" or "artificial neural networks" has become an increasingly important area of AI. • Neural networks approaches building intelligent systems using architectures and processing capabilities that mimic some biological processes of the human brain.

Basics Biological system • Brains are composed of neurons, which have the unique characteristic that of all types of cells in the body, they do not die. • biological neural networks • http: //psych. hanover. edu/Krantz/neu rotut. html – good basic tutorial and introduction

Portion of a Network: Two Interconnected Biological Cells

Biological system • Human brains are estimated to contain up to 100 billion neurons (1011) and hundreds of different types. • Neurons are found in groups called networks (now you know why this area of AI is called neural networks), each containing thousands of highly interconnected neurons.

Biological system • dendrites - takes in information, provide inputs to the cell, note they have plenty of surface area to facilitate connection to axons of other cells • axon – protuberance that delivers outputs from the neuron to connections with other neurons

Biological system • A neuron does nothing until the collective influence of all its inputs reaches a threshold level. • At that point, the neuron produces a full-strength output in the form of a narrow pulse that proceeds from the cell body, down the axon, and into the axon’s branches. • “it fires” Since it fires or does nothing it is considered an all or nothing device.

Biological system • synapse – gap between the axon and dendrites. • Stimulation at some synapses will encourage the neuron to fire, while stimulation at others discourages the neuron from firing. • Increases or decreases the strength of connection and causes excitation or inhibition of a subsequent neuron

Artificial Systems • Simulated neurons are viewed as a node connected to other nodes via links that correspond to axon-synapse -dendrite connections. • Each link is associated with a weight. • The weight determines the nature (+/ -) and strength of the node’s influence on another. • If the influence of all the links is strong enough the node is activate (similar to the firing of a neuron).

Processing Information in an Artificial Neuron

Artificial Systems (continued) Processing element: • Think of a PE as an artificial neuron. • Receives inputs, processes the outputs, and delivers a single output. • Inputs can be received as raw data or from another PE

Artificial Systems (continued) Network: • Composed of a collection of PE's grouped in layers. • This example has three layers, the middle layer is referred to as the hidden layer. Structure: • Several possible structures

Artificial Neural Network with One Hidden Layer

Processing Information in an Artificial Neural Network Inputs: • Each input is a value of a single attribute. • If I wanted to predict stock prices, one attribute of interest might be "volume" and therefore, I would input the volume (number of shares sold) on a specific day as one input.

Processing Information in an Artificial Neuron

Processing Information in a Network Outputs: • Solution to the problem. For instance, the projected price of the stock

Processing Information in an Artificial Neural Network Weights: • Used to express the relative strength of an input value or from a connecting PE (i. e. , in another layer). • These weights are essential, it is by adjusting these weights that a neural network learns.

Processing Information in a Network Summation function: • Used to compute a single value (weighted average) from all the inputs to a particular PE. • Think of it as the internal stimulation or activation level of the neuron

Processing Information in an Artificial Neural Network • Transformation (Transfer) Function: • based on the summation value, • the value, the transformation (transfer) function produces an output.

Summation Function for Single Neuron(a) and Several Neurons(b)

Processing Information in a Network • There are many possible transformation functions, the sigmoid function is popular. • Sometimes a threshold value is used, which is easier to explain and understand.

Processing Information in a Network • For instance, for summation values less than. 5 a “ 0” might be output, for summation values greater than or equal to. 5 a “ 1” might be output

Learning = Training in Neural Networks • Neural networks are trained using data referred to as a training set. • The process is one of computing outputs, compare outputs with desired answers, adjust weights and repeat.

Learning = Training in Neural Networks • It is necessary to have a fairly large training set, and you need to have the answer for each case in the training set. • Discrepancies between the "right answer" (from the training set) and the computed answer are measured and based on the error, adjustments made.

History of Neural Networks • Basic research on brains dates back quite far. • 1791 - Luigi Galvani (from Bologna) stimulated a frog's muscles with electricity, leading to the discovery that the brain has electrical activity • 1837 - Gocli observed the structure of neurons with axons and connections to dendrites

History of Neural Networks • 1887 - Sherrington: Synaptic interconnection suggested • 1920's - discovered that neurons communicate via chemical impulses called neurotransmitters. • 1930's - research on the chemical processes that produce the electrical impulses.

History of Neural Networks • 1943 - Mc. Cullock and Pitts showed that a NN could be used to code logical relationships such as "x AND y" or "x OR y" • 1950's - Hodgkin and Huxley were awarded the Nobel Prize for work developing the model and recording the electrical signal of the brain at the cellular level

History of Neural Networks • 1969 - Minsky and Papert wrote Perceptrons – showed that one-layer neural networks could not handle statements such as: [(x AND NOT y) OR (y AND NOT x)] – Based on this finding, they conjectured that multi-level NN's would not perform better – Result: funding for NN research dried up, for about 10 years

History of Neural Networks • 1987 - Robert Hecht-Nielsen mathematically disproved Minsky's and Papert's conjecture regarding multi-layer neural networks not being able to perform better than one-layer neural networks. • Since then, this area has been subject to more research.

Basic Network Structures • associative - single layer is representative • hidden layer - can have more than one hidden layer, note that it is uni -directional • double-layer - feeds forward and backward, develops its own categories for representing the data

Neural Network Structures

Artificial Neural Network Development Process Get More, Better Data Refine Structure Select Another Algorithm Reset

Developing Neural Networks Step 1: • collect data Step 2: • separate data into training and test sets, usually random separation • ensure that application is amenable to a NN approach

Developing Neural Networks Step 3: • define a network structure Step 4: • select a learning algorithm • affected by the available tools: shells available Step 5: • set parameter values • affects the length of the training period

Developing Neural Networks Step 6: • transform Data to Network Inputs • data must be NUMERIC, may need to preprocess the data, e. g. , normalize values for a range of 0 to 1 Step 7: • start training • determine and revise weights, check points Step 8: • stop and test: iterative process

Developing Neural Networks Step 9: • implementation • stable weights obtained • begin using the system

Example - Financial Market Analysis • Karl Bergerson of Neural Trading Co uses Neural$, a trading systems with Brain. Maker and a C-based E. S. for money-management rules. • Using 9 years of hand-picked financial data, trained the NN and ran it against a theoretical $10, 000 investment. • After 2 years, the fictional account had grown to $76, 034 (660% appreciation).

Financial Market Analysis continued • When tested on new data, 89% accurate. • Developer quoted "Neural nets are the best tools for pattern recognition, but you can't just dump data into one and expect to get wonderful results. The most important factor is your training data. You have to have your whole act together, training, design, and the right tools. ”

Financial Market Analysis continued • Some of the attributes used: price, volume, advance/decline etc. • The neural network predicts market fluctuation and the expert system component flags buying or selling opportunities

Sales Support • Veratex Corp. distributes medical and dental products. • They send unsolicited catalogs to physicians and dentists. • When a customer buys something, their name is added to the customer database. • 40 telemarketers then call the names in the database for reorders

Sales Support - continued The problem: • many dormant accounts, i. e. , customers who had not placed reorders. • The telemarketers are not trained to prospect for new clients and they only have about 20% of their time allocated for calling dormant accounts.

Sales Support - The Problem (continued) • The database contains 44, 000 customers, these represent potential business that is not being tapped. • Further, as the data ages, it becomes less reliable (i. e. , physicians and dentists move and retire).

Sales Support - continued The solution: • The company hired Churchill Systems to build a back-propogation (a learning algorithm) to identify those customers in the dormant pool most likely to place reorders. • With this information, telemarketers could focus their limited time on customers with the most potential.

Sales Support - continued • System was built using NNU 400 neural network utility (from IBM). • Inputs consisted of statistical and demographic data culled from Dun & Bradstreet and other sources. • The network was applied against the customer list, giving each customer a numerical rating which was put into the customer records and then used as a sort key.

Sales Support Results: • President of Churchill Systems "More Veratex accounts were reopened in five months, than similar periods (without the network). • "The patterns and interrelationships uncovered by the neural network proved to be an extremely valuable resource for Veratex marketing analysts. "

Sales Support - continued General comments: • "A lot of people think you can avoid knowledge engineering. Forget it you can't do it. You really have to get down to the business problem before you can do anything else. ” • In fact, Light claims that a large part of the neural network's development time entailed gathering, cleaning up, and organizing the appropriate data.

Horse Bloodlines • University of California at Davis School of Veterinary Medicine conducts blood tests to confirm the bloodlines of Thoroughbred horses. • Thoroughbreds cannot be raced unless their bloodlines are known. • To do this, 142 separate reaction tests must be run on a blood sample. – as many as 72, 000 tests per day.

Horse Bloodlines - continued • The problem: how to automate this function so that a technician didn't have to perform this job. • A neural network was trained to read these tests starting April 1987, was pilot tested for about one year (1989? , article printed in 1990).

Horse Bloodlines - continued • The neural network was trained to read a blood test and determine if a reaction occurred - a simple yes or no. • The neural network must be accurate, the results of the lab cannot be questioned or breeders will not use the lab.

Horse Bloodlines - continued • Specifically, the network must "look" at a drop of blood and decide if the cells in the drop of blood have clumped together (agglutinated). • The system "sees" using a video camera that divides the field of vision into 262, 144 pixels of information.

Horse Bloodlines - continued • Using this information, the developer believed it would have taken 28 million years to teach the network the concept of "clumpiness". • It was just too much raw data.

Horse Bloodlines - continued • Lendaris (1970) pioneered the scanning of aerial surveillance photographs by computer to detect orderly man-made features such as orchards, road intersections, etc. • The contribution that could be applied to the blood testing problem was the use of the Fourier transform, developed by a 19 th-century French physicist and mathematician.

Horse Bloodlines - continued • This transformation converts massive amounts of data into oscillating waves of energy and can be used to highlight sharp gradations, such as the edge of a building or the edge of a clump of blood. • The Fourier transformed 262, 144 pixels into 48 data points that the network was easily trained to recognize.

Horse Bloodlines - continued • The neural network tool used was supplied by Science Application International Corp (of San Diego), cost $25, 000 and called Delta II. • It includes both software and an accelerator board to enhance a 386 machine.

Horse Bloodlines - continued • The developer of the system to read blood tests is skeptical about the commercial development of Neural Nets: • "The difficulty is, what will you sell? A neural net is just an algorithm - a method of calculation like a [statistical] regression or multiplication. It is hard to protect a product like that - hard to get a commercial handle on it. "

Horse Bloodlines - continued • He believes that will succeed are a variety of hardware systems with the neural network learning method automated and embedded in the hardware. • This will also alleviate the user from having to understand as much about the neural network.

KBS vs. Neural Networks

Advantages of Neural Nets • Able to learn any complex non-linear mapping (31) • Do not make a priori assumptions about the distribution of • • the data/input-output mapping function (30) Very flexible with respect to incomplete, missing, noisy data, ‘fault tolerant’ (29) Easily updated, suitable to dynamic environments (15) Overcome some limitations of other statistical methods, while generalizing them (15) Hidden nodes, in feed-forward, can be regarded as latent/unobservable variables (5) Can implement on parallel hardware, increasing accuracy and learning speed (4) Can be highly automated, minimizing human involvement (3) Specially suited to tackle problems in non-conservative domains (3)

Disadvantages of Neural Nets • Lack theoretical background, no explanation, ‘black box’ (28) • Selection of network topology and parameters lacks theoretical background, ‘trial and error’ (21) • Learning process can be very time consuming (11) • Can overfit the training data, becoming useless for generalization (10) • No explicit set of rules to select a suitable ANN paradigm/learning algorithm (8) • Too dependent on the quality/ amount of data available (6) • Can get stuck in local optima, narrow valleys during training (5) • Techniques still rapidly evolving and not reliable or robust enough yet (3) • Lack classical statistical properties. Confidence intervals and hypothesis testing are not available (2)

Potential Problems with Neural Networks • The military has been experimenting with ANN techniques for sometime. • One application of interest was to identify objects on the battlefield at night. For instance, distinguishing the difference between a tank and a rock. • A scanner showed an automated neural network thousands of photographs of tanks, rocks, and other battlefield objects.

Potential Problems with Neural Networks (cont. ) • After training the ANN could correctly distinguish a tank from a rock 100% of the time. • Later, it was discovered that all the photos of the tanks had been taken with the same camera. – The tank photos were all slightly darker than the photos of the other objects. • What the ANN had really learned was to identify the camera used to take the picture, not the difference between rocks and tanks!