Decision Support Systems Artificial Neural Networks for Data

Decision Support Systems Artificial Neural Networks for Data Mining

Learning Objectives n n n 1 -2 Understand the concept and definitions of artificial neural networks (ANN) Learn the different types of neural network architectures Learn the advantages and limitations of ANN Understand how backpropagation learning works in feedforward neural networks Understand the step-by-step process of how to use neural networks Appreciate the wide variety of applications of neural networks; solving problem types Modified from Decision Support Systems and Business Intelligence Systems 9 E.

Opening Vignette: “Predicting Gambling Referenda with Neural Networks” n Decision situation n Proposed solution n Results n Answer and discuss the case questions 1 -3 Modified from Decision Support Systems and Business Intelligence Systems 9 E.

Neural Network Concepts n n Neural networks (NN): a brain metaphor for information processing Neural computing Artificial neural network (ANN) Many uses for ANN for n n pattern recognition, forecasting, prediction, and classification Many application areas n finance, marketing, manufacturing, and so on 1 -4 Modified from Decision Support Systems and Business Intelligence Systems 9 E.

Biological Neural Networks n Two interconnected brain cells (neurons) 1 -5 Modified from Decision Support Systems and Business Intelligence Systems 9 E.

Processing Information in ANN n A single neuron (processing element – PE) with inputs and outputs 1 -6 Modified from Decision Support Systems and Business Intelligence Systems 9 E.

Elements of ANN n n Processing element (PE) Network architecture n n n Hidden layers Parallel processing Network information processing n n Inputs Outputs Connection weights Summation function 1 -7 Modified from Decision Support Systems and Business Intelligence Systems 9 E.

Elements of ANN Neural Network with one Hidden Layer 1 -8 Modified from Decision Support Systems and Business Intelligence Systems 9 E.

Elements of ANN Summation Function for a Single Neuron (a) and Several Neurons (b) 1 -9 Modified from Decision Support Systems and Business Intelligence Systems 9 E.

Elements of ANN n Transformation (Transfer) Function n Linear function Sigmoid (logical activation) function [0 1] Tangent Hyperbolic function [-1 1] v Threshold value? 1 -10 Modified from Decision Support Systems and Business Intelligence Systems 9 E.

Neural Network Architectures n Several ANN architectures exist Feedforward n Recurrent n Associative memory n Probabilistic n Self-organizing feature maps n Hopfield networks n 1 -11 Modified from Decision Support Systems and Business Intelligence Systems 9 E.

Neural Network Architectures Recurrent Neural Networks 1 -12 Modified from Decision Support Systems and Business Intelligence Systems 9 E.

Neural Network Architectures n Architecture of a neural network is driven by the task it is intended to address n Classification, regression, clustering, general optimization, association Most popular architecture: Feedforward, multi-layered perceptron with backpropagation n 1 -13 Modified from Decision Support Systems and Business Intelligence Systems 9 E.

Learning in ANN A process by which a neural network learns the underlying relationship between input and outputs, or just among the inputs n Supervised learning n n n 1 -14 For prediction type problems (e. g. , backpropagation) Unsupervised learning n Modified from Decisiontype Supportproblems Systems and Business Intelligence Systems 9 E. For clustering (e. g. , adaptive

A Taxonomy of ANN Learning Algorithms 1 -15 Modified from Decision Support Systems and Business Intelligence Systems 9 E.

A Supervised Learning Process Three-step process: 1. Compute temporary outputs 2. Compare outputs with desired targets 3. Adjust the weights and repeat the process 1 -16 Modified from Decision Support Systems and Business Intelligence Systems 9 E.

How a Network Learns n Example: single neuron that learns the inclusive OR operation Learning parameters: n Learning rate n Momentum 1 -17 Modified from Decision Support Systems and Business Intelligence Systems 9 E.

Backpropagation Learning n Backpropagation of Error for a Single Neuron 1 -18 Modified from Decision Support Systems and Business Intelligence Systems 9 E.

Backpropagation Learning n 1 -19 The learning algorithm procedure: 1. Initialize weights with random values and set other network parameters 2. Read in the inputs and the desired outputs 3. Compute the actual output 4. Compute the error (difference between the actual and desired output) 5. Change the weights by working backward through the hidden layers 6. Repeat steps 2 -5 until weights stabilize Modified from Decision Support Systems and Business Intelligence Systems 9 E.

Development Process of an ANN 1 -20 Modified from Decision Support Systems and Business Intelligence Systems 9 E.

An MLP ANN Structure for the Box-Office Prediction Problem 1 -21 Modified from Decision Support Systems and Business Intelligence Systems 9 E.

Testing a Trained ANN Model n Data is split into three parts Training (~60%) n Validation (~20%) n Testing (~20%) n n k-fold cross validation Less bias n Time consuming n 1 -22 Modified from Decision Support Systems and Business Intelligence Systems 9 E.

Sensitivity Analysis on ANN Models n A common criticism for ANN: n The lack of expandability The black-box syndrome! n Answer: sensitivity analysis n Conducted on a trained ANN n The inputs are perturbed while the relative change on the output is measured/recorded n Results illustrates the relative n 1 -23 Modified from Decision Support Systems and Business Intelligence Systems 9 E.

Sensitivity Analysis on ANN Models For a good example n 1 -24 Sensitivity analysis reveals the most important injury severity factors in traffic accidents Modified from Decision Support Systems and Business Intelligence Systems 9 E.

A Sample Neural Network Project Bankruptcy Prediction A comparative analysis of ANN versus logistic regression (a statistical method) n Inputs n X 1: Working capital/total assets n X 2: Retained earnings/total assets n X 3: Earnings before interest and taxes/total assets n X 4: Market value of equity/total debt Modified from Decision Support Systems and Business Intelligence Systems 9 E. n 1 -25

A Sample Neural Network Project Bankruptcy Prediction n Data was obtained from Moody's Industrial Manuals Time period: 1975 to 1982 n 129 firms (65 of which went bankrupt during the period and 64 nonbankrupt) n n Different training and testing propositions are used/compared 90/10 versus 80/20 versus 50/50 n Resampling is used to create 60 data n 1 -26 Modified from Decision Support Systems and Business Intelligence Systems 9 E.

A Sample Neural Network Project Bankruptcy Prediction 1 -27 Network Specifics § Feedforward MLP § Backpropagation § Varying learning and momentum values § 5 input neurons (1 for each financial ratio), § 10 hidden neurons, § 2 output neurons (1 indicating a bankrupt firm and the other indicating a nonbankrupt firm) Modified from Decision Support Systems and Business Intelligence Systems 9 E.

A Sample Neural Network Project Bankruptcy Prediction - Results 1 -28 Modified from Decision Support Systems and Business Intelligence Systems 9 E.

Other Popular ANN Paradigms Self Organizing Maps (SOM) § First introduced by the Finnish Professor Teuvo Kohonen § Applies to clustering type problems 1 -29 Modified from Decision Support Systems and Business Intelligence Systems 9 E.

Other Popular ANN Paradigms Self Organizing Maps (SOM) n SOM Algorithm – 1. 2. 3. 4. 5. 1 -30 6. Initialize each node's weights Present a randomly selected input vector to the lattice Determine most resembling (winning) node Determine the neighboring nodes Adjusted the winning and neighboring nodes (make them more like the input vector) Repeat steps 2 -5 for until a stopping criteria Modified from Decision Support Systems and Business Intelligence Systems 9 E.

Other Popular ANN Paradigms Self Organizing Maps (SOM) n Applications of SOM Customer segmentation n Bibliographic classification n Image-browsing systems n Medical diagnosis n Interpretation of seismic activity n Speech recognition n Data compression n 1 -31 Modified from Decision Support Systems and Business Intelligence Systems 9 E.

Other Popular ANN Paradigms Hopfield Networks § First introduced by John Hopfield § Highly interconnected neurons § Applies to solving complex computational problems (e. g. , optimization problems) 1 -32 Modified from Decision Support Systems and Business Intelligence Systems 9 E.

Applications Types of ANN n Classification n n Regression n Feedforward networks (MLP), radial basis function Clustering n 1 -33 Feedforward networks (MLP), radial basis function, and probabilistic NN Adaptive Resonance Theory (ART) and SOM Association Modified from Decision Support Systems and Business Intelligence Systems 9 E.

Advantages of ANN n n n Able to deal with highly nonlinear relationships Not prone to restricting normality and/or independence assumptions Can handle variety of problem types Usually provides better results compared to its statistical counterparts Handles both numerical and categorical variables 1 -34 Modified from Decision Support Systems and Business Intelligence Systems 9 E.

Disadvantages of ANN n n They are deemed to be black-box solutions, lacking expandability It is hard to find optimal values for large number of network parameters n n n 1 -35 Optimal design is still an art: requires expertise and extensive experimentation It is hard to handle large number of variables Training may take a long time for large datasets; which may require case Modified from Decision Support Systems and Business Intelligence Systems 9 E.

End of the Chapter n Questions / comments… 1 -36 Modified from Decision Support Systems and Business Intelligence Systems 9 E.