Fuzzy Pattern Recognition Overview of Pattern Recognition Pattern

  • Slides: 43
Download presentation
Fuzzy Pattern Recognition

Fuzzy Pattern Recognition

Overview of Pattern Recognition • Pattern Recognition Procedure Unknown Speech /Image /Data Feature Extraction

Overview of Pattern Recognition • Pattern Recognition Procedure Unknown Speech /Image /Data Feature Extraction Feature Reduction Classification (supervised) Class Label Known Clustering (unsupervised or self-organizing) Performance Criteria Clusters Cluster Validity

Overview of Pattern Recognition • Supervised Learning for Classification – The class label is

Overview of Pattern Recognition • Supervised Learning for Classification – The class label is known for a set of samples. – Find the decision boundary from the given samples. – For unknown data set, do classification • Unsupervised Learning for Clustering – Set of data is given, find the group or grouping boundary • Reinforcement Learning (Reward/Penalty) – Unkind teacher is given – Trial and Error Scheme

Overview of Pattern Recognition • Classification and Clustering Problem: Which class to assign Problem:

Overview of Pattern Recognition • Classification and Clustering Problem: Which class to assign Problem: How to partition Class 1 How many clusters Class 2 ? Classification Clustering

Overview of Pattern Recognition • Pattern Recognition Algorithm – Based on statistical approach •

Overview of Pattern Recognition • Pattern Recognition Algorithm – Based on statistical approach • Parametric Approach – Bay’es Classifier with Gaussian Density – Nonlinear Boundary or Decision Function • Nonparametric Approach for Density Estimation – Parzen window – K-nearest method – Based on Neural Networks • Classifier – Multilayer Perceptron, ART, Neocogntion, … • Clustering – SOM(Self-Organizing Map)

Fuzzy Pattern Recognition • Classification – Rule-Based Classifier – Fuzzy Perceptron – Fuzzy K-NN

Fuzzy Pattern Recognition • Classification – Rule-Based Classifier – Fuzzy Perceptron – Fuzzy K-NN Algorithm • Clustering – – Fuzzy C-Mean Possibilistic C-Mean Fuzzy C-Shell Clustering Fuzzy Rough Clustering • Cluster Validity – Validity Measures Based on Fuzzy Set Theory

Fuzzy Pattern Recognition

Fuzzy Pattern Recognition

Fuzzy Classification • Rule-Based Classifier – Idea: Nonlinear Partition of Feature Space – How

Fuzzy Classification • Rule-Based Classifier – Idea: Nonlinear Partition of Feature Space – How to find the rule from sample data. • Project the labeled training data, and design membership functions • Fuzzy clustering and projection to obtain membership function

Fuzzy Classification • Fuzzy K-Nearest Neighbor Algorithm – Crisp K-NN Algorithm Class 1 Class

Fuzzy Classification • Fuzzy K-Nearest Neighbor Algorithm – Crisp K-NN Algorithm Class 1 Class 2 K=3 Class 1

Fuzzy Classification • Fuzzy K-Nearest Neighbor Algorithm – Fuzzy K-NN Algorithm Class 1 Class

Fuzzy Classification • Fuzzy K-Nearest Neighbor Algorithm – Fuzzy K-NN Algorithm Class 1 Class 2

Fuzzy Nearest Prototype Classification • Crisp and Fuzzy Nearest Prototype Classification Prototype of Class

Fuzzy Nearest Prototype Classification • Crisp and Fuzzy Nearest Prototype Classification Prototype of Class 1 Prototype of Class 2 Decision Boundary

 • Crisp Version • Fuzzy Version

• Crisp Version • Fuzzy Version

Fuzzy Perceptron • Crisp Single-Layer Perceptron (Two-class problem) Find the linear decision boundary of

Fuzzy Perceptron • Crisp Single-Layer Perceptron (Two-class problem) Find the linear decision boundary of separable data Linear Decision Boundary

Fuzzy Perceptron • Fuzzy Perceptron

Fuzzy Perceptron • Fuzzy Perceptron

Fuzzy Perceptron • Fuzzy Perceptron • Advantage – Generalize the crisp algorithm – Elegant

Fuzzy Perceptron • Fuzzy Perceptron • Advantage – Generalize the crisp algorithm – Elegant termination in non-separable case – Crisp case: Not terminate in finite time

Fuzzy Perceptron • Termination of FP – If misclassifications are all caused by very

Fuzzy Perceptron • Termination of FP – If misclassifications are all caused by very fuzzy data, then terminate the learning. • Note: FP can be combined with kernel-based method. (J. H. Chen & C. S. Chen, IEEE Trans. On NNs, 2002)

Fuzzy C-Mean • Clustering Objective – The aim of the iterative algorithm is to

Fuzzy C-Mean • Clustering Objective – The aim of the iterative algorithm is to decrease the value of an objective function • Notations – Samples – Prototypes – L 2 -distance:

Fuzzy C-Mean • Crisp objective: • Fuzzy objective

Fuzzy C-Mean • Crisp objective: • Fuzzy objective

Fuzzy C-Mean • Crisp C-Mean Algorithm – Initiate k seeds of prototypes p 1,

Fuzzy C-Mean • Crisp C-Mean Algorithm – Initiate k seeds of prototypes p 1, p 2, …, pk – Grouping: Assign samples to their nearest prototypes Form non-overlapping clusters out of these samples – Centering: Centers of clusters become new prototypes – Repeat the grouping and centering steps, until convergence

Fuzzy C-Mean • Crisp C-Mean Algorithm – Grouping: Assigning samples to their nearest prototypes

Fuzzy C-Mean • Crisp C-Mean Algorithm – Grouping: Assigning samples to their nearest prototypes helps to decrease the objective – Centering: Also helps to decrease the above objective, because and equality holds only if

Fuzzy C-Mean • Membership matrix: Uc×n – Uij is the grade of membership of

Fuzzy C-Mean • Membership matrix: Uc×n – Uij is the grade of membership of sample j with respect to prototype i • Crisp membership: • Fuzzy membership:

Fuzzy C-Mean • Objective function of FCM • Introducing the Lagrange multiplier λ with

Fuzzy C-Mean • Objective function of FCM • Introducing the Lagrange multiplier λ with respect to the constraint the objective function as:

Fuzzy C-Mean • Setting the partial derivatives to zero, From the 2 nd equation,

Fuzzy C-Mean • Setting the partial derivatives to zero, From the 2 nd equation, From this fact and the 1 st equation,

Fuzzy C-Mean • Therefore, updating rule is

Fuzzy C-Mean • Therefore, updating rule is

Fuzzy C-Mean • Setting the derivative of J with respect to pi to zero,

Fuzzy C-Mean • Setting the derivative of J with respect to pi to zero,

Fuzzy C-Mean • Update rule of ci: • To summarize:

Fuzzy C-Mean • Update rule of ci: • To summarize:

Fuzzy C-Mean K-means Fuzzy c-means

Fuzzy C-Mean K-means Fuzzy c-means

Fuzzy C-Mean

Fuzzy C-Mean

Fuzzy C-Mean • Gustafson-Kessel Algorithm

Fuzzy C-Mean • Gustafson-Kessel Algorithm

Cluster Validity to Determine Number of Clusters

Cluster Validity to Determine Number of Clusters

Extraction of Rule Base from Fuzzy Cluster

Extraction of Rule Base from Fuzzy Cluster

Possibilistic C-Mean • Problem of FCM – Equal Evidence = Ignorance

Possibilistic C-Mean • Problem of FCM – Equal Evidence = Ignorance

Possibilistic C-Mean • Objective Function of Fuzzy C-Mean – Constraint from Ruspini: Sum of

Possibilistic C-Mean • Objective Function of Fuzzy C-Mean – Constraint from Ruspini: Sum of membership of a datum over all classes should be 1. – Too restrictive condition for noisy data • Objective Function of PCM – Minimize intra-cluster distance – Make membership as large as possible

Possibilistic C-Mean • Necessary Condition • Determination of – Average cluster distance – Based

Possibilistic C-Mean • Necessary Condition • Determination of – Average cluster distance – Based on alpha-cut

Possibilistic C-Mean • Membership according to

Possibilistic C-Mean • Membership according to

Possibilistic C-Mean • Cluster Centers • Inner Product – Gustafson-Kessel (See previous page) –

Possibilistic C-Mean • Cluster Centers • Inner Product – Gustafson-Kessel (See previous page) – Spherical shell cluster

Possibilistic C-Mean • 2 -Pass Algorithm: – Initialize PC Partition – DO Until (Change

Possibilistic C-Mean • 2 -Pass Algorithm: – Initialize PC Partition – DO Until (Change in PC Partition is Small) • Update Prototype • Update PC Partition using average cluster distances – Based on the resulted PC Partition – DO Until (Change in PC Partition is Small) • Update Prototype • Update PC Partition using alpha-cut distances

Possibilistic C-Mean • Advantage – Robust to noisy data – Possibly good to get

Possibilistic C-Mean • Advantage – Robust to noisy data – Possibly good to get the fuzzy rule base FCM-Based C-Shell PCM-Based C-Shell

Other Notion of Distance • Other Notion of Distance – Weights on features –

Other Notion of Distance • Other Notion of Distance – Weights on features – Optimal Weights

Other Notion of Distance FCM with Euclidian Distance FCM with Adaptive Distance

Other Notion of Distance FCM with Euclidian Distance FCM with Adaptive Distance