Knowledge acquisition and processing new methods for neurofuzzy

Knowledge acquisition and processing: new methods for neuro-fuzzy systems Danuta Rutkowska Department of Computer Engineering Technical University of Częstochowa, Poland E-mail: drutko@kik. pcz. czest. pl SOFSEM 2004

Cognitive Technologies Knowledge Acquisition and Inference in the Framework of Soft Computing and Computing with Words SOFSEM 2004

Soft Computing, Computing with Words, . . . • • • Soft computing Computing with words Perception-based systems Computational Intelligence Artificial Intelligence Cognitive sciences Neural networks Fuzzy systems Evolutionary algorithms Intelligent systems

Soft computing techniques

Cognition The word „cognition” cognition comes from the latin word „cognitio”, which means „knowledge”. Cognitive sciences concern thinking, perception, reasoning, creation of meaning, and other functions of a human mind.

Soft computing and cognition The principal aim of soft computing is to exploit the tolerance of uncertainty and vagueness in the area of cognitive reasoning. [Nauck D. , Kruse R. : NEFCLASS-J – A JAVA-Based Soft Computing Tool, In. B. Azvine et al. (Eds. ), Intelligent Systems and Soft Computing, LNAI 1804, Springer-Verlag, Heidelberg, New York (2000), pp. 139 -160].

Artificial Intelligence and cognition The aim of artificial intelligence is to develop paradigms or algorithms that allow machines to perform tasks that involve cognition when performed by humans [A. P. Sage (ed. ), Coincise Encyclopedia of Information Processing in Systems and Organization Pergamon Press, New York, 1990]

Perception and fuzzy systems Perception is very important in human cognition The systems that incorporate perceptions expressed by words are fuzzy systems, introduced by Prof. L. A. Zadeh.

Perception-based systems Fuzzy systems are rule-based systems (knowledge-based systems) that can be viewed as perception-based systems. The rule base of a fuzzy system is composed of fuzzy IF-THEN rules that are similar to the rules used by humans in their reasoning.

Learning by examples is one of the simplest cognitive capabilities of a young child. Artificial neural networks with an inductive, supervised learning algorithm, imitate the cognitive behaviour.

Machine learning research has the potential to make a profound contribution to theory and practice of expert systems, systems as well as to other areas of artificial intelligence Its application to the problem of deriving rule sets from examples is already helping to circumvent the knowledge acquisition bottleneck. [P. Jackson, Introduction to Expert Systems, Addison Wesley, 1999, Chapter 20, p. 399]

Inductive learning The most common form of supervised learning task is called induction. An inductive learning program is one which is capable of learning from examples by a process of generalization. [P. Jackson, Introduction to Expert Systems, Addison Wesley, 1999, Chapter 20, p. 381]

Neural network (MLP)

Model of an artificial neuron

RBF network

Gaussian function

Normalized RBF network

General neuro-fuzzy architecture

Fuzzy reasoning for k-th rule antecedent input variable fuzzification input fuzzy set consequent output variable input value k-th output fuzzy set fuzzy relation

Aggregation and defuzzification aggregation for Mamdani approach output fuzzy set for all N rules S-norm aggregation for logical approach T-norm defuzzification output value centre of consequent fuzzy set Bk

Fuzzy implications: Mamdani, logical Mamdani approach logical approach

An example of a neuro-fuzzy network

More general form of this network

Another example of the NF network

T-norm A triangular norm T is a function of two arguments T: [0, 1]×[0, 1]→[0, 1] which satisfies the following conditions for a, b, c, d∈[0, 1]: Monotonicity : T(a, b)≤T(c, d); a≤c; b≤d Commutativity : T(a, b)=T(b, a) Associativity : T (T(a, b), c)=T(a, T(b, c)) Boundary conditions : T(a, 0)=0; T(a, 1)=a

T-conorm (S-norm) A T-conorm (S-norm) is a function of two arguments S: [0, 1]×[0, 1]→[0, 1], which satisfies the following conditions for a, b, c, d∈[0, 1] Monotonicity : S(a, b)≤S(c, d); a≤c; b≤d Commutativity : S(a, b)=S(b, a) Associativity : S (S(a, b), c)=S(a, S(b, c)) Boundary conditions : S(a, 0)=a; S(a, 1)=1

Neuro-fuzzy inference systems (NFIS)

Fuzzy-logic inference system

Fuzzy-logic inference system: fuzzifier

Fuzzy-logic inference system: fuzzy rule base

Fuzzy-logic inference system: fuzzy inference engine

Fuzzy-logic inference system: defuzzifier

General architecture of Neuro-Fuzzy Inference System NFIS

Flexible neuro-fuzzy system: Mamdani approach

Definition: Fuzzy implication A fuzzy implication is a function I: [0, 1]2→[0, 1] satisfying the following conditions: (I 1) if a 1≤a 3 then I(a 1, a 2)≥I(a 3, a 2), for all a 1, a 2, a 3 [0, 1] (I 2) if a 2≤a 3 then I(a 1, a 2)≤I(a 1, a 3), for all a 1, a 2, a 3 [0, 1] (I 3) I(0, a 2)=1, for all a 2 [0, 1] (falsity implies anything) (I 4) I(a 1, 1)=1, for all a 1 [0, 1] (anything implies tautology) (I 5) I(1, 0)=0 (booleanity)

Fuzzy implications

Flexible neuro-fuzzy system: Logical approach

Flexible neuro-fuzzy system: AND-type compromise NFIS

Flexible neuro-fuzzy system: OR-type compromise NFIS

Flexible neuro-fuzzy system L. Rutkowski and K. Cpałka „Flexible Neuro-Fuzzy Systems”, IEEE Trans. Neural Networks, vol. 14, pp. 554 -574, May 2003

Flexible neuro-fuzzy system: Soft NFIS (1/2)

Flexible neuro-fuzzy system: Soft NFIS (2/2)

Flexible neuro-fuzzy system: NFIS realized by parameterised families of triangular norms (1/2)

Flexible neuro-fuzzy system: NFIS realized by parameterised families of triangular norms (2/2)

Flexible neuro-fuzzy system: NFIS realized by triangular norms with weighted arguments (1/2)

Flexible neuro-fuzzy system: NFIS realized by triangular norms with weighted arguments (2/2)

Flexible neuro-fuzzy system: Glass Identification – experimental results

Flexible neuro-fuzzy system: Glass Identification – weights representation Weights representation in the Glass Identification problem (dark areas correspond to low values and vice versa)

Flexible neuro-fuzzy system: Glass Identification – comparison table

Neuro-fuzzy relational system

Neuro-fuzzy relational system with fuzzy matrix R

Neuro-fuzzy connectionist system (basic architecture)

Rule generation The neuro-fuzzy networks reflect fuzzy IF-THEN rules. The network architectures are created based on the rules. How to get the rules ?

Basic questions: • How many rules ? • What kind of the membership functions (Gaussian, triangular, trapezoidal, etc. ) ? • How to determine parameter values of the membership functions (centers, widths) ?

Many methods There are many methods of rule generation. However, most of the rules obtained by these methods, when applied in neuro-fuzzy systems for classification, result in some misclassifications.

Perception-based approach This method generates fuzzy IF-THEN rules, from a data set, by use of fuzzy granulation. The neuro-fuzzy systems, which utilize these rules, perform without misclassifications.

Multi-stage classification The perception-based approach allows to generate fuzzy rules and perform a multi-stage classification without misclassifications. This method will be illustrated on the IRIS example.

IRIS data set: 150 data items that contain measurements of iris flowers from three species of iris: Setosa, Versicolor, and Virginica; 50 data items for each of the iris species. The data include information about four features of the iris flowers: sepal length, sepal width, petal length, petal width.

Ranges of the measurements of iris flowers (in centimeters) Sepal length 4. 3 – 7. 9 Sepal width 2. 0 – 4. 4 Petal length 1. 0 – 6. 9 Petal width 0. 1 – 2. 5

Ranges within the classes Setosa Versicolor Virginica Sepal 4. 3 – 5. 8 4. 9 – 7. 0 4. 9 – 7. 9 length Sepal 2. 3 – 4. 4 2. 0 – 3. 4 2. 2 – 3. 8 width Petal length 1. 0 – 1. 9 3. 0 – 5. 1 4. 5 – 6. 9 Petal width 0. 1 – 0. 6 1. 0 – 1. 8 1. 4 – 2. 5

Granulated ranges of sepal length 4. 3 – 4. 9 Sestosa 4. 9 – 5. 8 Sestosa Versicolor Virginica 5. 8 – 7. 0 Versicolor Virginica 7. 0 – 7. 9

Granulated ranges of sepal width 2. 0 – 2. 2 Versicolor 2. 2 – 2. 3 Versicolor Virginica 2. 3 – 3. 4 Sestosa Versicolor Virginica Sestosa 3. 4 – 3. 8 – 4. 4

Granulated ranges of petal length 1. 0 – 1. 9 Sestosa 3. 0 – 4. 5 Versicolor 4. 5 – 5. 1 Versicolor Virginica 5. 1 – 6. 9

Granulated ranges of petal width 0. 1 – 0. 6 Sestosa 1. 0 – 1. 4 Versicolor 1. 4 – 1. 8 Versicolor Virginica 1. 8 – 2. 5

Linguistic labels for sepal length 4. 3 – 4. 9 short sepal A 11 4. 9 – 5. 8 medium long sepal A 12 5. 8 – 7. 0 long sepal A 13 7. 0 – 7. 9 very long sepal A 14

Linguistic labels for sepal width 2. 0 – 2. 2 very narrow sepal A 21 2. 2 – 2. 3 narrow sepal A 22 2. 3 – 3. 4 medium wide sepal A 23 3. 4 – 3. 8 wide sepal A 24 3. 8 – 4. 4 very wide sepal A 25

Linguistic labels for petal length 1. 0 – 1. 9 very short petal A 31 3. 0 – 4. 5 medium long petal A 32 4. 5 – 5. 1 long petal A 33 5. 1 – 6. 9 very long petal A 34

Linguistic labels for petal width 0. 1 – 0. 6 very narrow petal A 41 1. 0 – 1. 4 medium wide petal A 42 1. 4 – 1. 8 wide petal A 43 1. 8 – 2. 5 very wide petal A 44

Rule 1 IF sepal is short or medium long and medium wide or very wide and petal is very short and very narrow THEN Setosa IF x 1 is and x 2 is and x 3 is and x 4 is THEN Setosa

Rule 2 IF sepal is medium long or long and very narrow or medium wide and petal is medium long or long and medium wide or wide THEN Versicolor IF x 1 is and x 2 is and x 3 is and x 4 is THEN Versicolor

Rule 3 IF sepal is medium long or very long and narrow or medium wide or wide and petal is long or very long and wide or very wide THEN Virginica IF x 1 is and x 2 is and x 3 is and x 4 is THEN Virginica

NF network for the iris classification

Results of the 1 st stage classification 50 data vectors correctly classified to Setosa 32 data vectors correctly classified to Versicolor 42 data vectors correctly classified to Virginica 26 data vectors – „I do not know” decision: Versicolor or Virginica These data vectors participate in the 2 nd stage of the classification.

2 nd stage classification Two fuzzy IF-THEN rules are formulated, based on the granulated ranges, obtained for the data vectors with the „I do not know” decision in the 1 st stage. The NF network in the 2 nd stage is reduced to the components associated with the Versicolor and Virginica classes.

Results of the 2 nd stage classification 12 data vectors correctly classified to Versicolor 1 data vector correctly classified to Virginica 13 data vectors – „I do not know” decision: Versicolor or Virginica These data vectors participate in the 3 rd stage of the classification. Two new rules are created.

Results of the 3 rd stage classification 4 data vectors correctly classified to Versicolor 5 data vectors correctly classified to Virginica 4 data vectors – „I do not know” decision: Versicolor or Virginica These data vectors participate in the 4 th stage of the classification. Two new rules are created.

Results of the 4 th stage classification 2 data vectors correctly classified to Versicolor 2 data vectors correctly classified to Virginica All data vectors correctly classified after 4 stages of the classification. No misclassifications !

IRIS data: P 1, P 2

IRIS data: P 1, P 3

IRIS data: P 2, P 4

IRIS data: P 3, P 4

Diagnosis of a tumor of mucous membrane of uterus Attributes : • • • period of time after menopause BMI (Body Mass Index) 9 attributes LH (luteinizing hormone ) FSH (follicle-stimulating hormone ) PRL (prolactin ) E 1 (estron) Data: E 2 (estradiol) 52 records of positive diagnosis Aromatase estrogenic receptor Diagnosis: 13 records of negative diagnosis negative (class 0), positive (class 1)

Ranges of the attribute values 0. 5 - 34 20 - 46 0. 5 – 120. 3 1. 36 – 155. 4 2. 4 – 128. 1 156 - 542 0. 04 – 1. 48 2. 28 – 11. 85 0. 72 – 3. 85

Ranges within the classes Class 0 0. 5 - 20 20 - 46 1. 2 – 53. 9 1. 63 – 88. 2 3. 4 – 128. 1 170 - 412 0. 04 – 0. 27 2. 28 – 10. 51 0. 72 – 1. 05 Class 1 0. 5 - 34 20 - 45 0. 5 – 120. 3 1. 36 – 155. 4 2. 4 – 76. 6 156 - 542 0. 05 – 1. 48 3 – 11, 85 0. 91 – 3. 85

Rules for the medical diagnosis

NF network for the medical diagnosis

Results: correct diagnosis 3 cases with the “I do not know” response after the first stage of classification; 62 correct diagnosis for all 65 input vectors. (95. 4% correct decisions, 4. 6 % “I do not know” ) The “I do not know” answers, which mean positive or negative diagnosis, refer to the cases that are difficult to be recognized, because they belong to overlapping regions.

Conclusions (perception-based classification) The perception-based approach allows to generate fuzzy IF-THEN rules in the same way as humans do, and perform the multi-stage classification without misclassifications.

Final conclusions Neuro-fuzzy systems are soft computing methods utilizing artificial neural networks and fuzzy systems. Various connectionist architectures of neuro-fuzzy systems can be constructed. The knowledge acquisition concerns fuzzy IF-THEN rules, and is performed by a learning process. The systems realize an inference (fuzzy reasoning) based on these rules.