Advanced Computing Seminar Data Mining and Its Industrial

Advanced Computing Seminar Data Mining and Its Industrial Applications — Chapter 7 — Fuzzy Sets Zhongzhi Shi, Markus Stumptner, Yalei Hao, Gerald Quirchmayr Knowledge and Software Engineering Lab Advanced Computing Research Centre School of Computer and Information Science University of South Australia 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 1

Outline n Introduction Fuzzy Sets Fuzzy Logic Fuzzy Clustering Fuzzy C-Means Clustering Summary 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi n n n 2

Introduction n Idea of Fuzzy 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 3

Introduction n What does it offer? q q q generate precise solutions from certain or approximate information While other approaches require accurate equations to model real-world behaviors, fuzzy design can accommodate the ambiguities of realworld human language and logic. It provides both an intuitive method for describing systems in human terms and automates the conversion of those system specifications into effective models. 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 4

Introduction n Aristotle q "Law of the Excluded Middle, " n every proposition must either be True or False, A or not. A q q n Ex. , rose is either red or not red It cannot be red and not red Plato q there was a third region (beyond True and False) where these opposites "tumbled about. 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 5

Introduction n In the early 1900 s, Lukasiewicz q n described a three-valued logic, which can best be translated as the term `possible', and assigned it a numeric value between True and False. Knuth, a former student of Lukasiewicz q proposed a three-valued logic apparently missed by Lukasiewicz, whiched used an integral range [1, 0 +1] rather than [0, 1, 2]. 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 6

Introduction n 1960‘s q Lotfi A. Zadeh, a professor of UC Berkeley n n 2/19/2021 Observed that conventional computer logic was incapable of manipulating data representing subjective or vague human ideas such as "an atractive person" or "pretty hot". Fuzzy logic, hence was designed to allow computers to determine the distinctions among data with shades of gray, similar to the process of human reasoning. Chap 7 Fuzzy Set and Logic Zhongzhi Shi 7

Introduction n In 1965, q Zadeh published his seminal work "Fuzzy Sets“ n n 2/19/2021 Described the mathematics of fuzzy set theory, and by extension fuzzy logic. This theory proposed making the membership function (or the values False and True) operate over the range of real numbers [0. 0, 1. 0]. Chap 7 Fuzzy Set and Logic Zhongzhi Shi 8

Introduction n US and certain parts of Europe ignored it, fuzzy logic was excepted with open arms in Japan, China and most Oriental countries. The world's largest number of fuzzy researchers are in China with over 10, 000 scientists. The popularity of fuzzy logic in the Orient reflects the fact that Oriental thinking more easily accepts the concept of "fuzziness". 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 9

Outline n n n 2/19/2021 Introduction Fuzzy Sets Fuzzy Logic Fuzzy Clustering Fuzzy C-Means Clustering Summary Chap 7 Fuzzy Set and Logic Zhongzhi Shi 10

Fuzzy Sets n n n Universal Set X – always a crisp set. Crisp set assigns value {0, 1} to members in X Fuzzy set assigns value [0, 1] to members in X These values are called the membership functions m. Membership function of a fuzzy set A is denoted by : A: X [0, 1] A: [x 1/m 1, x 2/m 2, …, xn/mn} 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 11

Fuzzy Sets n Fuzzy sets q How fuzzy sets quantifying the same information can describe this natural drift. 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 12

Fuzzy Sets n Let's talk about people and "youthness“ q Set S (the universe of discourse) is the set of people n Fuzzy subset YOUNG q q n "to what degree is person x young? " To each person in the universe of discourse, we have to assign a degree of membership in the fuzzy subset YOUNG. A membership function based on the person's age q young(x) = { 1, if age(x) <= 20, (30 -age(x))/10, if 20 < age(x) <= 30, 0, if age(x) > 30 } 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 13

Fuzzy Sets n Graph of membership function q Example: the degree of truth of the statement "Parthiban is YOUNG" is 0. 50 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 14

Fuzzy Sets n. Sets with fuzzy boundaries A = Set of tall people Crisp set A 1. 0 Fuzzy set A 1. 0. 9 Membership . 5 5’ 10’’ 2/19/2021 Heights function 5’ 10’’ 6’ 2’’ Heights 15

Membership Functions ( M F s) n. Characteristics of MFs: Subjective measures q Not probability functions q �“tall” in Asia MFs. 8 �“tall” in the US . 5 �“tall” in NBA . 1 5’ 10’’ 2/19/2021 Heights 16

Fuzzy Sets n Formal definition: A fuzzy set A in X is expressed as a set of ordered pairs: Fuzzy set Membership function (MF) Universe or universe of discourse A fuzzy set is totally characterized by a membership function (MF). 2/19/2021 17

Fuzzy Sets with Discrete Universes n Fuzzy set C = “desirable city to live in” X = {SF, Boston, LA} (discrete and nonordered) C = {(SF, 0. 9), (Boston, 0. 8), (LA, 0. 6)} n Fuzzy set A = “sensible number of children” X = {0, 1, 2, 3, 4, 5, 6} (discrete universe) A = {(0, . 1), (1, . 3), (2, . 7), (3, 1), (4, . 6), (5, . 2), (6, . 1)} 2/19/2021 18

Fuzzy Sets with Cont. Universes n Fuzzy set B = “about 50 years old” X = Set of positive real numbers (continuous) B = {(x, B(x)) | x in X} 2/19/2021 19

Alternative Notation n A fuzzy set A can be alternatively denoted as follows: X is discrete X is continuous Note that S and integral signs stand for the union of membership grades; “/” stands for a marker and does not imply division. 2/19/2021 20

Fuzzy Partition n Fuzzy partitions formed by the linguistic values “young”, “middle aged”, and “old”: lingmf. m 2/19/2021 21

More Definitions n n n Support Core Normality Crossover points Fuzzy singleton a-cut, strong a-cut 2/19/2021 n n n Convexity Fuzzy numbers Bandwidth Symmetricity Open left or right, closed 22

MF Terminology MF 1. 5 a 0 Core X Crossover points a - cut Support 2/19/2021 23

Convexity of Fuzzy Sets n A fuzzy set A is convex if for any l in [0, 1], Alternatively, A is convex is all its a-cuts are convexmf. m 2/19/2021 24

Set-Theoretic Operations n Subset: n Complement: n Union: n Intersection: 2/19/2021 25

Set-Theoretic Operations subset. m 2/19/2021 fuzsetop. m 26

MF Formulation n Triangular MF: Trapezoidal MF: Gaussian MF: Generalized bell MF: 2/19/2021 27

MF Formulation 2/19/2021 disp_mf. m 28

MF Formulation n Sigmoidal MF: Extensions: Abs. difference of two sig. MF Product of two sig. MF disp_sig. m 2/19/2021 29

MF Formulation n L-R MF: Example: 2/19/2021 c=65 c=25 a=60 b=10 a=10 b=40 difflr. m 30

Cylindrical Extension Base set A Cylindrical Ext. of A cyl_ext. m 2/19/2021 31

2 D MF Projection Two-dimensional MF Projection onto X Projection onto Y project. m 2/19/2021 32

Fuzzy Complement n General requirements: q q q n Boundary: N(0)=1 and N(1) = 0 Monotonicity: N(a) > N(b) if a < b Involution: N(N(a) = a Two types of fuzzy complements: q Sugeno’s complement: q Yager’s complement: 2/19/2021 33

Fuzzy Complement Sugeno’s complement: 2/19/2021 Yager’s complement: negation. m 34

Fuzzy Intersection: T-norm n Basic requirements: q q n Boundary: T(0, 0) = 0, T(a, 1) = T(1, a) = a Monotonicity: T(a, b) < T(c, d) if a < c and b < d Commutativity: T(a, b) = T(b, a) Associativity: T(a, T(b, c)) = T(T(a, b), c) Four examples (page 37): q q Minimum: Tm(a, b) Algebraic product: Ta(a, b) Bounded product: Tb(a, b) Drastic product: Td(a, b) 2/19/2021 35

T-norm Operator Minimum: Tm(a, b) 2/19/2021 Algebraic product: Ta(a, b) tnorm. m Bounded product: Tb(a, b) Drastic product: Td(a, b) 36

Fuzzy Union: T-conorm or S-norm n Basic requirements: q q n Boundary: S(1, 1) = 1, S(a, 0) = S(0, a) = a Monotonicity: S(a, b) < S(c, d) if a < c and b < d Commutativity: S(a, b) = S(b, a) Associativity: S(a, S(b, c)) = S(S(a, b), c) Four examples (page 38): q q Maximum: Sm(a, b) Algebraic sum: Sa(a, b) Bounded sum: Sb(a, b) Drastic sum: Sd(a, b) 2/19/2021 37

T-conorm or S-norm Maximum: Sm(a, b) 2/19/2021 Algebraic sum: Sa(a, b) Bounded sum: Sb(a, b) tconorm. m Drastic sum: Sd(a, b) 38

Fuzzy Variables n Membership n Variables whose states are defined by linguistic concepts like low, medium, high. These linguistic concepts are fuzzy sets themselves. 2/19/2021 Very Low Med ium High Very high Temperature Trapezoidal. Chap 7 membership functions Fuzzy Set and Logic Zhongzhi Shi 39

Outline n n n Introduction Fuzzy Sets Fuzzy Logic Fuzzy Clustering Fuzzy C-Means Clustering Summary 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 40

Fuzzy Logic q q Fuzzy logic is a superset of conventional(Boolean) logic It extended to handle the concept of partial truth- truth values between "completely true" and "completely false". It is the logic underlying modes of reasoning which are approximate rather than exact. The importance of fuzzy logic derives from the fact that most modes of human reasoning and especially common sense reasoning are approximate in nature. 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 41

Fuzzy Logic n Essential characteristics of fuzzy logic: q q q In fuzzy logic, exact reasoning is viewed as a limiting case of approximate reasoning. In fuzzy logic everything is a matter of degree. Any logical system can be fuzzified (thus, define Boolean logic as a subset of Fuzzy logic ) In fuzzy logic, knowledge is interpreted as a collection of elastic or, equivalently , fuzzy constraint on a collection of variables Inference is viewed as a process of propagation of elastic constraints. 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 42

Fuzzy Logic n Bivalent sets q To characterize the temperature of a room 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 43

Fuzzy logic n Limitations of bivalent sets q mutually exclusive n n q it is not possible to have membership of more than one set ( opinion would widely vary as to whether 50 degrees Fahrenheit is 'cold' or 'cool' hence the expert knowledge we need to define our system is mathematically at odds with the humanistic world). Clearly, it is not accurate to define a transition from a quantity such as 'warm' to 'hot' by the application of one degree Fahrenheit of heat. In the real world a smooth drift from warm to hot would occur. 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 44

Fuzzy Logic n Fuzzy Set Operations—Union n 2/19/2021 The membership function of the Union of two fuzzy sets A and B with membership functions A and B respectively is defined as the maximum of the two individual membership functions. This is called the maximum criterion. § Union operation in Fuzzy set theory is the equivalent of the OR operation in Boolean algebra Chap 7 Fuzzy Set and Logic Zhongzhi Shi 45

Fuzzy logic n Union—maximum criterion 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 46

Fuzzy Logic n Fuzzy Set Operations—Union q q The membership function of the Union of two fuzzy sets A and B with membership functions A and B respectively is defined as the maximum of the two individual membership functions. This is called the maximum criterion. Union operation in Fuzzy set theory is the equivalent of the OR operation in Boolean algebra 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 47

Fuzzy Logic n Union—maximum criterion 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 48

Fuzzy Loic n Fuzzy Set Operations—Intersection q The membership function of the Intersection of two fuzzy sets A and B with membership functions A and B is defined as the minimum of the two individual membership functions. This is called the minimum criterion. n 2/19/2021 The Intersection operation in Fuzzy set theory is the equivalent of the AND operation in Boolean algebra. Chap 7 Fuzzy Set and Logic Zhongzhi Shi 49

Fuzzy logic n Intersection—minimum criterion 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 50

Fuzzy Logic n Fuzzy Set Operations—Complement q q The membership function of the Complement of a Fuzzy set A with membership function A is defined as the negation of the specified membership function. This is called the negation criterion. The Complement operation in Fuzzy set theory is the equivalent of the NOT operation in Boolean algebra. 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 51

Fuzzy logic n Complement—negation criterion 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 52

Fuzzy Logic n Common rules in classical set theory and Fuzzy set theory q De Morgans law q Associativity 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 53

Fuzzy Logic n Common rules in classical set theory and Fuzzy set theory q Commutativity q Distributivity 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 54

Outline n n n Introduction Fuzzy Sets Fuzzy Logic Fuzzy Clustering Fuzzy C-Means Clustering Summary 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 55

Cluster n A number of similar individuals that occur together as a: two or more consecutive consonants or vowels in a segment of speech b: a group of houses (. . . ) c: an aggregation of stars or galaxies that appear close together in the sky and are gravitationally associated. 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 56

Cluster Analysis n A statistical classification technique for discovering whether the individuals of a population fall into different groups by making quantitative comparisons of multiple characteristics. 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 57

Vehicle Example 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 58

Vehicle Clusters 3500 3000 Lorries Weight [kg] 2500 Sports cars 2000 1500 Medium market cars 1000 500 150 200 250 300 Top speed [km/h] 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 59

Terminology Object or data point 3500 3000 feature space label Lorries 2500 Weight [kg] cluster featur e Sports cars 2000 1500 Medium market cars 1000 500 150 200 250 300 Top speed [km/h] 2/19/2021 featur Chap 7 Fuzzy Set and Logic Zhongzhi Shi e 60

Membership matrix M data point k cluster centre i cluster centre j distance 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 61

c-partition All clusters C together fills the whole universe U A cluster C is never empty and it is smaller than the whole universe U 2/19/2021 Clusters do not overlap There must be at least 2 clusters in a c-partition and at most as many as the number of data points K Chap 7 Fuzzy Set and Logic Zhongzhi Shi 62

Objective function Minimise the total sum of all distances 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 63

2/19/2021 1. Place two cluster centres 2. Assign a fuzzy membership to each data point depending on distance Chap 7 Fuzzy Set and Logic Zhongzhi Shi 64

1. 2. 2/19/2021 Compute the new centre of each class Move the crosses (x) Chap 7 Fuzzy Set and Logic Zhongzhi Shi 65

Iteration 2 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 66

Iteration 5 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 67

Iteration 10 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 68

Iteration 13 (then stop, because no visible change) Each data point belongs to the two clusters to a degree 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 69

M = 0. 0025 0. 9975 0. 0091 0. 9909 0. 0129 0. 9871 0. 0001 0. 9999 0. 0107 0. 9893 0. 9393 0. 0607 0. 9638 0. 0362 0. 9574 0. 0426 0. 9906 0. 0094 0. 9807 0. 0193 The membership matrix M: 1. The last five data points (rows) belong mostly to the first cluster (column) 2. The first five data points (rows) belong mostly to the second cluster (column) 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 70

Each data point belongs to two clusters to different degrees 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 71

Fuzzy membership matrix M Point k’s membership of cluster i Fuzzines s exponent Distance from point k to current cluster centre i Distance from point k to other cluster centres j 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 72

Fuzzy membership matrix M Gravitation to cluster i relative to total gravitation 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 73

Outline n n n Introduction Fuzzy Sets Fuzzy Logic Fuzzy Clustering Fuzzy C-Means Clustering Summary 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 74

Fuzzy C-Means Clustering Input, Output n Input: Unlabeled data set is the number of data point in is the number of features in each vector n Main Output A c-partition of X, which is n Common Additional Output Set of vectors is called “cluster center” matrix U

Fuzzy C-Means Clustering Sample Illustration Rows of U (Membership Functions)

Fuzzy C-Means Clustering (FCM), Objective Function n Optimization of an “objective function” or “performance index” Constraint A-norm Distance Degree of Fuzzification

Fuzzy C-Means Clustering Minimizing Objective Function n Zeroing the gradient of with respect to Note: It is the Center of Gravity

Fuzzy C-Means Clustering Pick n Initial Choices q q q Number of clusters Maximum number of iterations (Typ. : 100) Weighting exponent (Fuzziness degree) n n q q m=1: crisp m=2: Typical Termination measure Termination threshold (Typ. 0. 01) 1 -norm

Fuzzy C-Means Clustering Guess, Iterate n n Guess Initial Cluster Centers Alternating Optimization (AO) q q REPEAT q q q UNTIL ( or )

Fuzzy C-Means Clustering Sample Termination Measure Plot Termination Measure Values Final Membership Degrees

Fuzzy C-Means Clustering Implementation Notes n Process could be shifted one half cycle q q q n n Initialization is done on Iterates become Termination criterion The convergence theory is the same in either case Initializing and terminating on V is advantageous q q q Convenience Speed Storage

Fuzzy C-Means Clustering Pros and Cons n Advantages q q n Unsupervised Always converges Disadvantages q q q Long computational time Sensitivity to the initial guess (speed, local minima) Sensitivity to noise n One expects low (or even no) membership degree for outliers (noisy points)

Summary Advantages of Fuzzy Systems n n n n Conceptually straightforward to understand. The mathematical concepts behind fuzzy reasoning are simple. Flexible: You can modify and add on fuzzy rules without starting from scratch. Tolerant of imprecise data. Everything is imprecise if you look closely enough, but more than that, many things are imprecise even at first glance Can model nonlinear functions of arbitrary complexity. Can create a fuzzy system to match any set of input/output data. Can be built on top of the experience of experts. Is close to natural language. 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 84

Summary Disadvantages of Fuzzy Systems n n Creating the fuzzy rules base can be troublesome It is difficult to create the fuzzy rules base from input/output data if no fuzzy rule extraction technique is used Accuracy of the inference depends directly on the number of fuzzy rules used in complex problem Increasing input variables and fuzzy membership fns used will increase the number of fuzzy rules exponentially. 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 85

References [1] Lofti A. Zadeh. Fuzzy sets, Inf. Control 8, 338 -353, 1965. [2] Jantzen. Tutorial On Fuzzy Clustering. http: //fuzzy. iau. dtu. dk/tutor/fcm/cluster. ppt [3] Nikhil R. Pal, James C. Bezdek, and Richard J. Hathaway. Sequential Competitive Learning and the Fuzzy c-Means Clustering Algorithms. Neural Networks, Vol. 9, No. 5, pp. 787 -796, 1996 [4] Lofti A. Zadeh. Fuzzy Logic, Computer, v. 21 n. 4, p. 83 -93, April 1988 [5] Novak, V. , Perfilieva, I. , Mockor, J. (2000): , Mathematical principles of fuzzy logic. Kluwer 2000. 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 86

www. intsci. ac. cn/shizz Thank you !!! 2/19/2021 Chap 7 Fuzzy Set and Logic Zhongzhi Shi 87