Fuzzy Pattern Trees for Regression and Fuzzy Systems

Fuzzy Pattern Trees for Regression and Fuzzy Systems Modeling Robin Senge & Eyke Hüllermeier Knowledge Engineering & Bioinformatics Lab Department of Mathematics and Computer Science Marburg University, Germany WCCI 2010, Barcelona

Outline § Problem Setting § Introduction to Fuzzy Pattern Trees (FPT) § Learning Fuzzy Pattern Trees from Data § Experiments § Relation to Fuzzy Rule-based Systems § Using Fuzzy Pattern Trees for Fuzzy System Modeling 2

Problem Setting Standard setting of supervised learning: § attribute-value representation of instances § let be input domains and the output domain be § input attribute domains discretized by fuzzy sets, e. g. , low, medium and high § rescale to by model functional relationship, i. e. 3

Example: Wine Quality acidity alcohol sulfates sulfur quality low med high G(y) 7. 4 0. 89 9. 4 0. 11 0. 00 0. 56 11 5 0. 50 7. 8 0. 03 10 0. 97 0. 00 0. 46 13 3 0. 30 7. 8 0. 22 10. 5 0. 78 0. 00 0. 8 25 6 0. 60 11. 2 1. 00 9. 3 0. 00 0. 91 17 3 0. 30 7. 4 0. 00 9. 8 0. 00 1. 00 0. 55 12 5 0. 50 7. 3 0. 00 10. 6 0. 81 0. 19 0. 53 21 4 0. 40 8. 9 0. 84 9. 4 0. 16 0. 00 0. 66 17 8 0. 80 § aim: predicting quality of wine based on its ingredients (UCI) § input attributes: acidity, alcohol, sulfates, sulfur, . . . § target (output) attribute is quality 4

Example Fuzzy Pattern Tree (FPT) wine quality 0. 5 AVG 0. 8 0. 2 alcohol high MIN 0. 8 0. 2 acidity low MAX 0. 8 acidity med 0. 3 sulfates med 10. 2 5

Operators Name T-Norm (generalized conjunction) Code Minimum MIN Algebraic AND ALG Lukasiewicz AND LUK Einstein AND EIN Name T-Conorm (generalized disjunction) Code Maximum MAX Algebraic OR COALG Lukasiewicz OR COLUK Einstein OR COEIN Name Averaging Operator Code Weighted Average WA Ordered Weighted Average OWA 6

Features of Fuzzy Pattern Trees § interpretability of the model class high wine quality AVG § modularity: recursive partitioning of critria into sub-criteria alcohol high MIN § flexibility without the tendency to overfit the data § monotonicity in single attributes § built-in feature selection acidity low MAX acidity med sulfates med 7

Learning Fuzzy Pattern Trees from Examples § iteratively refining = growing up trees § start with primitive pattern tree A AVG § growing tree in a top-down manner § selection based on tree performance measure § check relative performance improvement A A AVG A MIN A E B B MIN B MAX B BB MAX MIN DD C greedy beam search (details in the paper) 8

Experiments Are Fuzzy Pattern Trees competitive in terms of predictive accuracy? § 12 data sets from UCI and STATLIB baseline algorithms § Linear Regression (LR) § Multi Layer Perceptron (MLP) § 10 -fold-cross validation § Support Vector Machine with linear kernel (SMO-lin) § root mean squared error (RMSE) § Support Vector Machine with RBF kernel (SMO-rbf) § Fast decision tree learner with reduced error pruning (REPtree) § Fuzzy Rule Learner by Wang & Mendel (FR) 9

Results Ranks according to RMSE Dataset auto-mpg concrete flare 1 M flare 2 C forestfires housing imports-85 machine servo slump winequality-red winequality-white average rank PT-reg 1 2 6 4 6 2 5 2 2 3 1 4 3. 17 LR 5 5 1 1 4 5 3 6 5 2 2 2 3. 42 REPtree 4 1 2 2 3 3 7 7 3 7 6 1 3. 83 SMO-lin 6 7 5 5 2 6 1 1 7 4 3 3 4. 17 MLP 3 3 7 7 8 1 2 8 1 1 7 6 4. 5 SMO-rbf 7 6 3 6 1 7 6 5 8 6 4 5 5. 33 FR 8 8 7 8 8 4 6 8 8 8 7. 42 PT-reg appears to be (at least) competitive to baseline algorithms. 10

Fuzzy Pattern Trees vs. Rule-based Fuzzy Systems § Fuzzy Pattern Trees are closely related to Fuzzy Rule-based Systems fuzzy rules for property: low quality Score(quality is AND low)low(alcohol) =low(alcohol)} MAX { IF MIN {high(acidity), THEN quality is low high(acidity) THEN quality is low MIN {high(acidity), low(alcohol)}, MIN {low(acidity), medium(sulfates)} THEN quality is low IF low(acidity) AND medium(sulfates) THEN quality is low MIN {low(acidity), MIN {high(alcohol), medium(sulfur)} THEN quality is low IF high(alcohol) ANDmedium(sulfates)}, medium(sulfur) THEN quality is low MIN {high(alcohol), medium(sulfur)} } low quality MAX MIN acidity high alcohol low acidity low MIN sulfates med alcohol high sulfur med 11

Fuzzy Systems Modeling alcohol low § usually, not only one fuzzy set on but complete fuzzy partition § let be the fuzzy sets on model functional relationships, i. e. quality (three targets) med high low med 0. 89 0. 11 0. 00 0. 50 0. 03 0. 97 0. 00 0. 40 0. 60 0. 00 0. 22 0. 78 0. 00 0. 40 0. 60 1. 00 0. 00 1. 00 0. 50 0. 00 0. 81 0. 19 0. 00 0. 80 0. 20 0. 84 0. 16 0. 00 0. 20 0. 80 high quality medium quality low quality F-AND F-OR F-AND acid high sulfur low sulfate med acid low high AVG-OP alcohol med sulfate low acid high 12

Fuzzy Systems Modeling contd. low quality medium quality high quality medium quality low quality F-AND F-OR F-AND acid high sulfur low sulfate med acid low AVG-OP alcohol high AVG-OP alcohol med sulfate low acid high 13

Conclusions § Fuzzy Pattern Trees have been introduced as a new model class for regression and fuzzy systems design. § They do have several interesting features (interpretability , monotonicity, flexibility, feature selection). § Data-driven model construction : We can learn Fuzzy Pattern Trees from data. § Regression with Fuzzy Pattern Trees is competitive to state-of-theart algorithms in terms of predictive accuracy. For more information search the web for „kebi marburg“. 14