1 Last update 26 November 2007 Advanced databases

1 Last update: 26 November 2007 Advanced databases – Inferring implicit/new knowledge from data(bases): Single-table and multirelational data mining Bettina Berendt Katholieke Universiteit Leuven, Department of Computer Science http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/ Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

2 Agenda Motivation / Focus on decision tree learning Input: Concepts Input: Instances Input: Attributes and levels of measurement Data preparation for WEKA and beyond Output: Decision trees (and related patterns) Algorithm: ID 3 (and variants) Multirelational data mining Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

3 Recall: Knowledge discovery and styles of reasoning Business understanding Evaluation 1. n Learn a model from the data (observed instances) n Generally involves induction (during Modelling) Deployment 2. n Apply the model to new instances n Corresponds to deduction (if one assumes that the model is true) Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

4 Phases talked about today Business understanding Evaluation 1. n Learn a model from the data (observed instances) n Generally involves induction (during Modelling) Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

5 Decision tree learning (1): Decision rules What contact lenses to give to a patient? (Could be based on background knowledge, but can also be learned from the WEKA contact lens data) Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

6 Decision tree learning (2): Classification / prediction In which weather will someone play (tennis etc. )? (Learned from the WEKA weather data) Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

7 Learned from data like this: Outlook Temp Humidity Windy Play Sunny Hot High False No Sunny Hot High True No Overcast Hot High False Yes Rainy Mild High False Yes Rainy Cool Normal True No Overcast Cool Normal True Yes Sunny Mild High False No Sunny Cool Normal False Yes Rainy Mild Normal False Yes Sunny Mild Normal True Yes Overcast Mild High True Yes Overcast Hot Normal False Yes Rainy Mild High True No Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

8 Agenda Motivation / Focus on decision tree learning Input: Concepts Input: Instances Input: Attributes and levels of measurement Data preparation for WEKA and beyond Output: Decision trees (and related patterns) Algorithm: ID 3 (and variants) Multirelational data mining Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

What’s a concept? Styles of learning: Classification learning: predicting a discrete class Association learning: detecting associations between features Clustering: grouping similar instances into clusters Numeric prediction: predicting a numeric quantity Concept: thing to be learned Concept description: output of learning scheme Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/ 9

Classification learning Example problems: weather data, contact lenses, irises, labor negotiations Classification learning is supervised 10 Scheme is provided with actual outcome Outcome is called the class of the example Measure success on fresh data for which class labels are known (test data) In practice success is often measured subjectively Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

11 Example weather data Outlook Temp Humidity Windy Play Sunny Hot High False No Sunny Hot High True No Overcast Hot High False Yes Rainy Mild High False Yes Rainy Cool Normal True No Overcast Cool Normal True Yes Sunny Mild High False No Sunny Cool Normal False Yes Rainy Mild Normal False Yes Sunny Mild Normal True Yes Overcast Mild High True Yes Overcast Hot Normal False Yes Rainy Mild High True No Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

Association learning 12 Can be applied if no class is specified any kind of structure is considered “interesting” Difference to classification learning: Can predict any attribute’s value, not just the class, and more than one attribute’s value at a time Hence: far more association rules than classification rules Thus: constraints are necessary Minimum coverage and minimum accuracy Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

13 Clustering Finding groups of items that are similar Clustering is unsupervised The class of an example is not known Success often measured subjectively Sepal length Sepal width Petal length Petal width Type 1 5. 1 3. 5 1. 4 0. 2 Iris setosa 2 4. 9 3. 0 1. 4 0. 2 Iris setosa 51 7. 0 3. 2 4. 7 1. 4 Iris versicolor 52 6. 4 3. 2 4. 5 1. 5 Iris versicolor 101 6. 3 3. 3 6. 0 2. 5 Iris virginica 102 5. 8 2. 7 5. 1 1. 9 Iris virginica … … … Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

14 Numeric prediction Variant of classification learning where “class” is numeric (also called “regression”) Learning is supervised Scheme is being provided with target value Measure success on test data Outlook Temperature Humidity Windy Play-time Sunny Hot High False 5 Sunny Hot High True 0 Overcast Hot High False 55 Rainy Mild Normal False 40 … … … Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

15 Agenda Motivation / Focus on decision tree learning Input: Concepts Input: Instances Input: Attributes and levels of measurement Data preparation for WEKA and beyond Output: Decision trees (and related patterns) Algorithm: ID 3 (and variants) Multirelational data mining Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

What’s in an example? Instance: specific type of example Represented as a single relation/flat file Rather restricted form of input Thing to be classified, associated, or clustered Individual, independent example of target concept Characterized by a predetermined set of attributes Input to learning scheme: set of instances/dataset 16 No relationships between objects Most common form in practical data mining Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

17 Instances in the weather data Outlook Temp Humidity Windy Play Sunny Hot High False No Sunny Hot High True No Overcast Hot High False Yes Rainy Mild High False Yes Rainy Cool Normal True No Overcast Cool Normal True Yes Sunny Mild High False No Sunny Cool Normal False Yes Rainy Mild Normal False Yes Sunny Mild Normal True Yes Overcast Mild High True Yes Overcast Hot Normal False Yes Rainy Mild High True No Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

18 Agenda Motivation / Focus on decision tree learning Input: Concepts Input: Instances Input: Attributes and levels of measurement Data preparation for WEKA and beyond Output: Decision trees (and related patterns) Algorithm: ID 3 (and variants) Multirelational data mining Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

What’s in an attribute? Each instance is described by a fixed predefined set of features, its “attributes” But: number of attributes may vary in practice 19 Possible solution: “irrelevant value” flag Related problem: existence of an attribute may depend of value of another one Possible attribute types (“levels of measurement”): Nominal, ordinal, interval and ratio Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

Nominal quantities Values are distinct symbols Values themselves serve only as labels or names Nominal comes from the Latin word for name Example: attribute “outlook” from weather data 20 Values: “sunny”, ”overcast”, and “rainy” No relation is implied among nominal values (no ordering or distance measure) Only equality tests can be performed Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

Ordinal quantities Impose order on values But: no distance between values defined Example: attribute “temperature” in weather data 21 Values: “hot” > “mild” > “cool” Note: addition and subtraction don’t make sense Example rule: temperature < hot play = yes Distinction between nominal and ordinal not always clear (e. g. attribute “outlook”) Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

Interval quantities 22 Interval quantities are not only ordered but measured in fixed and equal units Example 1: attribute “temperature” expressed in degrees Fahrenheit Example 2: attribute “year” Difference of two values makes sense Sum or product doesn’t make sense Zero point is not defined! Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

Ratio quantities are ones for which the measurement scheme defines a zero point Example: attribute “distance” Distance between an object and itself is zero Ratio quantities are treated as real numbers 23 All mathematical operations are allowed But: is there an “inherently” defined zero point? Answer depends on scientific knowledge (e. g. Fahrenheit knew no lower limit to temperature) Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

24 Agenda Motivation / Focus on decision tree learning Input: Concepts Input: Instances Input: Attributes and levels of measurement Data preparation for WEKA and beyond Output: Decision trees (and related patterns) Algorithm: ID 3 (and variants) Multirelational data mining Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

Notes on Preparing the input (CRISP-DM Data preparation stage) Denormalization is not the only issue Problem: different data sources (e. g. sales department, customer billing department, …) 25 Differences: styles of record keeping, conventions, time periods, data aggregation, primary keys, errors Data must be assembled, integrated, cleaned up “Data warehouse”: consistent point of access External data may be required (“overlay data”) Critical: type and level of data aggregation Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

The ARFF format 26 % % ARFF file for weather data with some numeric features % @relation weather @attribute @attribute outlook {sunny, overcast, rainy} temperature numeric humidity numeric windy {true, false} play? {yes, no} @data sunny, 85, false, no sunny, 80, 90, true, no overcast, 83, 86, false, yes. . . Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

27 Additional attribute types ARFF supports string attributes: @attribute description string n Similar to nominal attributes but list of values is not prespecified It also supports date attributes: @attribute today date n Uses the ISO-8601 combined date and time format yyyy. MM-dd-THH: mm: ss Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

28 Sparse data In some applications most attribute values in a dataset are zero n E. g. : word counts in a text categorization problem ARFF supports sparse data 0, 26, 0, 0, 0 , 0, 63, 0, 0, 0, “class A” 0, 0, 0, 42, 0, 0, 0, “class B” {1 26, 6 63, 10 “class A”} {3 42, 10 “class B”} This also works for nominal attributes (where the first value corresponds to “zero”) Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

Attribute types Interpretation of attribute types in ARFF depends on learning scheme Numeric attributes are interpreted as 29 ordinal scales if less-than and greater-than are used ratio scales if distance calculations are performed (normalization/standardization may be required) Instance-based schemes define distance between nominal values (0 if values are equal, 1 otherwise) Integers in some given data file: nominal, ordinal, or ratio scale? Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

Nominal vs. ordinal 30 Attribute “age” nominal If age = young and astigmatic = no and tear production rate = normal then recommendation = soft If age = pre-presbyopic and astigmatic = no and tear production rate = normal then recommendation = soft Attribute “age” ordinal (e. g. “young” < “pre-presbyopic” < “presbyopic”) If age pre-presbyopic and astigmatic = no and tear production rate = normal then recommendation = soft Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

Missing values Frequently indicated by out-of-range entries Types: unknown, unrecorded, irrelevant Reasons: 31 malfunctioning equipment changes in experimental design collation of different datasets measurement not possible Missing value may have significance in itself (e. g. missing test in a medical examination) Most schemes assume that is not the case: “missing” may need to be coded as additional value “? ” in ARFF denotes a missing value Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

Inaccurate values 32 Reason: data has not been collected for mining it Result: errors and omissions that don’t affect original purpose of data (e. g. age of customer) Typographical errors in nominal attributes values need to be checked for consistency Typographical and measurement errors in numeric attributes outliers need to be identified Errors may be deliberate (e. g. wrong zip codes) Other problems: duplicates, stale data Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

Getting to know the data Simple visualization tools are very useful 33 Nominal attributes: histograms (Distribution consistent with background knowledge? ) Numeric attributes: graphs (Any obvious outliers? ) 2 -D and 3 -D plots show dependencies Need to consult domain experts Too much data to inspect? Take a sample! Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

34 Agenda Motivation / Focus on decision tree learning Input: Concepts Input: Instances Input: Attributes and levels of measurement Data preparation for WEKA and beyond Output: Decision trees (and related patterns) Algorithm: ID 3 (and variants) Multirelational data mining Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

Decision trees 35 “Divide-and-conquer” approach produces tree Nodes involve testing a particular attribute Usually, attribute value is compared to constant Other possibilities: Comparing values of two attributes Using a function of one or more attributes Leaves assign classification, set of classifications, or probability distribution to instances Unknown instance is routed down the tree Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

Nominal and numeric attributes 36 Nominal: number of children usually equal to number values attribute won’t get tested more than once Other possibility: division into two subsets Numeric: test whether value is greater or less than constant attribute may get tested several times Other possibility: three-way split (or multi-way split) Integer: less than, equal to, greater than Real: below, within, above Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

Missing values 37 Does absence of value have some significance? Yes “missing” is a separate value No “missing” must be treated in a special way Solution A: assign instance to most popular branch Solution B: split instance into pieces Pieces receive weight according to fraction of training instances that go down each branch Classifications from leave nodes are combined using the weights that have percolated to them Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

Classification rules 38 Popular alternative to decision trees Antecedent (pre-condition): a series of tests (just like the tests at the nodes of a decision tree) Tests are usually logically ANDed together (but may also be general logical expressions) Consequent (conclusion): classes, set of classes, or probability distribution assigned by rule Individual rules are often logically ORed together Conflicts arise if different conclusions apply Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

39 An example If If If outlook = sunny and humidity = high then play = no outlook = rainy and windy = true then play = no outlook = overcast then play = yes humidity = normal then play = yes none of the above then play = yes Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

Trees for numeric prediction 40 Regression: the process of computing an expression that predicts a numeric quantity Regression tree: “decision tree” where each leaf predicts a numeric quantity n Predicted value is average value of training instances that reach the leaf Model tree: “regression tree” with linear regression models at the leaf nodes n Linear patches approximate continuous function Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

41 An example Outlook Temperature Humidity Windy Play-time Sunny Hot High False 5 Sunny Hot High True 0 Overcast Hot High False 55 Rainy Mild Normal False 40 … … … Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

42 Agenda Motivation / Focus on decision tree learning Input: Concepts Input: Instances Input: Attributes and levels of measurement Data preparation for WEKA and beyond Output: Decision trees (and related patterns) Algorithm: ID 3 (and variants) Multirelational data mining Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

Constructing decision trees Strategy: top down Recursive divide-and-conquer fashion 43 First: select attribute for root node Create branch for each possible attribute value Then: split instances into subsets One for each branch extending from the node Finally: repeat recursively for each branch, using only instances that reach the branch Stop if all instances have the same class Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

Which attribute to select? 44 Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

Which attribute to select? 45 Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

46 Criterion for attribute selection Which is the best attribute? Popular impurity criterion: information gain Want to get the smallest tree Heuristic: choose the attribute that produces the “purest” nodes Information gain increases with the average purity of the subsets Strategy: choose attribute that gives greatest information gain Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

Computing information 47 Measure information in bits Given a probability distribution, the info required to predict an event is the distribution’s entropy Entropy gives the information required in bits (can involve fractions of bits!) Formula for computing the entropy: Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

Example: attribute Outlook 48 Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

Computing information gain 49 Information gain: information before splitting – information after splitting gain(Outlook ) = info([9, 5]) – info([2, 3], [4, 0], [3, 2]) = 0. 940 – 0. 693 = 0. 247 bits Information gain for attributes from weather data: gain(Outlook ) gain(Temperature ) gain(Humidity ) gain(Windy ) = 0. 247 bits = 0. 029 bits = 0. 152 bits = 0. 048 bits Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

Continuing to split 50 gain(Temperature ) = 0. 571 bits gain(Humidity ) = 0. 971 bits gain(Windy ) = 0. 020 bits Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

Final decision tree 51 Note: not all leaves need to be pure; sometimes identical instances have different classes Splitting stops when data can’t be split any further Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

Wishlist for a purity measure Properties we require from a purity measure: 52 When node is pure, measure should be zero When impurity is maximal (i. e. all classes equally likely), measure should be maximal Measure should obey multistage property (i. e. decisions can be made in several stages): Entropy is the only function that satisfies all three properties! Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

Properties of the entropy The multistage property: Simplification of computation: 53 Note: instead of maximizing info gain we could just minimize information Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

Discussion / outlook decision trees Top-down induction of decision trees: ID 3, algorithm developed by Ross Quinlan 54 Various improvements, e. g. C 4. 5: deals with numeric attributes, missing values, noisy data Gain ratio instead of information gain (see Witten & Frank slides, ch. 4, pp. 40 -45) Similar approach: CART … Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

55 Weather data with mixed attributes Some attributes have numeric values If If If Outlook Temperature Humidity Windy Play Sunny 85 85 False No Sunny 80 90 True No Overcast 83 86 False Yes Rainy 75 80 False Yes … … … outlook = sunny and humidity > 83 then play = no outlook = rainy and windy = true then play = no outlook = overcast then play = yes humidity < 85 then play = yes none of the above then play = yes Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

56 Dealing with numeric attributes Discretize numeric attributes Divide each attribute’s range into intervals Sort instances according to attribute’s values Place breakpoints where class changes (majority class) This minimizes the total error Example: temperature from weather data 64 65 68 Yes | No | Yes 69 70 71 72 72 75 75 80 81 83 85 Yes | No No Yes | Yes Yes | No Outlook Temperature Humidity Windy Play Sunny 85 85 False No Sunny 80 90 True No Overcast 83 86 False Yes Rainy 75 80 False Yes … … … Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

The problem of overfitting This procedure is very sensitive to noise 57 One instance with an incorrect class label will probably produce a separate interval Also: time stamp attribute will have zero errors Simple solution: enforce minimum number of instances in majority class per interval Example (with min = 3): 64 65 68 Yes | No | Yes 69 70 71 72 72 75 75 80 81 83 85 Yes | No No Yes | Yes Yes | No 64 Yes 69 70 71 72 72 75 75 80 81 83 85 Yes | No No Yes Yes | No Yes No 65 No 68 Yes Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

58 Agenda Motivation / Focus on decision tree learning Input: Concepts Input: Instances Input: Attributes and levels of measurement Data preparation for WEKA and beyond Output: Decision trees (and related patterns) Algorithm: ID 3 (and variants) Multirelational data mining Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

59 Motivation: Fictitious E-commerce example – who is a big spender? Typical possible result so far: However, this might be the case: Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

60 On terminology (1) Reduces to propositional logic „Propositional rule“ Info from multiple relations from a relational DB; expressed in a subset of first-order logic (aka predicate l. aka relational logic) „Relational rule“ Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

61 On terminology (2) „multirelational data mining“ „relational data mining“ Intersection of machine learning and logic programming: „inductive logic programming“ Info from multiple relations from a relational DB; expressed in a subset of first-order logic (aka predicate l. aka relational logic) „Relational rule“ Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

The (most common) problem: Learning the logical definition of a relation n n Given: (Training) examples: tuples that belong or do not belong to the target relation Background knowledge: other relation definitions etc. in the same logical language Find: n 62 A predicate definition that defines the target relation in terms of the relations from the background knowledge Formally (learning from entailment setting, Muggleton 1991) Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

63 Enter real data In most cases: must relax criteria of consistency and completeness Statistical criteria Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

64 Example Given: Learn the predicate definition: Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

65 First solution idea: join or aggregate towards a single table Slightly modify the example: n customer(Cust. ID; Name; Age; Spends. ALot) n purchase(Cust. ID; Product. ID; Date; Value; Payment. Mode) Combination 1: natural join purchase 1(Cust. ID; Product. ID; Date; Value; Payment. Mode; Name; Age; Spends. ALot) Combination 2: aggregation customer 1(Cust. ID; Age; NOf. Purchases; Total. Value; Spends. ALot) Can we find this pattern? : Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

66 Second solution idea: A cleverer way of propositionalization n Early approach: LINUS (Lavrac, Dzeroski, & Grobelnik, 1991) n Use the background knowledge to create new attributes n Works for a restricted class of problems: Function-free program clauses which are typed (each var. Is associated with a predetermined set of values), Constrained (all var. s in the body of a clause also appear in the head), and Nonrecursive (the predicate symbol in the head does not appear in any of the literals in the body) Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

67 Third solution idea: Upgrade propositional approaches n n n MRDM algorithms have much in common with their propositional counterparts: Learning as search (seach for patterns valid in the given data) A lattice of hypotheses as search space Key differences: Representation of data and patterns Refinement operators Testing coverage (whether a rule explains an example) A general „recipe“ for upgrading: Van Laer & De Raedt (2001) Keep as much as possible, upgrade only the key notions needed Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

An example: First-order decision trees: TILDE (Blockeel & De Raedt, 1998) with propositional counterpart and special case C 4. 5 68 This tree predicts the maintenance action that should be taken on machine M: maintenance(M, A) How to expand a node? • Search in the space of clauses (cf. prop. : attributes/values) • use constraints to limit no. of candidates („declarative bias“) • Each refinement operator: -App lies a sub stit Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/ utio

69 Major difference between propositional and relational dec. trees n Is in the tests that can appear in internal nodes n Each test is a query: a conjuntion of literals with existentially quantified variables n If the query succeeds take the „yes“ branch n Variables can be shared among nodes n The actual test in a node = the conjunction of literals in the node itself and the literals on the path from the root of the tree Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

70 Why order matters (another example) Forall x Exists y : haspart(x, y) Exists y Forall x : haspart(x, y) Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

71 Interpretation of the tree in terms of a decision list n List: the order of tests expressed by the clauses matters! Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

72 Next lecture Motivation / Focus on decision tree learning Input: Concepts Input: Instances Input: Attributes and levels of measurement Data preparation for WEKA and beyond Output: Decision trees (and related patterns) How good are these models? Evaluation Algorithm: ID 3 (and variants) Multirelational data mining Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/

73 References / background reading; acknowledgements n All parts on single-table processing are based on Witten, I. H. , & Frank, E. (2005). Data Mining. Practical Machine Learning Tools and Techniques with Java Implementations. 2 nd ed. Morgan Kaufmann. http: //www. cs. waikato. ac. nz/%7 Eml/weka/book. html In particular, pp. 8 -57 are based on the instructor slides for that book available at http: //books. elsevier. com/companions/9780120884070/ (chapters 1 -4): http: //books. elsevier. com/companions/9780120884070/revisionnotes/01~PDFs/chapter 1. pdf (and. . . chapter 2. pdf, chapter 3. pdf, chapter 4. pdf) or http: //books. elsevier. com/companions/9780120884070/revisionnotes/02~ODP%20 Files/chapter 1. odp (and. . . chapter 2. odp, chapter 3. odp, chapter 4. odp) n The MRDM part is based on Džeroski, S. (2003). Multi-Relational Data Mining: An Introduction. SIGKDD Explorations 5(1): 1 -16. http: //www. cs. wisc. edu/EDAM/Dzeroski. pdf Berendt: Advanced databases, winter term 2007/08, http: //www. cs. kuleuven. be/~berendt/teaching/2007 w/adb/