RIPPER Fast Effective Rule Induction Machine Learning 2003

RIPPER Fast Effective Rule Induction Machine Learning 2003 Merlin Holzapfel & Martin Schmidt Mholzapf@uos. de Martisch@uos. de

Rule Sets - advantages easy to understand n usually better than decision Tree learners n representable in first order logic n – > easy to implement in Prolog n prior knowledge can be added

Rule Sets - disadvantages n scale poorly with training set size n problems with noisy data – likely in real-world data n goal: – develop rule learner that is efficient on noisy data – competitive with C 4. 5 / C 4. 5 rules

Problem with Overfitting n overfitting also handles noisy cases n underfitting is too general n solution pruning: – reduced error pruning (REP) – post pruning – pre pruning

Post Pruning (C 4. 5) n overfit & simplify – construct tree that overfits – convert tree to rules – prune every rule separately – sort rules according accuracy – consider order when classifying n bottom - up

Pre pruning n some examples are ignored during concept generation n final concept does not classify all training data correctly n can be implemented in form of stopping criteria

Reduced Error Pruning n seperate and conquer – split data in training and validation set – construct overfitting tree – until pruning reduces accuracy • evaluate impact on validation set of pruning a rule • remove rule so it improves accuracy most

Time Complexity n REP has a time complexity of O(n 4) – initial phase of overfitting alone has a complexity of O(n²) n alternative concept Grow: – faster in benchmarks – time complexity still O(n 4) with noisy data

Incremental Reduced Error Pruning - IREP n by Fürnkranz & Widmer (1994) n competitive error rates n faster than REP and Grow

How IREP Works n iterative application of REP n random split of sets bad split has negative influence (but not as bad as with REP) n immediately pruning after a rule is grown (top-down approach) no overfitting

Cohens IREP Implementation n build rules until new rule results in too large error rate – divide data (randomly) into growing set(2/3) and pruning set(1/3) – grow rule from growing set – immediately prune rule • Delete final sequence of conditions – delete condition that maximizes function v until no deletion improves value of v – add pruned rule to ruleset – delete every example covered by rule (p/n)

Cohens IREP - Algorithm

IREP and Multiple Classes n order classes according to increasing prevalence (C 1, . . , Ck) – find rule set to separate C 1 from other classes IREP(Pos. Data=C 1, Neg. Data=C 2, . . . , Ck) – remove all instances learned by rule set – find rule set to separate C 2 from C 3, . . . , Ck. . . – Ck remains as default class

IREP and Missing Attributes n handle missing attributes: – for all tests involving A • if attribute A of an instance is missing test fails

Differences Cohen <> Original n pruning: final sequence <> single final condition n stopping condition: error rate 50% <> accuracy(rule) < accuracy(empty rule) n application: missing attributes, numerical variables, multiple classes <> two-class problems

Time Complexity IREP: O(m log² m), m = number of examples (fixed number of classification noise)

37 Benchmark Problems

Generalization Performance n IREP performs worse on benchmark problems than C 4. 5 rules n won-lost-tie ratio: 11 -23 -3 n error ratio – 1. 13 excluding mushroom – 1. 52 including mushroom

Improving IREP n three modifications: – alternative metric in pruning phase – new stopping heuristics for rule adding – post pruning of whole rule set (non-incremental pruning)

the Rule-Value Metric n old metric not intuitive R 1: p 1 = 2000, n 1 = 1000 R 2: p 1 = 1000, n 1 = 1 metric preferes R 1 (fixed P, N) leads to occasional failure to converge n new metric (IREP*)

Stopping Condition 50%-heuristics often stops too soon with moderate sized examples n sensitive to the ‘small disjunct problem‘ n solution: n – after a rule is added, the total description length of rule set and missclassifications (DL=C+E) – If DL is d bits larger then the smallest length so far stop (min(DL)+d<DLcurrent) – d = 64 in Cohen‘s implementation MDL (Minimal Description Length) heuristics

IREP* n IREP* is IREP, improved by the new rule-value metric and the new stopping condition n 28 -8 -1 against IREP n 16 -21 -0 against C 4. 5 rules error ratio 1. 06 (IREP 1. 13) respectively 1. 04 (1. 52) including mushrooms

Rule Optimization n post prunes rules produced by IREP* – The rules are considered in turn – for each rule R, two alternatives are constructed • Ri‘ new rule • Ri‘‘ based on Ri – final rule is chosen according to MDL

RIPPER IREP* is used to obtain a rule set 2. rule optimization takes place 3. IREP* is used to cover remaining positive examples 1. Repeated Incremental Pruning to Produce Error Reduction

RIPPERk n apply steps 2 and 3 k times

RIPPER Performance n 28 -7 -2 against IREP*

Error Rates RIPPER obviously is competitive

Efficency of RIPPERk n modifications do not change complexity

Reasons for Efficiency n find model with IREP* and then improve – effiecient first model with right size – optimization takes linear time n C 4. 5 has expensive optimization improvement process – to large initial model n RIPPER is especially more efficient on large noisy datasets

Conclusions n IREP is efficient rule learner for large noisy datasets but performs worse than C 4. 5 n IREP improved to IREP* n IREP* improved to RIPPER n k iterated RIPPER is RIPPERk n RIPPERk more efficient and performs better than C 4. 5

References n Fast Effective Rule Induction William W. Cohen [1995] n Incremental Reduced Error Pruning J. Fürnkranz & G. Widmer [1994] n Efficient Pruning Methods William W. Cohen [1993]