Breeding Decision Trees Using Evolutionary Techniques Papagelis Athanasios

Breeding Decision Trees Using Evolutionary Techniques Papagelis Athanasios - Kalles Dimitrios Computer Technology Institute & AHEAD RM

Introduction We use GAs to evolve simple and accurate binary decision trees Simple genetic operators over tree structures Experiments with UCI datasets n n very good size competitive accuracy results Experiments with synthetic datasets n Superior accuracy results

Current tree induction algorithms…. . Use greedy heuristics n n To guide search during tree building To prune the resulting trees Fast implementations Accurate results on widely used benchmark datasets (like UCI datasets) Optimal results ? n No Good for real world problems? n There are not many real world datasets available for research.

More on greedy heuristics They can quickly guide us to desired solutions On the other hand they can substantially deviate from optimal WHY? n n They are very strict Which means that they are VERY GOOD just for a limited problem space

Why GAs should work ? GAs are not n Hill climbers w Blind on complex search spaces n Exhaustive searchers w Extremely expensive They are … n Beam searchers w They balance between time needed and space searched Application on bigger problem space n n Good results for much more problems No need to tune or derive new algorithms

Another way to see it. . Biases Preference bias n Characteristics of output w We should choose about it n e. g small trees Procedural bias n How we will search? w We should not choose about it w Unfortunately we have to: n n Greedy heuristics make strong hypotheses about search space GAs make weak hypotheses about search space

The real world question… Are there datasets where hill-climbing techniques are really inadequate ? n e. g unnecessarily big – misguiding output Yes there are… n Conditionally dependent attributes w e. g XOR n Irrelevant attributes w Many solutions that use GAs as a preprocessor so as to select adequate attributes Direct genetic search can be proven more efficient for those datasets

The proposed solution Select the desired decision tree characteristics (e. g small size) Adopt a decision tree representation with appropriate genetic operators Create an appropriate fitness function Produce a representative initial population Evolve for as long as you wish!

Initialization procedure Population of minimum decision trees n Simple and fast Choose a random value as test value Choose two random classes as leaves A=2 Class=1 Class=2

Genetic operators

Payoff function Balance between accuracy and size set x depending on the desired output characteristics. • Small Trees ? x near 1 • Emphasis on accuracy ? x grows big

Advanced System Characteristics Scalled payoff function (Goldberg, 1989) Alternative crossovers n Evolution towards fit subtrees w Accurate subtrees had less chance to be used for crossover or mutation. Limited Error Fitness (LEF) Ross, 1997) n (Gathercole & significant CPU timesavings insignificant accuracy loses and

Second Layer GA Test the effectiveness of all those components n coded information about the mutation/crossover rates and different heuristics as well as a number of other optimizing parameters Most recurring results: n n mutation rate 0. 005 crossover rate 0. 93 use a crowding avoidance technique Alternative crossover/mutation techniques did not produce better results than basic crossover/mutation

Search space / Induction costs 10 leaves, 6 values, 2 classes n Search space >50, 173, 704, 142, 848 (HUGE!) Greedy feature selection n O(ak) a=attributes, k=instances (Quinlan 1986) O(a 2 k 2) one level lookahead (Murthy and Salzberg, 1995) O(adkd) for d-1 levels of lookahead Proposed heuristic n O(gen* k 2*a). Extended heuristic n O(gen*k*a)

How it works? An example (a) An artificial dataset with eight rules (26 possible value, three classes) First two activation-rules as below: w (15. 0 %) c 1 A=(a or b or t) & B=(a or h or q or x) w (14. 0%) c 1 B=(f or l or s or w) & C=(c or e or f or k) n Huge Search Space !!!

How it works? An example (b)

Illustration of greedy heuristics problem An example dataset (XOR over A 1&A 2) A 1 T T F F T T A 2 F F T T A 3 T F F T F Class T T F F F T

C 4. 5 result tree A 3=t A 1=t A 2=f A 2=t f t t A 2=f A 2=t f f Totally unacceptable!!! t t f

More experiments towards this direction Name Attrib. Class Function Noise Instanc. Random Attributes Xor 1 10 (A 1 xor A 2) or (A 3 xor A 4) No 100 6 Xor 2 10 (A 1 xor A 2) xor (A 3 xor A 4) No 100 6 Xor 3 10 (A 1 xor A 2) or (A 3 and A 4) or (A 5 and A 6) 10% class error 100 4 Par 1 10 Three attributes parity problem No 100 7 Par 2 10 Four attributes parity problem No 100 6

Results for artificial datasets C 4. 5 GATree Xor 1 67± 12. 04 100± 0 Xor 2 53± 18. 57 90± 17. 32 Xor 3 79± 6. 52 78± 8. 37 Par 1 70± 24, 49 100± 0 Par 2 63± 6. 71 85± 7. 91

Results for UCI datasets

C 4. 5 / One. R deficiencies Similar preference biases n Accurate, small decision trees w This is acceptable Not optimized procedural biases n Emphasis on accuracy (C 4. 5) w Not optimized tree’s size n Emphasis on size (One. R) w Trivial search policy n Pruning as a greedy heuristic has similar disadvantages

Future work Minimize evolution time n n n crossover/mutation operators change the tree from a node downwards we can classify only the instances that belong to the changed-node’s subtree. But we need to maintain more node statistics Average needed re-classification

Future work (2) Choose the output class using a majority vote over the produced tree forest (experts voting) Pruning is a greedy heuristic n A GA’s pruning?