SLIQ A Fast Scalable Classifier for Data Mining

SLIQ: A Fast Scalable Classifier for Data Mining Manish Mehta, Rakesh Agrawal, Jorma Rissanen 1996. Presentation by: Vladan Radosavljevic

Outline n n n Introduction Motivation SLIQ Algorithm n n n Building tree Pruning Example Results Conclusion

Introduction n Most of the classification algorithms are designed for memory resident data – limited suitability for mining large datasets Solution – build a scalable classifier - SLIQ – Supervised Learning in Quest, Quest was the data mining project at the IBM

Motivation n Recall (ID 3, C 4. 5, CART):

Motivation n n NON SCALABLE DECISION TREES: The complexity lies in determining the best split for each attribute The cost of evaluating splits for numerical attributes is dominated by the cost of sorting values at each node The cost of evaluating splits for categorical attributes is dominated by the cost of searching for the best subset Pruning n n crossvalidation inapplicable for large datasets divide data in two parts - training and test set - sizes, distribution? ? ?

Motivation n n Improve scalability of tree classifiers Previous proposals: n n n Sampling data at each node Discretization of numerical attributes Partitioning input data and build tree for each partition All methods achieve low accuracy! SLIQ – improve learning time without loss in accuracy!

SLIQ n Key features: n n n Tree classifier, handling both numerical and categorical attributes Presort numerical attributes before tree has been built Breadth first growing strategy Goodness test – Gini index Inexpensive tree pruning algorithm based on Minimum Description Length (MDL)

SLIQ - Algorithm n n n Eliminate the need to sort the data at each node Create sorted list for each numerical attribute Create class list

SLIQ - Algorithm n Example:

SLIQ - Algorithm n Split evaluation:

SLIQ - Algorithm n Example:

SLIQ - Algorithm n Update class list:

SLIQ - Algorithm n Example:

SLIQ - Algorithm n n n For large-cardinality categorical attributes (determined based on threshold) the best split is computed in greedy way, otherwise all possible splits are evaluated When node becomes pure stop splitting it, then condense attribute lists by discarding examples that correspond to the pure node SLIQ is able to scale for large datasets with no loss in accuracy – the splits evaluated with or without pre-sorting are identical

SLIQ - Pruning n n Post pruning algorithm based on Minimum Description Length principle Find a model that minimizes: Cost(M, D) = Cost(D|M) + Cost(M) - cost of the model Cost(D|M) - cost of encoding the data D if model M is given

SLIQ - Pruning n n Cost of the data: classification error Cost of the model: n n Encoding the tree: number of bits Encoding the splits: n numerical attribute - constant (empirically 1) categorical attribute - depends on cardinality The MDL pruning evaluate the code length at each node to determine whether to prune or both child or leave the node intact

SLIQ - pruning n Three pruning strategies: n n n Full – pruning both children and convert node to the leaf Partial – prune into the leaf or prune the left child or prune the right child or leave node intact Hybrid – apply Full method and then partial (prune left, prune right or leave intact)

Results n SLIQ was tested on the datasets:

Results n Pruning strategy comparison:

Results n Accuracy:

Results n Scalability:

Conclusion n SLIQ demonstrates to be a fast, low-cost and scalable classifier that builds accurate trees Based on empirical test which compared SLIQ to other tree based classifiers, SLIQ achieves a comparable accuracy while producing smaller decision trees Scalability? ? ? Memory problem when increasing number of attributes or number of classes

References [1] M. Mehta, R. Agrawal and J. Rissanen, "SLIQ: A Fast Scalable Classifier for Data Mining", in Proceedings of the 5 th International Conference on Extending Database Technology, Avignon, France, Mar. 1996.

THANK YOU!