Boosting Regression Dan Baker February 19 2007 What

Boosting Regression Dan Baker February 19, 2007

What is Boosting? Committee machines Meta-learner “rules of thumb” “Wisdom of Crowds” Improved Performance Boosting’s cousin Bagging Boosting for improved accuracy Bagging for decreased variance

No really, what is boosting? Incremental Stages Misclassified Instances of Last Stage Final Estimate = combined hypothesis “Boosts” a weak learner to a strong learner Weak learner - slightly better than guessing

Boosting Approaches Boosting by Filtering Boosting by Sub-Sampling Requires a large training dataset Fixed Training Size Examples resampled at each stage Boosting by Reweighting Uses all training data Useful if learner can handle weighting

Adaboost Adaptive Boosting by Freund and Schapire, 1996 Popular boosting algorithm Many variants Pseudocode: http: //en. wikipedia. org/wiki/Ada. Boost

Adaboost Variants Adaboost Two-class Adaboost. M 1 Multi-class Adaboost. M 2 Multi-class Adaboost. R Real-valued* Adaboost. RT Real-valued* * Both are for regression problems but both also work by casting the regression problem as a classification problem.

Adaboost for Regression Most methods cast as classification But some don’t Zemel and Pitassi. A gradient-based* boosting algorithm for regression problems. 2001 * Adaptive Boosting Methods for Classification Problems have been derived as Gradient Descent Algorithms.

Committee Machines & COCOMO Learners Bagging by Subsampling = No Results Boosting a COCOMO Learner Breaking the rules – not a weak learner LSR in COCOMO not easily adaptable to weighted instances Problem: Not Classification Opt 1: Cast as classification Opt 2: Change algorithm (mo’ math)

An Interesting Study. . . Why does boosting work? Focuses on the misclassified instances of the prior round Some research* has shown that using random weights rivals performance! *This research may not generalize to boosting in general. H. Yoshii. A big mistake concerning boosting. In Information. Based Induction Sciences IBIS 2001, pages 285 -290, 2001.

Research Direction Boosting by subsampling Boosting by oversampling* Boosting by reweighting * Why doesn’t this show up in research? Cast as a Classification Problem Don’t cast it Try different: Weighting methods Combined Hypothesis methods

Questions?