A Statisticians Games Bootstrap Bagging and Boosting Yaochu

A Statistician’s Games*: Bootstrap, Bagging and Boosting Yaochu Jin Future Technology Research Honda R&D Europe (Germany) March 21, 2000 * Please refer to “Game theory, on-line prediction and boosting” by Y. Freund and R. Schapire, Proceedings of 9 th Conference on Computational Learning Theory.

Bootstrap -- Problem Description • The bootstrap was introduced as a general method for assessing the statistical accuracy of an estimator Given data: x = (x 1, . . . , xn) Have an estimator: = s(x) ? How to assess the accuracy of

Bootstrap -- the Idea Bootstrap estimate of the standard error:

Bootstrap -- Pros and Cons Easy to implement Need a large number of independent bootstrap samples (B>=1000) • Uncertainty of the estimate 1) Jackknife-after-Bootstrap(JAB) 2) Weighted JAB

Bagging is Not Related to Begging • Using bootstrap techniques to improve the estimator • Bagging -- Bootstrap aggregating

Bagging -- the Idea The final estimate: = ( 1 + 2 +. . . + B )/B

Bagging -- Pros and Cons The estimator can be significantly improved if the learning algorithm is unstable Reduce the variance, bias unchanged Degrade the performance of stable procedures

Adaptive Bagging Reduce both variance and bias

Boosting • To boost a “weak” learning algorithm into a “strong” learning algorithm • A week learning algorithm can be inaccurate rules of thumb that is slightly better than random guess

Ada. Boost Update distribution Choose weight t = 1/2 ln(1 - t / t) Calculate error t Initialize Distribution D 1(i) = 1/n The final estimate: = ( 1 1 + 2 2 +. . . + n B )/B

Ada. Boost -- Pros and Cons Reduce both variance and bias Theoretical guarantee (maximizes the likelihood) Easy to implement (compared to Bayesian methods) Relation to Support Vector Machines Need large number of estimators (B>=1000) Sensitive to noise

Further Information on B 3 http: //www-stat. stanford. edu/~tibs/ ftp: //ftp. stat. berkeley. edu/pub/users/breiman/ http: //www. research. att. com/~yoav/ http: //www. research. att. com/~schapire/