CS 480680 Intro to ML Lecture 06 Bagging

CS 480/680: Intro to ML Lecture 06: Bagging and Boosting 1 09/25/18 Yao-Liang Yu

Announcement • A 2 on course webpage • Need to submit on Kaggle • Do not cheat! • ICLR reproducibility challenge • https: //reproducibilitychallenge. github. io/iclr_2019/ • Final exam: Dec 19, 4 -6: 30 pm 2 09/25/18 Yao-Liang Yu

Decision Stump Recalled Eye features low No high Yes • Seems to be overly weak… • Let’s make it strong! 3 09/25/18 Yao-Liang Yu

More Generally • Which algorithm to use for my problem? • Cheap answers • Deep learning, but then which architecture? • I don’t know; whatever my boss tells me • Whatever I can find in scikit-learn • I try many and pick the “best” • Why don’t we combine a few algs? But how? 4 09/25/18 Yao-Liang Yu

The Power of Aggregation • Train ht on data set Dt, say each with accuracy • Assuming Dt are independent, hence ht are independent • Predict with the majority label among (2 T+1) ht’s iid ~ Bernoulli(p) • What is the accuracy? 5 09/25/18 Yao-Liang Yu 1

Bootstrap Aggregating ( B r e i m a n ’ 9 6 ) • Can’t afford to have many independent training sets • Bootstrapping! 1 2 sample with replacement … 1 1 3 4 4 h 1 4 3 n 1 n 5 … … sample with replacement 2 2 2 4 5 … … e. g. majority vote 6 09/25/18 Yao-Liang Yu n-1 h. T n n 1 n

Bagging for Regression • Simply average the outputs of ht, each trained on a bootstrapped training set • With T independent ht, averaging would reduce variance by a factor of T 7 09/25/18 Yao-Liang Yu

When Bagging Works • Bagging is essentially averaging • Beneficial if base classifiers have high variance (instable); e. g. , performance changes a lot if training set is slightly perturbed • Like decision trees but not k-NNs 8 09/25/18 Yao-Liang Yu

Randomizing Output ( B r e i m a n ’ 0 0 ) • For regression, add small Gaussian noise to each yi (leaving xi untouched) • Train many ht and average their outputs • For classification • Use one-hot encoding and reduce to regression • Randomly flip a small proportion of labels in training set • Train many ht and majority vote 9 09/25/18 Yao-Liang Yu

Random Forest ( B r e i m a n ’ 0 1 ) • 10 09/25/18 Yao-Liang Yu

Leo Breiman (1928 -2005) 11 09/25/18 Yao-Liang Yu

Boosting • Given many classifiers ht, each slightly better than random guessing • Is it possible to construct a classifier with nearly optimal accuracy? • Yes! First shown by Schapire (1990) 12 09/25/18 Yao-Liang Yu

The Power of Majority (Freund, 1995) (Schapire, 1990) 13 09/25/18 Yao-Liang Yu

Hedging ( F r e u n d & S c h a p i r e ’ 9 7 ) 14 09/25/18 Yao-Liang Yu

What Guarantee? # of rounds # of experts total loss of best expert your total loss goes to 0 as T choose beta appropriately 15 09/25/18 Yao-Liang Yu

Adaptive Boost ( F r e u n d & S c h a p i r e ’ 9 7 ) (Schapire & Singer, 1999) 16 09/25/18 Yao-Liang Yu

Look Closely can optimize y = 1 or 0 expected error of h adaptive h(x) closer to y, bigger exponent, discount 17 09/25/18 Yao-Liang Yu

Does It Work? 18 09/25/18 Yao-Liang Yu

Exponential decay of … training error! slightly smaller tha # of weak classifiers • 19 09/25/18 Yao-Liang Yu

Will Adaboost Overfit? margin: y h(x) 20 09/25/18 Yao-Liang Yu

Seriously? ( G r o v e & S c h u u r m a n s ’ 9 8 ; B r e i m a n ’ 9 9 ) 21 09/25/18 Yao-Liang Yu

Pros and Cons • “Straightforward” way to boost performance • Flexible with any base classifier • Less interpretable • Longer training time • hard to parallelize (in contrast to bagging) 22 09/25/18 Yao-Liang Yu

Extensions • Logit. Boost • Grad. Boost • L 2 Boost • … you name it • Multi-class • Regression • Ranking 23 09/25/18 Yao-Liang Yu

Face Detection ( V i o l a & J o n e s ’ 0 1 ) • Each detection window results in ~160 k features • Speed is crucial for real-time detection! 24 09/25/18 Yao-Liang Yu

Cascading 38 layers with ~6 k features selected 25 09/25/18 Yao-Liang Yu

Examples 26 09/25/18 Yao-Liang Yu

Asymmetry ( V i o l a & J o n e s ’ 0 2 ) • Too many non-faces vs few faces • Trivial to achieve small false positives by sacrificing true positives confusion matrix • Asymmetric loss y=1 y = -1 27 09/25/18 Yao-Liang Yu 0 0

SATzilla ( X u e t a l. , 2 0 0 8 ) 28 09/25/18 Yao-Liang Yu

Questions? 29 09/25/18 Yao-Liang Yu