The Bag of Little Bootstraps Michael I Jordan

The Bag of Little Bootstraps Michael I. Jordan April 26, 2017 with Ariel Kleiner, Purna Sarkar and Ameet Talwalkar 1

Setup Observe data X 1, . . . , Xn ^ Form a “parameter” estimate qn = q(X 1, . . . , Xn) Want to compute an assessment x of the quality of ^ our estimate qn (e. g. , a confidence region) 2

The Unachievable Frequentist Ideally, we would ① Observe many independent datasets of size n. ^ ② Compute qn on each. ^ ③ Compute x based on these multiple realizations of qn. 3

The Unachievable Frequentist Ideally, we would ① Observe many independent datasets of size n. ^ ② Compute qn on each. ^ ③ Compute x based on these multiple realizations of qn. But, we only observe one dataset of size n. 4

Sampling 5

Approximation 6

Pretend The Sample Is The Population 7

The Bootstrap (Efron, 1979) • Plug in the empirical distribution in the place of the population distribution in computing the risk 8

The Bootstrap: Computational Issues • Seemingly a wonderful match to modern parallel and distributed computing platforms • But the expected number of distinct points in a bootstrap resample is ~ 0. 632 n – e. g. , if original dataset has size 1 TB, then expect resample to have size ~ 632 GB • Can’t feasibly send resampled datasets of this size to distributed servers • Even if one could, can’t compute the estimate locally on datasets this large 9

Subsampling (Politis, Romano & Wolf, 1999) n 10

Subsampling n b 11

Subsampling • There are many subsets of size b < n • Choose some sample of them and apply the estimator to each • This yields fluctuations of the estimate, and thus error bars • But a key issue arises: the fact that b < n means that the error bars will be on the wrong scale (they’ll be too large) • Need to analytically correct the error bars 12

Subsampling Summary of algorithm: ① Repeatedly subsample b < n points without replacement from the original dataset of size n ② Compute q*b on each subsample ③ Compute x based on these multiple realizations of q*b ④ Analytically correct to produce final estimate of x for qn The need for analytical correction makes subsampling less automatic than the bootstrap Still, much more favorable computational profile than bootstrap Let’s try it out in practice… 13

Empirical Results: Bootstrap and Subsampling • Multivariate linear regression with d = 100 and n = 50, 000 on synthetic data. • x coordinates sampled independently from Student. T(3). • y = w. Tx + e, where w in Rd is a fixed weight vector and e is Gaussian noise. • Estimate qn = wn in Rd via least squares. • Compute a marginal confidence interval for each component of wn and assess accuracy via relative mean (across components) absolute deviation from true confidence interval size. • For subsampling, use b(n) = ng for various values of g. • Similar results obtained with Normal and Gamma data generating distributions, as well as if estimate a misspecified model. 14

Empirical Results: Bootstrap and Subsampling 15

Bag of Little Bootstraps • I’ll now present a new procedure that combines the bootstrap and subsampling, and gets the best of both worlds 16

Bag of Little Bootstraps • I’ll now discuss a new procedure that combines the bootstrap and subsampling, and gets the best of both worlds • It works with small subsets of the data, like subsampling, and thus is appropriate for distributed computing platforms 17

Bag of Little Bootstraps • I’ll now present a new procedure that combines the bootstrap and subsampling, and gets the best of both worlds • It works with small subsets of the data, like subsampling, and thus is appropriate for distributed computing platforms • But, like the bootstrap, it doesn’t require analytical rescaling 18

Towards the Bag of Little Bootstraps n b 19

Towards the Bag of Little Bootstraps b 20

Approximation 21

Pretend the Subsample is the Population 22

Pretend the Subsample is the Population • And bootstrap the subsample! • This means resampling n times with replacement, not b times as in subsampling 23

The Bag of Little Bootstraps (BLB) • The subsample contains only b points, and so the resulting empirical distribution has its support on b points • But we can (and should!) resample it with replacement n times, not b times • Doing this repeatedly for a given subsample gives bootstrap confidence intervals on the right scale---no analytical rescaling is necessary! • Now do this (in parallel) for multiple subsamples and combine the results (e. g. , by averaging) 24

The Bag of Little Bootstraps (BLB) 25

Bag of Little Bootstraps (BLB) Computational Considerations A key point: • Resources required to compute q generally scale in number of distinct data points • This is true of many commonly used estimation algorithms (e. g. , SVM, logistic regression, linear regression, kernel methods, general M-estimators, etc. ) • Use weighted representation of resampled datasets to avoid physical data replication Example: if original dataset has size 1 TB with each data point 1 MB, and we take b(n) = n 0. 6, then expect • subsampled datasets to have size ~ 4 GB • resampled datasets to have size ~ 4 GB (in contrast, bootstrap resamples have size ~ 632 GB) 26

Empirical Results: Bag of Little Bootstraps (BLB) 27

Empirical Results: Bag of Little Bootstraps (BLB) 28

BLB: Theoretical Results BLB is asymptotically consistent and higherorder correct (like the bootstrap), under essentially the same conditions that have been used in prior analysis of the bootstrap. Theorem (asymptotic consistency): Under standard assumptions (particularly that q is Hadamard differentiable and x is continuous), the output of BLB converges to the population value of x as n, b approach ∞. 29

BLB: Theoretical Results Higher-Order Correctness Assume: • q is a studentized statistic. • x(Qn(P)), the population value of x for qn, can be written as where the pk are polynomials in population moments. • The empirical version of x based on resamples of size n from a single subsample of size b can also be written as where the are polynomials in the empirical moments of subsample j. • b ≤ n and 30

BLB: Theoretical Results Higher-Order Correctness Then: 31

Knowing When You’re Wrong: Building Fast and Reliable Approximate Query Processing Systems Sameer Agarwal, Michael Jordan, Ariel Kleiner, Samuel Madden, Henry Milner, Barzan Mozafari, Ion Stoica, & Ameet Talwalkar UC Berkeley MIT 32

Approximate Query Processing A±ε SELECT foo (*) FROM TABLE WITHIN 2 Query Interface Error Estimation Query Execution Data Subsets/ Synopsis 33

Problem: Error Estimation is Unreliable ed s Clo rms Fo ap r t ts o Bo 69, 428 Hive Aggregate Queries (Feb ’ 13) 34

Problem: Error Estimation is Unreliable ed s Clo rms Fo Inapplicabl e ap r t ts o Bo 69, 428 Hive Aggregate Queries (Feb ’ 13) 35

Problem: Error Estimation is Unreliable ed s Clo rms Fo Bo o ap r t ts Correct Estimation 69, 428 Hive Aggregate Queries (Feb ’ 13) 36

Problem: Error Estimation is Unreliable ed s Clo rms Fo Bo o ap r t ts Under Estimation 69, 428 Hive Aggregate Queries (Feb ’ 13) 37

Problem: Error Estimation is Unreliable ed s Clo rms Fo ap r t ts o Bo Over Estimatio n 69, 428 Hive Aggregate Queries (Feb ’ 13) 38

Problem: Error Estimation is Unreliable ed s Clo rms Fo ap r t ts o Bo 18, 321 Hive Aggregate Queries (Feb ‘ 13) 39

Problem: Error Estimation is Unreliable ed s Clo rms Fo ap r t ts Inapplicabl e o Bo 18, 321 Hive Aggregate Queries (Feb ‘ 13) 40

Problem: Error Estimation is Unreliable ed s Clo rms Fo Bo o ap r t ts Correct Estimation 18, 321 Hive Aggregate Queries (Feb ‘ 13) 41

Problem: Error Estimation is Unreliable ed s Clo rms Fo Bo t s t o rap Under Estimati on 18, 321 Hive Aggregate Queries (Feb ‘ 13) 42

Problem: Error Estimation is Unreliable ed s Clo rms Fo ap r t ts o Bo Over Estimatio n 18, 321 Hive Aggregate Queries (Feb ‘ 13) 43

Problem: Error Estimation is Unreliable 96% ed s Clo rms Fo Bo t s t o rap Over Estimates 96. 8% 69, 428 Hive Aggregate Queries (Feb ’ 13) 44

Problem: Error Estimation is Unreliable 97% ed s Clo rms Fo Bo t s t o rap Over Estimates 97. 2% 18, 321 Hive Aggregate Queries (Feb ‘ 13) 45

Response Time (s) Overall Query Execution 46

Response Time (s) Overall Query Execution Error Estimation Overhead 47

Overall Query Execution Response Time (s) Diagnostics Overhead 48

Summary: Query Execution A ±ε SELECT foo (*) FROM TABLE WITHIN 2 Y/N Query Interface Error Estimation Query Exection Data Storage 49