1 CHERRYPICK ADAPTIVELY UNEARTHING THE BEST CLOUD CONFIGURATIONS

1 CHERRYPICK: ADAPTIVELY UNEARTHING THE BEST CLOUD CONFIGURATIONS FOR BIG DATA ANALYTICS By Omid Alipourfard, Hongqiang Harry Liu, Jianshu Chen, Shivaram Venkataraman, Minlan Yu, and Ming Zhang NSDI, 2017 Presented by Ari Ball-Burack Tuesday, 19 November, 2019

2 Context • Big data analytics applications are run on clusters of virtual machines (VMs) • Choosing a cloud configuration (#instances, CPU, memory, disk, network) is important (recurring jobs) • But it can be difficult and costly • Relationship between configuration, running time, and cost is complex and nonlinear • Brute force is expensive — >40 AWS & Azure instance types, 18 Google types + customizable memory & #cores, ∞ cluster sizes

3 Related Work • Agarwal et al. & Ferguson et al. report that up to 40% of analytics jobs are recurring • Can model application performance, but hard to adapt • Application structure is required for accurate modeling • Static search has high overhead • Coordinate descent (one dimension at a time) • Suboptimal in the case of local minima • Venkataraman et al. : Ernest, 2016 • Trains a performance model for machine learning applications • Doesn’t work well for other applications e. g. SQL queries

4 Contributions • “…we just need a model that is accurate enough for us to separate the best configuration from the rest. ” • Uses Bayesian Optimization to minimize “cost” subject to a running time constraint • Non-parametric (no pre-defined cost function format), near-optimal solution in few samples, and uncertainty tolerance

5 Bayesian Optimization Specifics • Cost is a running time & price product • Acquisition function: maximize Expected Improvement • Prior function: Gaussian Process • Start with I (= 3) initial samples • Stop when EI is less than thresh (= 10%) and at least N (= 6) configurations have been explored • Built on top of Spearmint in Python

6 Bayesian Optimization Example

7 Experimental Results • Evaluated using 5 types of analytics applications on 66 (AWS EC 2) cloud configurations • Compared to exhaustive search, coordinate descent, random search with a budget, and Ernest • Cherry. Pick finds exact optimum 45 -90% of the time, within 5% of optimum at median — better with lower EI • Exhaustive search requires 5 -9. 5 times time, 6 -9 times cost • More stable (slightly better median, closer-to-optimal tail) than coordinate descent • Better & more stable than random search w/ budget • At 4 x search cost, RSw. B performs similarly • Similar run cost to Ernest but with lower search cost

8 Experimental Results

9 Evaluation • Strengths • Clear improvement over existing systems (Ernest) and more naïve approaches • Authors are thorough in descriptions of drawbacks (dependence on representative workloads, acquisition function slowdown for larger search spaces) and directions for future research (Monte Carlo a. f. , non- Gaussian Process prior function)

10 Evaluation • Critiques • “ 45 -90% chance” of finding optimal configuration is vague! • Representative workload problem — acknowledged but still difficult (requires human intuition, sometimes purposeful inflation) • “We admit that a better prior [than Gaussian Process] might be found given some domain knowledge of specific applications” • Why not make this an optional parameter, as EI value is? • For recurring jobs, might a 6 -9 times search cost be worth a 5% reduction in running cost?

11 Questions?