VGr ADS Tools Activities Chuck Koelbel VGr ADS

VGr. ADS Tools Activities Chuck Koelbel VGr. ADS Workshop, February 23, 2006

Tools: Where We Are • Achieved — Initial workflow scheduling methods – “Anirban scheduler” [Rice, UH, UCSD, ISI] Supported by performance prediction, NWS — Initial fault tolerance implementations – FT-MPI [UTK] – Optimal checkpoint scheduling [UCSB] — Platform-independent application launch and optimization – LLVM, run-time reoptimization experiments [Rice] • Working On It — Virtual Grid scheduling methods — Building workflow DAGs

Ongoing Tools Thrusts • Scheduling methods — Most of rest of talk [Rice, UCSD, UCSB, UTK, ISI] — All based on pre-scheduling (aka off-line scheduling) of workflow (aka datflow, aka DAGs) using performance prediction • Performance prediction — Queue delay model [UCSB] • Other — Launching and reoptimization [Rice] — DAG construction [Rice]

Scheduling Methods • Two-level (choose VG, map onto it) — Richard Huang (UCSD), Anirban Mandal & Ryan Zhang (Rice) • Batch queue (include est. queue delay in cost model) — Anirban Mandal (Rice), Dan Nurmi (UCSB) • Cluster (assign block of tasks to cluster) — Anirban Mandal (Rice) • Provisioning (minimize reservation time + execution time) — Gurmeet Singh (ISI) • Robust (schedule to reduce sensitivity to variability) — Zhiao Shi (UTK)

Scheduling Comparison Objective function Costs Important Assumptions Two-level Scheduler + Makespan DAG nodes & edges; (Rice) Proc type Dedicated resources available for duration of application Batch queue Scheduler + Makespan + Queue delays DAG nodes & edges; Proc type, Queue delay Resources controlled by queues, but no relevant allocation limits Cluster Scheduler + Makespan DAG nodes & edges; Proc type Dedicated resources available for duration of application; homogeneous DAG nodes on each level Provisioning Reservation time and Sched quality Reservation, Schedule costs Resources controlled by provisioning mechanism Robustness subject to makespan DAG edges(? ) Shared resources resulting in runtime performance variability Min / Max

Results Huang - 2 -level scheduler, Montage DAG Mandal - cluster scheduler, EMAN DAG Maximize f Shi - robust scheduler, ? ? ? DAG

Tools Research Going Forward • Interface between vg. ES and schedulers — What capabilities can schedulers expect from vg. ES? — How can schedulers exploit this capability? — How can schedulers work around this capability? • Some interesting operating points — — vg. ES provisions VG / application takes what’s given vg. ES returns shared VG nodes / application adapts to perf variance vg. ES returns queued VG resources / application manages queues vg. ES provisions VG, monitors for additional resources / application starts immediately, adapts to changes

Tools Research Going Forward • Generating vg. DL request for 2 -level methods — Balance request complexity vs. difficulty scheduling onto VG VG 1 = Cluster. Of (node) [1: N] [Rank=Cluster. nodes] {node = [CPU=Opteron]} VG 2 = Cluster. Of (node) [1: N] [Rank=Cluster. nodes*node. clock] {node = [CPU=Opteron]} VG 3 = Cluster. Of (node) [1: N] [Rank=Perf. Model(Cluster. nodes, Cluster. bw, node. clock, node. mem)] {node = [CPU=Opteron]} — Automatic vg. DL generation from DAGs – Template-driven? Heuristic-driven? — Extended vg. DL capabilities – Global constraints (e. g. total # of nodes) – Temporal constraints (e. g. available within 60 min) – Probabalistic constraints (e. g. 95% likely to succeed)

Tools Research Going Forward • New scheduling criteria — Deadline scheduling — Economic scheduling — Real-time scheduling • New scheduling situations — Rescheduling – Adapting to new resources – Adapting to resource failures — Incremental scheduling – Managing dynamic applications – “Horizon scheduling” for limited-time predictions — Hybrid static / dynamic scheduling – Contingency scheduling – Static planning for dynamic optimizations

Backup Slides Beyond This Point

Two-Level Scheduling (Huang) • Target Application — Workflows represented by DAG • Performance Metrics — Application Turn-Around Time – Resource Selection – Scheduling Time – Application Makespan • Major Assumptions of the Scheduler — Resources are dedicated — Resources available for duration of application • Scheduling Algorithms (so far) — Greedy — Modified Critical Path

Experimental Setup • • • Use synthetic resource generator to generate 1000 clusters (33, 667 hosts) Execute one “simple” (greedy) and one “complex” (Modified Critical Path) scheduling heuristic Tests on Montage DAG Scheduling Resources Heuristic Complex Resource Universe Complex Top x percent Fastest Hosts Complex Appropriate Virtual Grid Simple Resource Universe Simple Top x percent Fastest Hosts Simple Appropriate Virtual Grid

Initial Results Original CCR • • • CCR = 0. 1 Two-phase scheduling necessary to avoid excessive scheduling time Appropriate virtual grids necessary for better performance Using more complex heuristic did not improve performance if you have the appropriate resource abstractions!

Batch Queue Scheduling (Mandal) • Make batch-queue predictions on-the-fly from the ``live’’ systems — New NWS functionality • Parameterize the performance models using the 95% upper bound on the median prediction as a prediction of delay — The performance models can take into account the amount of time needed to start a computation • Run a top-down (heuristic) scheduler to choose a resource set — Scheduler is smart enough to understand that the start-up delay can be amortized Joint work with Dan Nurmi and Rich Wolski

Top-Down Scheduling For each heuristic Until all components mapped Map available components to resources Select mapping with minimum makespan While all available components not mapped For each (component, resource) pair ECT(c, r) = rank(c, r) + EAT(r) End For each Run min-min, max-min and sufferage Store mapping End while Top-Down

Scheduling onto Batch-Queue Systems • Details: Modification of Top-Down scheduler — At every scheduling step, take into account the estimated time the job has to wait in the queue in the estimated completion time for the job [ECT(c, r) in the algorithm] — Keep track of the queue wait times for each cluster and the number of nodes that correspond to the queue wait time — With each mapping, update the estimated availability time [EAT in the algorithm] with the queue wait time, as required

Scheduling onto Batch-Queue Systems: Example Cluster 0 Input DAG R 0 Queue Wait Time [Cluster 0] = 20 # nodes for this wt. time = 1 Queue Wait Time [Cluster 1] = 10 # nodes for this wt. time = 2 T R 1 Cluster 1 R 2 R 3

Scheduling onto Batch-Queue Systems: Example Cluster 0 Input DAG R 0 Queue Wait Time [Cluster 0] = 20 # nodes for this wt. time = 1 Queue Wait Time [Cluster 1] = 10 # nodes for this wt. time = 2 T R 1 Cluster 1 R 2 R 3

Discussions • Experiments to evaluate EMAN scheduling with batch-queues — Control experiment – Schedule with and without queue-wait estimates, run application with the two schedules on Teragrid and compare turnaround times — Accuracy of the results - how close to actual • Other future issues — Predictive/opportunistic approach – Submit to queues even before data arrives with hope that data arrives by the time job moves to the front of the queue — Point-valued predictions of probabilistic systems are problematic – Need to schedule based on ranges or distributions — Probabilistic deadline scheduling

Cluster Scheduling (Mandal) • • • Motivation: Scheduler scaling problem for ‘large’ Grids Idea: Schedule directly onto clusters Input: — Workflow DAG with restricted structure - nodes at the same level do the same computation — Set of available Clusters (num. Nodes, arch, CPU speed etc. ) and intercluster network connectivity (latency, bandwidth) — Per-node performance models for each cluster • Output: — Mapping: for each level the number of instances mapped to each cluster • Objective: — Minimize makespan

Scheduling onto Clusters: Modeling • Abstract modeling of mapping problem for a DAG level — Given: – N instances – M clusters – r 1. . r. M nodes/cluster – t 1. . t. M - rank value per node per cluster (incorporates both computation and communication) — Aim: – To find a partition (n 1, n 2, … n. M) of N such that overall time is minimized with n 1+n 2+. . n. M = N • Analytical solution: • No ‘obvious’ solution because of discrete nature of problem

Scheduling onto Clusters • Iterative solution For each instance, i from 1 to N For each cluster, j from 1 to M Tentatively map i onto j Record makespan for each j by taking care of round(j) End For each Find cluster, p with minimum makespan increase Map i to p Update round(p), num. Mapped(p) End For each • Big picture: — Iterative assignment of tasks to clusters — DP approach O(#instances * #clusters)

Scheduling onto Clusters: Evaluation • Application — Representative DAGs from Montage and EMAN with varying widths — Known performance models • Simulation Platform — Resource Model: Synthetic cluster generator (Kee et al SC’ 04) — Network Model: BRITE to generate network topology, generate latency/bandwidth following a truncated normal distribution • Experiment — Varying number of clusters (nodes) – 250 to 1000 clusters (8. 5 K to 36 K nodes) — Ran three scheduling approaches – Heuristic (min-min/max-min/sufferage heuristics based) – Greedy (simple greedy heuristic based) – Simple (the Cluster level scheduler) — Compared turnaround time (Makespan + Scheduling Time)

Scheduling onto Clusters: Results Montage Application 103 node Montage DAG • 717 node Montage DAG Cluster level Scheduler (Simple) offers — Scalability - scales to ‘large’ Grids — Improved turnaround time — No significant degradation of application makespan quality

Scheduling onto Clusters: Results EMAN Application 171 node EMAN DAG • 666 node EMAN DAG Cluster level Scheduler (Simple) offers — Scalability - scales to ‘large’ Grids — Improved turnaround time — No significant degradation of application makespan quality

Robust Task Scheduling (Shi) • • Task scheduling: Assigning tasks of a meta-task (workflow-type application) to a set of resources, and achieving certain goals, e. g. minimizing the schedule length NP-complete, finding an optimal solution is either impossible or impractical. Heuristics (list scheduling, duplication, clustering) , optimization (Genetic algorithm, Simulated annealing etc. ) Previously we focused on list scheduling algorithm considering the case that processors has different capabilities.

Non-deterministic environment • • Actual resource environment is non-deterministic inherently due to resource sharing. Previously we used expected values of execution times of tasks, network speed. The optimal solution for task scheduling problem with expected values of resource characteristics is NOT optimal for the corresponding problem with non-deterministic values. We focus on variable execution time in this work.

Possible Solutions • Static scheduling — Overestimate the execution time to avoid exceeding the allotted use of machine at the expense of machine utilization — Compute the schedules for various scenarios and at run time adopt the one which fits the current status. — Find schedules more robust to variable execution time. • Dynamic scheduling — at each point of scheduling (when a task is ready to be executed), gather current resource information and compute a new schedule for unscheduled tasks

Robustness • Schedule delay — M 0(s): Makespan of the schedule s obtained with expect values (execution time) — M(s): Makespan of schedule s with real execution time • Robustness — Each realization of expect values gives different schedule delay.

Slack • • Slack of a task node is defined as follow: slack(ni) = makespan – [b_level(ni)+t_level(ni)] Slack is closely related to robustness. — large slack means a task node can tolerate large increase of execution time without increasing the makespan

Robustness and slack Disjunctive graph P 1 1 3 8 P 2 2 5 7 P 3 4 6 9 10 Disjunctive graph is used to calculate expected makespan and real makespan slack(ni) = makespan – [b_level(ni)+t_level(ni)]

Task execution time modeling • Least Time to Compute (LTC) matrix : {ltcij} — time to compute task i on processor j — generated from a single number with twice gamma distribution on 2 dimensions (machine, task) — different values of gamma parameters represent different heterogeneities of machine or task • Uncertainty level: {ulij} — expected actual time to compute / least time to compute — generated from a single number with twice gamma distribution on 2 dimensions (machine, task) • Actual computation time: actij = ltcij * ulij

Genetic Algorithm • • • 1. [Start] Generate initial population of n chromosomes (suitable solutions for the problem) 2. [Fitness] Evaluate the fitness f(x) of each chromosome x in the population 3. [New population] Create a new population by repeating following steps until the new population is complete — [Selection] Select two parent chromosomes from a population according to their fitness — [Crossover] With a crossover probability cross over the parents to form new offspring (children). If no crossover was performed, offspring is the exact copy of parents. — [Mutation] With a mutation probability mutate new offspring at each locus (position in chromosome). — [Accepting] Place new offspring in the new population 4. [Replace] Use new generated population for a further run of the algorithm 5. [Test] If the end condition is satisfied, stop, and return the best solution in current population 6. [Loop] Go to step 2

Single objective optimization makespan optimization robustness optimization

Multi-objective optimization • • • Goal: Minimize makespan and maximize robustness at the same time. Conflict - there cannot be a single optimum solution which simultaneously optimizes both objectives. Solution – seek balance between the 2 objectives.

Multi-objective optimization • Classical methods — weighted sum — -constraint • Weigthed sum — scalarizes multiple objectives into a single objective — • -constraint — optimize one of the objectives , subject to some constraints imposed on the other objectives

Weighted sum Objective function aws: average weighted slack: ni is scheduled on pj

-constraint § Objective: – maximize aws (average weighted slack) – subject to: ms < ε*ms 0 § Solutions: – feasible (ms < ε*ms 0) – infeasible (ms ≥ ε*ms 0) § Fitness:

Summary • Studied robust scheduling in non-deterministic environment using GA. — Provided a measurement of robustness. — Robust schedule could be generated through the optimization of average weighted slack (AWS) of a task graph. — Makespan and robustness are two conflicting objectives. — Multi-objective optimization methods are employed. — Weighted sum method is easy to use and intuitive. Setting up an appropriate weight vector depends on the scaling of each objective function. Normalization of objectives is usually required. — -constraint methods let user optimize one objective while imposing constraints on other objectives.