Large Grand INRIA Hybrid Preemptive Scheduling of MPI

Large Grand INRIA Hybrid Preemptive Scheduling of MPI applications Aurélien Bouteiller, Hinde Lilia Bouziane, Thomas Hérault, Pierre Lemarinier, Franck Cappello MPICH-V team INRIA Grand-Large LRI, University Paris South 1

Large Grand INRIA Problem definition • Context: Clusters and Grids (made of clusters) shared by many users (less available resources than required at a given time) In this study : finite sets of MPI applications. Time sharing of parallel applications is attractive to increase fairness between users, compared to Batch scheduling • It is very likely that several applications will reside in the virtual memory at the same time, exceeding the total physical memory Out-of-core scheduling of parallel applications on clusters! (scheduling // applications on cluster under mem. constraint) • Most of the proposed approaches tries to avoid this situation (by limiting job admission based on mem. requirement, delaying some jobs unpredictably if the jobs exec. time is not known) Issue: Novel approach (out-of-core) that avoid delaying some jobs? Constraint: No OS modification (no kernel patch) 2

Large Grand INRIA Outline • Introduction (related work) • A Hybrid approach dedicated to out-of-core • Evaluation • Concluding remarks 3

Large Grand INRIA Related work 1 Scheduling parallel applications on distributed memory machines: a long history of research, still very active (5 papers in 2004 in main conferences: IPDPS, Cluster, SC, Grid, Europar)! Time Co-scheduling: all processes of each Proc 1 application are scheduled independently Proc 2 (no coordination) Appl 1 Appl 2 Appl 3 Appl 1 Expected advantage: overlapping comm. And comp. Gang-scheduling: all processes of each application are executed simultaneously (coordination) Time slice Proc 1 Proc 2 sometimes called “co. Communication Scheduling overhead 4 scheduling” Expected advantage: scheduling communicating processes

Large Grand INRIA Related work 2 Comparison between Gang and Co scheduling: Gang scheduling out-performs co-scheduling • D. G. Feitelson and L. Rudolph. Gang Scheduling Performance Benefits for Fine-Grained Synchronization. Journal of Parallel and Distributed Computing, 16(4): 306– 318, December 1992. Co-scheduling out-performs gang scheduling • Eitan Frachtenberg, Dror G. Feitelson, Fabrizio Petrini and Juan Fernandez, “Flexible Co. Scheduling: Mitigating Load Imbalance and Improving Utilization of Heterogeneous Resources”, IPDPS 2003 (Gang schedule only applications that take advantage of if – after classification) • Gyu Sang Choi, Jin-Ha Kim, Deniz Ersoz, Andy B. Yoo, Chita R. Das, “Coscheduling in Clusters: Is It a Viable Alternative? ”, SC 2004 (increases priority of processes during communications) • Peter Strazdins and John Uhlmann, « Local scheduling outperforms gang scheduling on a beowulf cluster » Technical report, Department of Computer Science, Australian National University, January 2004, Cluster 2004. (Ethernet, Score for Gang, Mat. Mul and Linpack) Multiple parameter problem Conclusion depends on assumptions ! 5

Large Grand INRIA Related work 3 Metrics for measuring performance: Metrics and Benchmarking for Parallel Job Scheduling [Fe 98] Performance • Makespan • Throughput • Response time Not so much investigated. Still very important: Fairness • Standard deviation of the response time for a set of homogeneous applications • Minimum is the best fairness 6

Large Grand INRIA Outline • Introduction (related work) • A Hybrid approach dedicated to out-of-core • Evaluation • Concluding remarks 7

Large Grand INRIA Principle: Expected benefits: Potential limitation: Our approach 1/2: Hybrid • • A given set of parallel applications to schedule Application Sub. Sets: a set of applications fitting in memory Co-scheduling applications within a Sub. Set Gang scheduling Sub. Sets of applications Overlapping Comm. and I/O with computation in a Sub. Set No memory page miss/replacement during Subset exec. Allow known Co-scheduling optimizations in Sub. Set • High “Subset context” switching overhead Example: 1 set of 6 apps. with 2 Subsets of 3 applications: Time slice In-core co-scheduling Out-of-core gang scheduling Communications Application Subset context 8 switch

Large Grand INRIA Our approach 2/2: Checkpointing OS virtual memory management Memory page 1 page 2 page 3 page 1 Disk Checkpointing Basic OS virtual memory management: Memory • Paging in pages on request page 1 • Paging out pages on replacementpage 2 (LRU) page 2 page 3 • Interaction with OS scheduler (some OS are deliberately unfair in out-of-core free page 3 situation. ) free page 1 Poor performance for HPC Disk applications Adaptive Memory Paging for Gang-Scheduling [Ry 04] Best performance for gang scheduling is obtained by: 1) 2) 3) Selective paging out (swapping out only pages of descheduled processes) Aggressive paging out (evicting pages of descheduled processes at once) Adaptive paging In (swapping in pages of scheduled process at once) 4) Good but requires deep kernel modifications. Our approach: user level application Subset checkpointing Checkpointing provides the same benefits than 1), 2) and 3). Works for Co-scheduling as well as for Gang-Scheduling Does not require any kernel modification! We need a parallel application (MPI) checkpoint mechanism 9

Large Grand INRIA Implementation using MPICH-V Framework MPICH-V framework: a set of components A MPICH-V protocol: a composition of a subset of these components Event Loggers Checkpoint servers Channel Memories Dispatcher Fault detector 2 node Network Mpi processes Daemons 3 node 1 Checkpoint Scheduler node 10

Large Grand INRIA 6 1 2 3 Checkpoint protocol selection: Coordinated or uncoordinated? protocols implemented in MPICH-V: coordinated (Chandy-Lamport) uncoordinated + pessimist mess. log. uncoordinated + causal log. The coordinated one provides the best performance for fault free execution 11

Large Grand Coordinated Checkpoint: INRIA time 2 ways Ckpt. Sig. Flushing the network (Chandy-Lamport) P 1 P 0 B 1 B 0 Flush P 1 B 0 + B 1 P 1 Mess. delivery Ckpt. Comm. Buff. Ckpt. Image (P 1) = P 0 B 1 + B 1 P 1 Rstrt. Mess. delivery time Rstrt. Sig. P 0 In-transit Mess. Ckpt. Image (P 1) = P 0 Checkpointing the communication stack (Parakeet, Meoisys, Score) Checkpoint may last longer Restart may last longer 1) We expect minor Perf. Diff. between the 2 approaches 12 2) Checkpoint/restart of the comm. Stack requires OS modifications So we implemented the Chandy-Lamport approach

Large Grand INRIA MPICH-V/CL protocol Coordinated checkpointing, (Chandy-Lamport) Reference protocol for coordinated checkpointing 1) When receiving a Checkpoint tag, start checkpoint + store any incoming mess. 2) Store all incoming messages in checkpoint image 3) Send checkpoint tag to all neighbors in the topology. 4) Checkpoint is finished when a Tag has been received from all neighbors 4) After a crash, all nodes retrieve checkpoint images 13 5) Deliver stored in-transit messages to restarted processes

Large Grand INRIA Implementation details MPICH-V Dispatchers Nodes. Network Ckpt. Sched. Master Scheduler Network Ckpt. Sched. Ckpt. Serv. Daemons Co-scheduling: Several Dispatchers (no master/checkpoint scheduler) Gang and (Hybrid): Master Scheduler + several checkpoint schedulers 1) 2) 3) 4) Master Scheduler issues a checkpoint order to the Checkpoint Scheduler(s) of running application(s) When receiving this order, a Checkpoint Scheduler launches a coordinated checkpoint. Every running daemon computes the MPI process image and store it on the local disc. All daemons send a completion message to the Checkpoint Scheduler. All running daemons stop the MPI process and their execution The Master Scheduler selects the Checkpoint Scheduler(s) of other application(s) and sends a restart order. Every Checkpoint Scheduler receiving this order spawns new daemons restarting MPI processes from local images. 14

Large Grand INRIA Outline • Introduction (related work) • A Hybride approach dedicated to out-of-core • Evaluation • Concluding remarks 15

Large Grand INRIA Methodology • LRI cluster: – – – Athlon 1800+ 1 GB memory IDE ATA 100 Disc Ethernet 100 Mbs Linux 2. 4. 2 Beowulf Cluster • Benchmark (MPI): – NAS BT (computation bound) – NAS CG (communication bound) • Time measurement: – Homogeneous Applications – Simultaneous launch (scripts) – Time is measured between the first launch and the last termination – Fairness is measured by response time standard deviation • Gang Scheduling time slice: 200 or 600 sec 16 – Gang sched. also implemented by checkpointing (not OS signal)

Large Grand INRIA Context switch overlap policy We can imagine several policies to switch between set contexts Which one is the best for in-core and out-of-core situations? A) Sequential store and load: 1 X: 1 context in memory Time slice B) Store and load in parallel: 2 X: 2 contexts in memory Execution t Context Storage C) Load Prefetech: Context Load 2 X: 2 contexts in memory Policies for NAS Bench. BT –C- 25 1) Overlapping policies do not provide substantial 2)improvements for the in-core situation 3)2) They need 2 x the memory capacity to stay in-core. the sequential policy is the best We used it for the other xps. In core Near out-of-core <3% 1 X 2 X 2 X 2 X 17

Large Grand INRIA Co VS. Gang (Ckpt based) • Which scheduling strategy is the best for communication bound and compute bound applications? Makespan: Execution time of N applications with Co and Gang Scheduling NAS Benchmark CG and BT BT Co-scheduling Checkpoint based Gang scheduling 8000 7000 6000 5000 4000 3000 2000 1000 0 0 1 6 9 12 15 18 21 Number of CG-C-8 executed “simultaneously” In-core 1) 2) Out-of-core >> More than 24 k Sec. CG Co-scheduling Checkpoint based Gang scheduling 1 12 18 24 Number of BT-B-9 executed “simultaneously” In-core Out-of-core Co-scheduling is the best for in-core executions (but small advantage due to ~Checkpoint overhead + tinny Comm. /comp. overlap) Gang scheduling outperforms co-scheduling for out-of-core (ckpt. ) 18 3) Memory constraint is managed by checkpointing not by delaying jobs

Large Grand INRIA Ckpt Gang VS. Ckpt Hybrid Makespan: Execution time of N applications with Co, Gang and Hybrid Scheduling Chkp overhead Number of CG-C-8 executed “simultaneously” In-core • • Out-of-core Comm. /comp. Overlap >> More than 3000 Gang scheduling Hybrid scheduling (set of 5) Time (minutes) Co-scheduling Gang scheduling Hybrid scheduling (set of 5) Number of BT-B-9 executed “simultaneously” In-core Out-of-core Gang and Hybrid scheduling outperform co-scheduling for out-of-core Hybrid scheduling compares favorably to Gang scheduling on BT and OOC 19 thanks to communication and computation overlap.

Large Grand INRIA Overhead comparison • What is the performance degradation due to time sharing? Relative slowdown: (total time / # concur exec) / best of seq. times Co-scheduling CG BT Gang-Scheduling Relative Slowdown Hybrid-Scheduling 1, 6 1, 2 1, 5 1, 4 1, 1 1, 3 ref 1 1, 2 1, 1 ref 1 0, 9 0, 8 6 9 12 15 18 # concurrent CG-C-8 executions 1) 2) 3) 4) 21 0, 8 12 14 16 18 20 22 24 # concurrent BT-B-9 executions Gang and Hybrid scheduling add no performance penalty to CG (and also no improvement), Gang scheduling add 10% performance penalty to BT, Hybrid scheduling improves the performance by almost 10%, 20 Difference is mostly due to communication/computation overlap.

Large Grand INRIA Co-scheduling Fairness (Linux) • How fair is co-scheduling for in-core and out-of-core? Response time of BT 9 with modified memory sizes Time(s) 1400 In-core M=1210, Diff=8 1200 Slightly out-of-core 3000 M=2251, SD=298, Diff=961 2500 1000 2000 800 1500 600 1000 400 500 200 0 0 1 2 3 4 5 Application rank 6 1 2 3 4 5 6 7 8 9 Threads 10 11 12 13 14 Co-scheduling is highly unfair in out-of-core situation! Page miss statistics for 7 and 9 BT C 25 (out-of-core) # appls. # page misses per minute experienced by each node of an application (mean) app 0 app 1 app 2 app 3 app 4 . . . 7 8. 5 239. 5 264. 25 951 1145. 25 . . . 9 484. 58 405. 78 524. 4 510. 66 . . . app 8 509. 4 # page misses per minute (mean over all applications) Stdr. Deviat. 704 474. 9 507. 4 47. 1 21 1) Fairness deficiency for slightly out-of-core seems due to virtual mem. Mgnt. 2) Of course there should be some solution, but involving Kernel modification

Large Grand INRIA Outline • Introduction (related work) • A Hybrid approach dedicated to out-of-core • Evaluation • Concluding remarks 22

Large Grand INRIA Concluding remarks • Checkpoint based Gang Scheduling outperforms Co-scheduling and certainly classical (OS signal based) Gang scheduling on out-of-core situation (thanks to a better memory management) • Compared to known approaches, based on job admission control, the benefit of ckpt is that it avoids to delay some jobs • Hybrid scheduling, combining the two approaches + checkpointing, outperforms Gang scheduling on BT (presumably thanks to overlapping communications and computations) • More generally, Hybrid scheduling can take advantage of advanced co-scheduling approaches within a gang subset Work in progress: • • Test with other applications / benchmarks Compare with traditional gang scheduling based on OS signals Experiments with high speed networks Experiments on Hybrid scheduling with Co-scheduling optimizations 23

Large Grand INRIA Meet us! at the INRIA booth 2345 INRIA Booth 2345 Mail contact: bouteiller@mpich-v. net

Large Grand INRIA References [Ag 03] S. Agarwal, G. Choi, C. R. Das, A. B. Yoo, and S. Nagar. Co-ordinated Coscheduling in time-sharing Clusters through a Generic Framework. In Proceedings of International Conference on Cluster Computing, December 2003. [Ar 98] A. C. Arpaci-Dusseau, D. E. Culler, and A. M. Mainwaring. Implicit Scheduling With Implicit Information in Distributed Systems. In Proceedings of the 1998 ACM SIGMETRICS joint International Conference on Measurement and Modeling of Computer Systems, pages 233– 243, June 1998. [Ba 00] Anat Batat and Dror G. Feitelson, « Gang Scheduling with Memory Considerations » , in proceedings of IPDPS 2000. [Bo 03] Aurélien Bouteiller, Pierre Lemarinier, Géraud Krawezik, and Franck Cappello, « Coordinated checkpoint versus message log for fault tolerant MPI » , In IEEE International Conference on Cluster Computing (Cluster 2003). IEEE CS Press, december 2003. [Ch 85] K. M. Chandy and L. Lamport, « Distributed snapshots: Determining global states of distributed systems » In Transactions on Computer Systems, volume 3(1), pages 63– 75. ACM, February 1985. [Fe 98] D. G. Feitelson and L. Rudolph, “Metrics and Benchmarking for Parallel Job Scheduling”. In Job Scheduling Strategies for Parallel Processing, LNCS vol. 1495, pp. 1– 24, Springer-Verlag, Mar 1998. [Fr 03] Eitan Frachtenberg, Dror G. Feitelson, Fabrizio Petrini and Juan Fernandez, “Flexible Co. Scheduling: Mitigating Load Imbalance and Improving Utilization of Heterogeneous Resources”, IPDPS 2003 [Ho 98] Atsushi Hori, Hiroshi Tezuka, and Yutaka Ishikawa, « Overhead analysis of preemptive gang scheduling » , Lecture Notes in Computer Science, 1459 : 217– 230, April 1998. [Ky 04] Kyung Dong Ryu, Nimish Pachapurkar, Liana L. Fong, « Adaptive Memory Paging for Efficient Gang Scheduling of Parallel Applications”, in proceedings of IPDPS 2004. [Na 99] S. Nagar, A. Banerjee, A. Sivasubramaniam, and C. R. Das. Alternatives to Coscheduling a Network of Workstations. Journal of Parallel and Distributed Computing, 59(2): 302– 327, November 1999. [Ni 02] Dimitrios S. Nikolopoulos and Constantine D. Polychronopoulos, « Adaptive Scheduling under Memory Pressure on Multiprogrammed Clusters”, CCGRID 2002 [Sa 04] Gyu Sang Choi, Jin-Ha Kim, Deniz Ersoz, Andy B. Yoo, Chita R. Das, “Coscheduling in Clusters: Is It a Viable Alternative? ”, to appear in SC 2004 [Se 99] S. Setia, M. S. Squillante, and V. K. Naik. The Impact of Job Memory Requirements on Gang-Scheduling Performance. ACM SIGMETRICS Performance Evaluation Review, 26(4): 30– 39, 1999. [So 98] P. G. Sobalvarro, S. Pakin, W. E. Weihl, and A. A. Chien. Dynamic Coscheduling on Workstation Clusters. In Proceedings of the IPPS Workshop on Job Scheduling Strategies for Parallel Processing, pages 231– 256, March 1998. [St 04] Peter Strazdins and John Uhlmann, « Local scheduling outperforms gang scheduling on a beowulf cluster » Technical report, Department of Computer Science, Australian National University, January 2004, to appear in Cluster 2004. [Wi 03] Yair Wiseman, Dror G. Feitelson, « Paired Gang Scheduling » , IEEE TPDS, June 2003

Large Grand INRIA Related work optimizations: -Memory management (mainly based on job admission control): • Impact of Memory Requirements on Gang-Scheduling Performance [Se 99] (cont. of multiprog. ) • Gang Scheduling with Memory Considerations [Ba 00] (job admission control to avoid swapping) • Memory aware Co-scheduling [Ch 04] (job admission control to avoid swapping). • Adaptive Memory Paging for Gang-Scheduling [Ry 04] (Improving memory paging in-out) -Communications (concerns co-scheduling): • ICS (Implicit Co-scheduling), SB (spin Blocking), CC (Coordinated Co-scheduling): self descheduling after timeout on communication [Ar 98][Na 99] • DCS (Dynamic Co-scheduling): incoming message triggers receiver scheduler [So 98] • PB (Periodic Boost): schedule receiver from Periodic check of receiving buffer [Na 99] 26

Large Grand INRIA Is result for in-core situation Kernel dependent (Linux)? Kernel 2. 4. 2 was used in our experiment How time sharing efficiency evolves with Linux kernel maturation (from 2. 4 to 2. 6)? In-core Time (s) 3000 Comp 2. 4. 2 Comm. 2. 4. 2 Exec 2. 4. 2 Comp 2. 6. 2 Comm 2. 6. 2 Exec 2. 6. 2 Comp 2. 6. 7 Comm. 2. 6. 7 Exec 2. 6. 7 1000 Time (s) 1000 10 100 1 2 3 4 # of concurrent CG A 4 5 1 2 3 4 # of concurrent BT A 9 Yes, performance of co-sheduling (in core) depends on the kernel 1) Kernel 2. 6. 2 is less efficient (much less for CG) 2) Kernel 2. 6. 7 and 2. 4. 2 provides overall similar performance 27 Careful selection of kernel version OR restriction (desactivation) of Co-scheduling 5