Enterprise Job Scheduling for Clustered Environments Stratos Paulakis

Enterprise Job Scheduling for Clustered Environments Stratos Paulakis, Vassileios Tsetsos, and Stathes Hadjiefthymiades Pervasive Computing Research Group Communication Networks Laboratory Department of Informatics and Telecommunications University of Athens – Greece Santorini ‘ 07 @ Greece

Outline • • Introduction System Design System Implementation Performance Evaluation & Comparison with Quartz Scheduler • Conclusions

Introduction • System-level scheduling vs. Application-level scheduling • Po. Lo. S platform for LBS – IST FP 5 Project – Scheduler was a core architectural component (timetriggered SMS/WAP services) • Clustering: modern solution for scalability and faulttolerance in enterprise systems • Objective – Design and implementation of a cluster-aware version of the original Scheduler

Functional Requirements • Time-accuracy and low delay – Jobs should commence execution as close as possible to their registered time – Delay tolerance depends on the application • Efficiency and scalability – High throughput is mandatory in large-scale applications • Robustness through job persistence – System crashes should not result in data loss – It imposes a performance overhead • High availability and fault tolerance – Near-zero downtime – No missed job execution • Logging – Billing, administration, SLAs

Technical Requirements • Asynchronous decoupling – The scheduling process should run independently from the job executions • Parallel job execution – Multi-threaded job execution • Load balancing – Maximizing utilization of available resources – Client- or server-side • Clustering – Deals with most requirements – Challenge: global timer is a singleton object

Related Systems Kronova Quartz Flux Custom Java Jobs √ √ √ Simple Time-Scheduling (start/stop time, period) √ √ √ Event-Driven Scheduling √ √ √ API √ √ √ JMX-compatible X √ X Logging √ √ √ JTA-enabled X √ X Tracking Data X √ X Clustering √ √ √ Load-Balancing √ √ √ Poor Moderate (the only point of failure is the DB) Poor Fault-Tolerance (Fail-Over)

Architecture Node A Node B Management Caching Execution Scheduling Node C - Master Management Caching Execution Scheduling Queue

Caching Subsystem • • • Distributed in-memory cache Synchronous and asynchronous replication Optimistic and pessimistic locking A DB is asynchronously updated Implementation: JBoss. Cache – Aspect-oriented programming techniques for cache updates

Scheduling Subsystem • JMX Timer – JBoss implementation • Java 5. 0 JMX Timer class has limitations in multi-threading (high timer delays) • Singleton design pattern – one instance may be instantiated across the cluster • Recovers job trigger times after master node crash Job trigger Cache job data Put job into queue

Performance Evaluation Setup • Setup – Cluster with 2 nodes (AMD 64 bit - 3. 4 GHz, 1 GB RAM) – JBoss 4. 0. 4, My. SQL 5. 0 – JMeter for workload generation, JProfiler for performance profiling • Metrics – Maximum throughput: scalability measure – Delays • Average total delay • T 1: timer delay, T 2: persistence delay, T 3: queuing delay

Performance Evaluation Setup II • • Po. Lo. S - Synchronous replication Po. Lo. S - Asynchronous replication Clustered Quartz JMeter parameters: – Discrete user requests: 1000, 2000, …, 4000 – Ramp up period: 120 seconds – Repetitions: 10 – Period: variable (60, 120, 180, …, 300 seconds) – Job logic: a logging command

Maximum Throughput • • Original non-clustered scheduler: ~3000 jobs/min Po. Lo. S sync: 2998 jobs/min - 6000 user requests Po. Lo. S async: 2810 jobs/min – 6000 user requests Quartz: 240 jobs/min – 4000 user requests – Transaction isolation errors during persistence high latency – Server crashed for more than 4000 user requests Job period = 120 sec

Delays 1000 jobs T 2 Polos sync 3. 2 ms Polos async 0. 75 msec Quartz Job period = 60 sec 91905 msec

Delay Distribution T 1 T 2 Polos sync 41549 ms 7. 1 ms Polos async 490 ms 1 ms Quartz 120 ms 198550 ms 2000 jobs with period 60 sec

Conclusions • JBoss JMX timer resulted in lower timer delays (T 1) but the MDBs could not operate at that high rate • Asynchronous replication is much more efficient than synchronous • When the maximum throughput is reached, delays increase dramatically

Thank You! Questions? ? ? http: //p-comp. di. uoa. gr