Load Balancing Definition A load is balanced if

Load Balancing Definition: A load is balanced if no processes are idle • How? – Partition the computation into units of work (tasks or jobs) – Assign tasks to different processors • Load Balancing Categories – Static (load assigned before application runs) – Dynamic (load assigned as applications run) o Centralized (Tasks assigned by the master or root process) o De-centralized (Tasks reassigned among slaves) – Semi-dynamic (application periodically suspended and load balanced) • Load Balancing Algorithms are: – Adaptive if they adapt to system load levels using thresholds – Stable if load balancing traffic is independent of load levels – Symmetric if both senders and receivers initiate action – Effective if load balancing overhead is minimal Note: Load balancing is an NP-Complete problem

Improving the Load Balance By realigning processing work, we improve speed-up

Static Load Balancing Done prior to executing the parallel application • Round Robin – Tasks assigned sequentially to processors – If tasks > processors, the allocation wraps around • Randomized: Tasks are assigned randomly to processors • Partitioning – Tasks are represented by a graph – Recursive Bisection – Simulated Annealing – Genetic Algorithms – Multi-level Contraction and Refinement • Advantages – Simple to implement – Minimal run time overhead • Disadvantages – Predicting execution times is often not knowable – Affect of communication dynamics is often ignored – The number of iterations required by processors to converge on a solution is often indeterminate Note: The Random algorithm is a popular benchmark for comparison

A Load Balancing Partitioning Graph • • The Nodes represent tasks The Edges represent communication cost The Node values represent processing cost A second node value could represent reassignment cost

Dynamic Load Balancing Done as a parallel application executes • Centralized – A single process hands out tasks – Processes ask for more work when their processing completes – Double buffering (ask for more while still working) can be effective • Decentralized – Processes detect that their work load is low – Processes sense an overload condition • When new tasks are spawned during execution • When a sudden increase in task load occurs – Questions • • Which neighbors should participate in the rebalancing? How should the adaptive thresholds be set? What are the communications needed to balance? How often should balancing occur?

Centralized Load Balancing Work Pool, Processer Farm, or Replicated Worker Algorithm Master Processor: Maintains the work pool (queue, heap, etc. ) While ( task=Remove()) != null) Receive(pi, request_msg) Send(pi, task) While(more processes) Receive(pi, request_msg) Send(pi, termination_msg) Master Slave Processor: Perform task and then ask for another task = Receive(pmaster, message) While (task!=terminate) Process task Send(pmaster, request_msg) task = Receive(pmaster, message) In this case, the slaves do not spawn new tasks How would the pseudo code change if they did? Slaves

Decentralized Load Balancing (Worker processes interact among themselves) • There is no Master Processor • Each Processor maintains a work queue • Processors interact with neighbors to request and distribute tasks

Decentralized Mechanisms Balancing is among a subset of the total running processes Application • Receiver Initiated – Process requests tasks when it is about to go idle – Effective when the load is heavy – Unstable when the load is light (A request frequency threshold is necessary) Balancing Algorithm Task Queue • Sender Initiated – Process with a heavy load distributes the excess – Effective when the load is heavy – Can cause thrashing when loads are heavy (synchronizing system load with neighbors is necessary)

Process Selection • Global or Local? – Global involves all of the processors of the network • May require expensive global synchronization • May be difficult if the load dynamic is rapidly changing – Local involves only neighbor processes • Overall load may not be balanced • Easier to manage and less overhead than the global approach • Neighbor selection algorithms – Random: randomly choose another process • Easy to implement and studies show reasonable results – Round Robin: Select among neighbors using modular arithmetic • Easy to implement. Results similar to random selection – Adaptive Contracting: Issue bids to neighbors; best bid wins • Handshake between neighbors needed • It is possible to synchronize loads

Choosing Thresholds • How do we estimate system load? – Synchronization averages task queue length or processes – Average number of tasks or projected execution time • When is the load low? – When a process is about to go idle – Goal: prevent idleness, not achieve perfect balance – A low threshold constant is sufficient • When is the load high? – – When some processes have many tasks and others are idle Goal: prevent thrashing Synchronization among processors is necessary An exponentially growing threshold works well • What is the job request frequency? – Goal: minimize load balancing overhead

Gradient Algorithm Maintains a global pressure grid • Node Data Structures – For each neighbor • Distance, in hops, to the nearest lightly-loaded process – A load status flag indicating if the current processor is lightlyloaded, or normal L 1 2 1 1 2 2 • Routing – Spawned jobs go to the nearest lightly-loaded process • Local Synchronization – Node status changes are multicast to its neighbors

Symmetric Broadcast Networks (SBN) Global Synchronization Stage 3 5 Stage 2 1 Stage 1 3 7 Stage 0 4 2 0 6 • Characteristics – A unique SBN starts at each node – Each SBN is lg P deep – Simple operations algebraically Successor 1 = (p+2 s-1) %P; 1≤s≤ 3 compute successors Successor 2 = (p-2 s-1); 1≤s<3 – Easily adapts to the hypercube • Algorithm Note: If successor 2<0 – Starts with a lightly loaded process successor 2 +=P – Phase 1: SBN Broadcast – Phase 2: Gather task queue lengths – Load is balanced during the load and gather phases

Line Balancing Algorithm Uses a pipeline approach • Master or slave processors adjust pipeline • Slave processors – Request and receives tasks if queue not full – Pass tasks on if task request is posted • Non blocking receives are necessary to implement this algorithm pi Request task if queue not full pi+1 requests task Receive task from request Deliver task to pi+1 Dequeue and process task Note: This algorithm easily extends to a tree topology

Semi-dynamic • Pseudo code Run algorithm Time to check balance? Suspend application IF load is balanced, resume application Re-partition the load Distribute data structures among processors Resume execution • Partitioning – Model application execution by a partitioning graph – Partitioning is an NP-Complete problem – Goals: Balance processing and minimize communication and relocation costs – Partitioning Heuristics • Recursive Bisection, Simulated Annealing, Multi-level, Min. Ex

Partitioning Graph P 1 Load = (9+4+7+2) + (4+3+1+7) = 37 P 2 Load = (6+2+4+8+5) + (4+3+1+7) = 40 P 9 R 6 P 7 R 5 c 3 c 6 P 2 R 1 c 3 c 5 P 1 P 6 R 6 c 4 P 4 R 4 c 1 P 4 R 1 c 8 P 2 R 1 c 7 P 8 R 3 P 2 c 2 P 5 R 3 Question: When can we move a task to improve load balance?

Distributed Termination • Insufficient condition for distributed termination – Empty task queues at every process • Sufficient condition for distributed termination requires – All local termination conditions satisfied – No messages in transit that could restart an inactive process • Termination algorithms – – Acknowledgment Ring Tree Fixed energy distribution

Acknowledgement Termination • Process Receives task – Immediately acknowledge if source is not parent – Acknowledge parent as process goes idle • Process goes idle after it – completes processing local tasks – Sends all acknowledgments – Receives all pending acknowledgments Pi Acknowledge first task First task Inactive Pj • Notes – The process sending an initial task that activates another process becomes that process's parent – A process always goes inactive before its parent – If the master goes inactive, termination occurs Active

Single Pass Ring Termination • Pseudo code P 0 sends a token to P 1 when it goes idle Pi receives token IF Pi is idle it passes token to Pi+1 ELSE Pi sends token to Pi+1 when it goes idle P 0 receives token Broadcast final termination message • Assumptions – Processes cannot reactivate after going idle – Processes cannot pass new tasks to an idle process Token P 0 P 1 P 2 Pn

Dual Pass Ring Termination Handles task sent to a process that already passed the token on Key Point: Processors pass either Black or White tokens on only if they are idle Pseudo code (Only idle processors send tokens) WHEN P 0 goes idle and has token, it sends white token to p 1 IF Pi sends a task to pj where j<i Pi becomes a black process WHEN Pi>0 receives token and goes idle IF Pi is a black process Pi colors the token black, Pi becomes White ELSE Pi sends token to p(i+1)%P unchanged in color IF P 0 receives token and is idle IF token is White, application terminates ELSE po sends a White token to P 1 Process: white=ready for termination, black: sent a task to Pj-x Token: white=ready for termination, black=communication possible

Tree Termination • If a Leaf process terminates, it sends a token to it’s parent process • Internal nodes send tokens to their parent when all of their child processes terminate • If the root node receives the token, the application can terminate AND Terminated Leaf Nodes

Fixed Energy Termination Energy defined by an integer or long value • P 0 starts with full energy – When Pi receives a task, it also receives an energy allocation – When Pi spawns tasks, it assigns them to processors with additional energy allocations within its allocation – When a process completes it returns its energy allotment • The application terminates when the master becomes idle • Implementation – Problem: Integer division eventually becomes zero – Solution: o Use two level energy allocation <generation, energy> o The generation increases each time energy value goes to zero

Fair Scheduling in Web Servers CS 213 Lecture 17 L. N. Bhuyan

Objective • Create an arbitrary number of service quality classes and assign a priority weight for each class. • Provide service differentiation for different use classes in terms of the allocation of CPU and disk I/O capacities