Thread Pools 1 Whats A Thread Pool A

Thread Pools 1

What’s A Thread Pool? • A programming technique which we will use. • A collection of threads that are created once (e. g. when server starts). • That is, no need to create a new thread for every clients request. • Instead, the application (in our case – the server) uses an already prepared thread if there’s a free one, or waits until such one exists. 2

Why Thread Pools? • Thread pools improve resource utilization - The overhead of creating a new thread is significant • Thread pools enable applications to control and bound their thread usage - Creating too many threads in one JVM can cause the system to run out of memory (or reach the OS thread-limit, if you’re writing a server for a very high-profile site) and even crash 3

Thread Pools in Servers • Thread pools are especially important in client-server applications - The processing of each individual task is short-lived and the number of requests is large - Servers should not spend more time and consume more system resources creating and destroying threads than processing actual user requests • When too many requests arrive, thread pools enable the server to force clients to wait until threads are available 4

The “Obvious” Implementation • There is a pool of threads • Each task asks for a thread when starting and returns the thread to the pool after finishing • When there are no available threads in the pool the thread that initiates the task waits till the pool is not empty • What is the problem here? “Synchronized” model the client waits until server takes care of its request… 5

The “Obvious” Implementation is Problematic • When the pool is empty, the submitting thread has to wait for a thread to be available - We usually want to avoid blocking that thread - A server may want to perform some actions when too many requests arrive • Technically, Java threads that finished running cannot run again 6

A Possible Solution Every thread looks for tasks in the queue Task Queue wait() If Q is Empty Worker Threads All the worker threads wait for tasks 7

A Possible Solution Task Queue Worker Threads “A-synchronized” model – “Launch and forget” The number of worker threads is fixed. When a task is inserted to the queue, notify is called. 8

A Possible Solution Task Queue notify() Worker Threads The number of worker threads is fixed. When a task is inserted to the queue, notify is called. 9

A Possible Solution The task is executed by the thread Task Queue Worker Threads 10

A Possible Solution The task is executed by the thread Task Queue notify() Worker Threads 11

A Possible Solution The remaining tasks are executed by the other threads Task Queue Worker Threads 12

A Possible Solution When a task ends the thread is released Task Queue Q is not empty, take the task from the Q and run it (if the Q was empty, wait() would have been called) While the Worker Threads 13

A Possible Solution A new task is executed by the released thread Task Queue Worker Threads 14

A Possible Solution A new task is executed by the released thread Task Queue Worker Threads This is a special case of a design pattern known as the Producer/Consumer model, which can be convenient for a wide range of (also very different) applications / algorithms. Basically, you divide a task you have to repeat many times into a few steps, prepare a thread pool in charge of each step and a queue for elements waiting between every pair of adjacent steps. 15

Thread Pool Implementation public class Task. Manager { Linked. List task. Queue = new Linked. List(); List threads = new Linked. List(); public Task. Manager(int num. Threads) { for(int i=0; i<num. Threads; ++i) { Thread worker = new Worker(task. Queue); threads. add(worker); worker. start(); }} public void execute(Runnable task) { synchronized(task. Queue) { task. Queue. add. Last(task); task. Queue. notify(); }}} 16

Thread Pool Implementation public class Worker extends Thread { Linked. List task. Queue = null; public Worker(Linked. List queue) { task. Queue = queue; } public void run() { Runnable task = null; while (true) { synchronized (task. Queue) { while (task. Queue. is. Empty()) { try {task. Queue. wait(); } catch (Interrupted. Exception ignored) {}} task = (Runnable) task. Queue. remove. First(); } task. run(); }}} 17

Risks in Using Thread Pools That’s a problem in the client’s implementation of run() • Threads can leak - A thread can endlessly wait for an I/O operation to complete. For example, the client may stop the interaction with the socket without closing it properly - What if task. run() throws a runtime exception Again, problem in the client’s side code (as opposed to other exception that client’s application programmer has to catch in order to succeed compiling)? • Solutions: The client side’s programmer The server side’s programmer can also define a timer with a maximum timeout for the completion of run() - Bound I/O operations by timeouts using wait(time) - Catch possible runtime exceptions The server side’s programmer 18

Pool Size • Each thread consumes resources - memory, management overhead, etc. • Therefore, you have to Tune thread pool size according to the number and characterizations of expected tasks • There should also be a limit on the size of the task queue - What should we do when too many requests arrive? Use several pool sizes alternately according to stress characteristics but don’t change the size too often… 19