Computer Systems Principles Concurrency Patterns Emery Berger and
Computer Systems Principles Concurrency Patterns Emery Berger and Mark Corner University of Massachusetts Amherst UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science
Web Server web server § Client (browser) – Requests HTML, images § Server – Caches requests – Sends to client not found http: //server/Easter-bunny/ 200 x 100/75. jpg client UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science
Possible Implementation while (true) { wait for connection; read from socket & parse URL; look up URL contents in cache; if (!in cache) { fetch from disk / execute CGI; put in cache; } send data to client; } UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 3
Possible Implementation while (true) { wait for connection; // net read from socket & parse URL; // cpu look up URL contents in cache; // cpu if (!in cache) { fetch from disk / execute CGI; //disk put in cache; // cpu } send data to client; // net } UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 4
Problem: Concurrency § Sequential fine until: web server – More clients – Bigger server • Multicores, multiprocessors § Goals: – Hide latency of I/O • Don’t keep clients waiting – Improve throughput • Serve up more pages clients UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science
Building Concurrent Apps § Patterns / Architectures – Thread pools – Producer-consumer – “Bag of tasks” – Worker threads (work stealing) § Goals: – Minimize latency – Maximize parallelism – Keep progs. simple to program & maintain UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 6
Thread Pools § Thread creation relatively expensive § Instead: use pool of threads – When new task arrives, get thread from pool to work on it; block if pool empty – Faster with many tasks – Limits max threads (thus resources) – ( Thread. Pool. Executor class in Java) UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 7
Producer-Consumer § Can get pipeline parallelism: – One thread (producer) does work • E. g. , I/O – and hands it off to other thread (consumer) producer consumer UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 8
Producer-Consumer § Can get pipeline parallelism: – One thread (producer) does work • E. g. , I/O – and hands it off to other thread (consumer) producer consumer UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 9
Producer-Consumer § Can get pipeline parallelism: – One thread (producer) does work • E. g. , I/O – and hands it off to other thread (consumer) producer consumer UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 10
Producer-Consumer § Can get pipeline parallelism: – One thread (producer) does work • E. g. , I/O – and hands it off to other thread (consumer) producer consumer UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 11
Producer-Consumer § Can get pipeline parallelism: – One thread (producer) does work • E. g. , I/O – and hands it off to other thread (consumer) producer consumer Linked. Blocking. Queue Blocks on put() if full, poll() if empty UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 12
Producer-Consumer Web Server § Use 2 threads: producer & consumer – queue. put(x) and x = queue. poll(); while (true) { wait for connection; read from socket & parse URL; look up URL contents in cache; if (!in cache) { fetch from disk / execute CGI; put in cache; } send data to client; } while (true) { do something… queue. put (x); } while (true) { x = queue. poll(); do something… } UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 13
Producer-Consumer Web Server § Pair of threads – one reads, one writes while (true) { wait for connection; read from socket & parse URL; queue. put (URL); } while (true) { URL = queue. poll(); look up URL contents in cache; if (!in cache) { fetch from disk / execute CGI; put in cache; } send data to client; } UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 14
Producer-Consumer Web Server § More parallelism – optimizes common case (cache hit) while (true) { wait for connection; read from socket & parse URL; queue 1. put (URL); } 1 while (true) { URL = queue 1. poll(); look up URL contents in cache; if (!in cache) { queue 2. put (URL); return; } send data to client; } 2 while (true) { URL = queue 2. poll(); fetch from disk / execute CGI; put in cache; send data to client; } UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 15
When to Use Producer-Consumer § Works well for pairs of threads – Best if producer & consumer are symmetric • Proceed roughly at same rate – Order of operations matters § Not as good for – Many threads – Order doesn’t matter – Different rates of progress UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 16
Producer-Consumer Web Server § Should balance load across threads while (true) { wait for connection; read from socket & parse URL; queue 1. put (URL); } 1 while (true) { URL = queue 1. poll(); look up URL contents in cache; if (!in cache) { queue 2. put (URL); } send data to client; } 2 while (true) { URL = queue 2. poll(); fetch from disk / execute CGI; put in cache; send data to client; } UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 17
Bag of Tasks § Collection of mostly independent tasks worker UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 18
Bag of Tasks § Collection of mostly independent tasks worker UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 19
Bag of Tasks § Collection of mostly independent tasks worker UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 20
Bag of Tasks § Collection of mostly independent tasks worker UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 21
Bag of Tasks § Collection of mostly independent tasks worker UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 22
Bag of Tasks § Collection of mostly independent tasks worker UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 23
Bag of Tasks § Collection of mostly independent tasks add. Work worker § Bag could also be Linked. Blocking. Queue (put, poll) UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 24
Exercise: Restructure into BOT § Re-structure this into bag of tasks: – add. Work & worker threads – t = bag. poll() or bag. put(t) while (true) { wait for connection; read from socket & parse URL; look up URL contents in cache; if (!in cache) { fetch from disk / execute CGI; put in cache; } send data to client; } UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 25
Exercise: Restructure into BOT § Re-structure this into bag of tasks: – add. Work & worker – t = bag. poll() or bag. put(t) add. Work: while (true) { wait for connection; t. URL = URL; t. sock = socket; bag. put (t); } Worker: while (true) { t = bag. poll(); look up t. URL contents in cache; if (!in cache) { fetch from disk / execute CGI; put in cache; } send data to client via t. sock; } UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 26
Bag of Tasks Web Server § Re-structure this into bag of tasks: – t = bag. poll() or bag. put(t) add. Work: while (true){ wait for connection; bag. put (URL); } worker add. Work worker: while (true) { URL = bag. poll(); look up URL contents in cache; if (!in cache) { fetch from disk / execute CGI; put in cache; } send data to client; } UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 27
Bag of Tasks vs. Prod/Consumer § Exploits more parallelism § Even with coarse-grained threads – Don’t have to break up tasks too finely § What does task size affect? – possibly latency… smaller might be better § Easy to change or add new functionality § But: one major performance problem… UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 28
What’s the Problem? add. Work worker UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 29
What’s the Problem? § Contention – single lock on structure – Bottleneck to scalability add. Work worker UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 30
Work Queues § Each thread has own work queue (deque) – No single point of contention executor § Threads now generic “executors” – Tasks (balls): blue = parse, yellow = connect… UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 31
Work Queues § Each thread has own work queue (deque) – No single point of contention executor UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 32
Work Queues § Each thread has own work queue (deque) – No single point of contention executor UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 33
Work Queues § Each thread has own work queue (deque) – No single point of contention executor UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 34
Work Queues § Each thread has own work queue – No single point of contention executor § Now what? UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 35
Work Stealing § When thread runs out of work, steal work from random other thread worker UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 36
Work Stealing § When thread runs out of work, steal work from top of random deque worker § Optimal load balancing algorithm UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 37
Work Stealing Web Server § Re-structure: read. URL, look. Up, add. To. Cache, output – my. Queue. put(new read. URL (url)) while (true) { wait for connection; read from socket & parse URL; look up URL contents in cache; if (!in cache) { fetch from disk / execute CGI; put in cache; } send data to client; } UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 38
§ read. URL, look. Up, add. To. Cache, output § class Work { while (true) { public: wait for connection; read from socket & parse URL; virtual void run(); look up URL contents in cache; if (!in cache) { }; fetch from disk / execute CGI § class read. URL : public Work { put in cache; } public: send data to client; } void run() {…} read. URL (socket s) { …} }; UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 39
read. URL look. Up add. To. Cache worker output UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 40
§ class read. URL { public: void run() { read from socket, f = get file my. Queue. put (new look. Up(_s, f)); } read. URL(socket s) { _s = s; } }; UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 41
§ class look. Up { public: void run() { look in cache for file “f” if (!found) my. Queue. put (new add. To. Cache(_f)); else my. Queue. put (new Output(s, cont)); } look. Up (socket s, string f) { _s = s; UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 42
§ class add. To. Cache { public: void run() { fetch file f from disk into cont add file to cache (hashmap) my. Queue. put (new Output(s, cont)); } UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 43
Work Stealing Web Server § Re-structure: read. URL, look. Up, add. To. Cache, output – my. Queue. put(new read. URL (url)) read. URL(url) { wait for connection; read from socket & parse URL; my. Queue. put (new look. Up (URL)); } UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 44
Work Stealing Web Server § Re-structure: read. URL, look. Up, add. To. Cache, output – my. Queue. put(new read. URL (url)) read. URL(url) { wait for connection; read from socket & parse URL; my. Queue. put (new look. Up (URL)); } add. To. Cache(URL) { fetch from disk / execute CGI; put in cache; my. Queue. put (new output(contents)); } look. Up(url) { look up URL contents in cache; if (!in cache) { my. Queue. put (new add. To. Cache (URL)); } else { my. Queue. put (new output(contents)); } } UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 45
Work Stealing § Works great for heterogeneous tasks – Convert add. Work and worker into units of work (different colors) § Flexible: can easily re-define tasks – Coarse, fine-grained, anything in-between § Automatic load balancing § Separates thread logic from functionality § Popular model for structuring servers UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 46
The End UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 47
- Slides: 47