Parallel Programming in Distributed Systems Or Distributed Systems

Parallel Programming in Distributed Systems Or Distributed Systems in Parallel Programming Philippas Tsigas Chalmers University of Technology Computer Science and Engineering Department © Philippas Tsigas

WHY PARALLEL PROGRAMMING IS ESSSENTIAL IN DISTRIBUTED SYSTEMS AND NETWORKING © Philippas Tsigas 2

How did we reach there? Picture from Pat Gelsinger, Intel Developer Forum, Spring 2004 (Pentium at 90 W) tsigas@cs. chalmers. se Philippas Tsigas 3

Concurrent Software Becomes Essential 1 Core 1) Scalability becomes an issue for all software. 24 GHz 2) Modern software development relies on the ability to compose libraries into larger programs. 12 GHz 2 Cores 4 Cores 6 GHz 8 Cores 3 GHz © Philippas Tsigas 3 GHz Our work is to help the programmer to develop efficient parallel programs but also survive the multicore transition. 4

DISTRIBUTED APPLICATIONS © Philippas Tsigas 5

Distributed Applications Demand Quite High Level Data Sharing: q q Commercial computing (media and information processing) Control Computing (on board flight-control system) © Philippas Tsigas 6

Data Sharing: Gameplay Simulation as an example This is the hardest problem… q 10, 000’s of objects q Each one contains mutable state Slide: Tim Sweeney CEO Epic Games POPL 2006 q Each one updated 30 times per second q Each update touches 5 -10 other objects Manual synchronization (shared state concurrency) is hopelessly intractable here. Solutions? © Philippas Tsigas 7

NETWORKING © Philippas Tsigas 8

40 multithreaded packet-processing engines http: //www. cisco. com/assets/cdc_content_elements/ embedded-video/routers/popup. html § § On chip, there are 40 32 -bit, 1. 2 -GHz packet-processing engines. Each engine works on a packet from birth to death within the Aggregation Services Router. each multithreaded engine handles four threads (each thread handles one packet at a time) so each Quantum. Flow Processor chip has the ability to work on 160 packets concurrently © Philippas Tsigas 9

DATA SHARING © Philippas Tsigas 10

Data Sharing: Gameplay Simulation as an example This is the hardest problem… q 10, 000’s of objects q Each one contains mutable state Slide: Tim Sweeney CEO Epic Games POPL 2006 q Each one updated 30 times per second q Each update touches 5 -10 other objects Manual synchronization (shared state concurrency) is hopelessly intractable here. Solutions? © Philippas Tsigas 11

Blocking Data Sharing A typical Counter Impl: Thread 1: get. Number() class Counter { int next = 0; synchronized int get. Number () { int t; t = next; next = t + 1; return t; } } next = 2 0 1 tsigas@cs. chalmers. se Philippas Tsigas Thread 2: get. Number() Lock acquired t=0 result=0 Lock released result=1 12

Do we need Synchronization? What can go wrong here? Thread 1: get. Number() Thread 2: get. Number() t=0 result=0 class Counter { int next = 0; int get. Number () { int t; t = next; next = t + 1; return t; } } next = 1 0 tsigas@cs. chalmers. se Philippas Tsigas 13

Blocking Synchronization = Sequential Behavior © Philippas Tsigas 14

BS ->Priority Inversion q A high priority task is delayed due to a low priority task holding a shared resource. The low priority task is delayed due to a medium priority task executing. q Solutions: Priority inheritance protocols q Works ok for single processors, but for multiple processors … Task H: Task M: Task L: © Philippas Tsigas 15

Critical Sections + Multiprocessors q Reduced Parallelism. Several tasks with overlapping critical sections will cause waiting processors to go idle. Task 1: Task 2: Task 3: Task 4: © Philippas Tsigas 16

§ The BIGEST Problem with Locks? Blocking Locks are not composable All code that accesses a piece of shared state must know and obey the locking convention, regardless of who wrote the code or where it resides. © Philippas Tsigas 17

Interprocess Synchronization = Data Sharing § § Synchronization is required for concurrency Mutual exclusion (Semaphores, mutexes, spin-locks, disabling interrupts: Protects critical sections) - Locks limits concurrency - Busy waiting – repeated checks to see if lock has been released or not - Convoying – processes stack up before locks - Blocking Locks are not composable - All code that accesses a piece of shared state must know and obey the locking convention, regardless of who wrote the code or where it resides. § A better approach is … not to lock 18

A Lock-free Implementation tsigas@cs. chalmers. se Philippas Tsigas 19

How did it start? � ”Synchronization is an enforcing mechanism used to impose constraints on the order of execution of threads. . Synchronization is used to coordinate threads execution and manage shared data. ” Does it have to be like that? When we share data do we have to impose constraints on the execution of threads?

HOW SAFE IS IT: LET US START FROM THE BEGINING © Philippas Tsigas 21

Shared Abstract Data Types § Object in memory Op B - Supports some set of operations (ADT) - Concurrent access by many processes/threads - Useful to e. g. Op A v Exchange data between threads v Coordinate thread activities P 1 P 2 Op Op P 3 B B Op A P 4 22

Executing Operations invocation response P 1 P 2 P 3 Borrowed from H. Attiya 23

Interleaving Operations Concurrent execution 24

Interleaving Operations (External) behavior 25 25

Interleaving Operations, or Not Sequential execution 26 26

Interleaving Operations, or Not Sequential behavior: invocations & response alternate and match (on process & object) Sequential specification: All the legal sequential behaviors, satisfying the semantics of the ADT - E. g. , for a (LIFO) stack: pop returns the last item pushed 27

Correctness: Sequential consistency [Lamport, 1979] § For every concurrent execution there is a sequential execution that - Contains the same operations - Is legal (obeys the sequential specification) - Preserves the order of operations by the same process 28

Sequential Consistency: Examples Concurrent (LIFO) stack push(4) push(7) pop(): 4 Last In push(4) push(7) pop(): 4 First Out 29 29

Sequential Consistency: Examples Concurrent (LIFO) stack push(4) push(7) pop(): 7 Last In First Out 30 30

Safety: Linearizability § Linearizable ADTs - Sequential specification defines legal sequential executions - Concurrent operations allowed to be interleaved - Operations appear to execute atomically v External observer gets the illusion that each operation takes effect instantaneously at some point between its invocation and its response(preserves order of all push(4) operation) T 1 push(7) concurrent LIFO stack pop(): 4 time T 2 Last In First Out 31

Safety II An accessible node is never freed. 32

Liveness Non-blocking implementations - Wait-free implementation of an ADT [Lamport, 1977] v Every operation finishes in a finite number of its own steps. - Lock-free (≠ FREE of LOCKS) implementation [Lamport, 1977] v At least one operation (from a set of concurrent operation) finishes in a finite number of steps (the data structure as a system always make progress) 33

Liveness II § every garbage node is eventually collected 34

Abstract Data Types (ADT) q Cover most concurrent applications q At least encapsulate their data needs q An object-oriented programming point of view q Abstract representation of data & set of methods (operations) for accessing it q Signature q Specification data 35

Implementing High-Level ADT Using lower-level ADTs & procedures data ------------------------------------------------------------------ data 36

Lower-Level Operations q High-level operations translate into primitives on base objects that are available on H/W q Obvious: read, write q Common: compare&swap (CAS), LL/SC, FAA 37

CAN I FIND A JOB IF I STUDY THIS? © Philippas Tsigas 38

8 Feb 2002 Release of NOBLE version 1. 0 23 Jan 2002 Expert Group Formation (JSR: Java Concurrency Utilities) 8 Jan 2004 JSR first Release 29 Aug 2006 INTEL’s TBB release 1. 0

ERLANG § § § § § OTP_R 15 A: R 15 pre-release Written by Kenneth, 23 Nov 2011 We have recently pushed a new master to Git. Hub tagged OTP_R 15 A. This is a stabilized snapshot of the current R 15 development (to be released as R 15 B on December 14: th) which, among other things, includes: OTP-9468 'Line numbers in exceptions' OTP-9451 'Parallel make' OTP-4779 A new GUI for Observer. Integrating pman, etop and tv into observer with tracing facilities. OTP-7775 A number of memory allocation optimizations have been implemented. Most optimizations reduce contention caused by synchronization between threads during allocation and deallocation of memory. Most notably: Synchronization of memory management in scheduler specific allocator instances has been rewritten to use lock-free synchronization. Synchronization of memory management in scheduler specific pre-allocators has been rewritten to use lock-free synchronization. The 'mseg_alloc' memory segment allocator now use scheduler specific instances instead of one instance. Apart from reducing contention this also ensures that memory allocators always create memory segments on the local NUMA node on a NUMA system. § § § § © Philippas Tsigas § OTP-9632 An ERTS internal, generic, many to one, lock-free queue for communication between threads has been introduced. The many to one scenario is very common in ERTS, so it can be used in a lot of places in the future. Currently it is used by scheduling of certain jobs, and the async thread pool, but more uses are planned for the future. Drivers using the driver_async functionality are not automatically locked to the system anymore, and can be unloaded as any dynamically linked in driver. Scheduling of ready async jobs is now also interleaved in between other jobs. Previously all ready async jobs were performed at once. OTP-9631 The ERTS internal system block functionality has been replaced by new functionality for blocking the system. The old system block functionality had contention issues and complexity issues. The new functionality piggy-backs on thread progress tracking functionality needed by newly introduced lock-free synchronization in the runtime system. When the functionality for blocking the system isn't used, there is more or less no overhead at all. This since the functionality for tracking thread progress is there and needed anyway. . and much more. This is not a full release of R 15 but rather a pre-release. Feel free to try our R 15 A release and get back to us with your findings. Your feedback is important to us and highly welcomed. Regards, The OTP Team 40

Locks are not supported Not in CUDA, not in Open. CL � Fairness of hardware scheduler unknown � Thread block holding a lock might be swapped out indefinitely, for example

No Fairness Guarantees … … while(atomic. CAS(&lock, 0, 1)); ctr++; lock = 0; … Thread holding lock is never scheduled!

Where do we stand at?