Threaded Programming Giuseppe Avolio CERN Outline l Concurrency

Threaded Programming Giuseppe Avolio (CERN)

Outline l Concurrency l l A look at the HW side l l What and why Where the need of synchronization comes from Writing concurrent software l l The need of a memory model Data organization l l Building blocks l l Object modelling (Some) synchronization primitives and atomics Tasks and thread pools ISOTDAQ 2015 Giuseppe Avolio - CERN 2

The Verbum Correctness comes first!!! ISOTDAQ 2015 Giuseppe Avolio - CERN 3

Concurrency and Threads INTRODUCTION ISOTDAQ 2015 Giuseppe Avolio - CERN 4

Concurrency, What? When we talk about concurrency in terms of computers, we mean a single system performing multiple independent activities in parallel, rather than sequentially, or one after the other. (Anthony Williams, C++ Concurrency in Action) Task switching on a single core computer Parallelism on a dual-core computer ISOTDAQ 2015 Giuseppe Avolio - CERN 5

Concurrency, Why? Writing correct programs is hard; writing correct concurrent programs is harder. […] So, why do we bother with concurrency? (Brian Goetz, Java Concurrency in Practice) Concurrency ISOTDAQ 2015 Separation of concerns Group related piece of code together Performance (throughput) Break a task into small pieces and run them in parallel Giuseppe Avolio - CERN Identify and separate areas of functionality Perform the same operation on several data sets concurrently 6

Concurrency, Why? Source: Intel IDF 2014 No more for-free faster code ISOTDAQ 2015 Design software for concurrency Giuseppe Avolio - CERN 7

Concurrency’s Flavors l Multiple processes l l l Separate applications running at the same time Messages can be exchanged using the inter-process mechanism provided by the Operating System (signals, sockets, files…) Cons l l l Inter-process is usually complicated or slow Overhead: duplicating resources needed by the OS and the application itself Process 1 Thread Operating System Inter-process Thread Process 2 Pros l l Easier to write correct concurrent code Processes can be spawn on different nodes connected over a network ISOTDAQ 2015 Giuseppe Avolio - CERN 8

Concurrency’s Flavors l Multiple threads l Threads are often called lightweight processes l l Resources l l A process may run one or several threads Thread 1 Shared global memory address space (i. e. , access to the same variables) But its thread has its own stack and local variables Each thread can be executed simultaneously and asynchronously with respect to the other ones Lower overhead with respect to running multiple processes ISOTDAQ 2015 Giuseppe Avolio - CERN Shared Memory Thread 2 Process 9

Risks of Threads Safety Liveness Performance The execution’s order of threads is unpredictable Deadlock: all the threads wait for the same resource Service time Responsiveness A thread may access or modify variables another thread is using Starvation: a thread is perpetually denied access to a resource it needs in order to make progress Throughput Resources Access to shared data must be coordinated: synchronization ISOTDAQ 2015 Livelock: a thread keeps retrying an operation that will always fail Giuseppe Avolio - CERN Scalability 10

Program Speedup "Amdahls. Law" by Daniels 220 at English Wikipedia - Own work based on: File: Amdahls. Law. png. Licensed under CC BY-SA 3. 0 via Wikimedia Commons http: //commons. wikimedia. org/wiki/File: Amdahls. Law. svg#mediaviewer/File: Amdahls. Law. svg ISOTDAQ 2015 Giuseppe Avolio - CERN 11

Multithreading from the HW’s point of view INSIDE THE CPU ISOTDAQ 2015 Giuseppe Avolio - CERN 12

A Modern CPU Intel 8 -Core i 7 CPU l CPUs are much faster than memory systems Source: http: //images. anandtech. com/doci/8426/HSWE%20 Die%20 Mapping%20 Hi-Res. jpg l l Several level of caches are used in order to store hot data l l ISOTDAQ 2015 Hundreds CPU cycles are needed to access the main memory (RAM) The shown CPU has two dedicated levels of cache (L 1 and L 2) for each core and a shared third level (L 3) Access to caches is much faster than access to the main memory Giuseppe Avolio - CERN 13

CPU’s Caches l l Caches play an important role in order to understand what happens when different threads access the same date Caches are read in blocks called cache lines l l Usually a power of two in size, from 16 to 256 bytes All the CPUs must maintain a coherent view of the data carried in the cache lines l l Coherence protocols exit to prevent inconsistent data Messages are defined to coordinate the movement of cache lines through the system ISOTDAQ 2015 Giuseppe Avolio - CERN 14

"Diagrama MESI" by Jugones 55 - Own work. Licensed under GFDL via Wikimedia Commons http: //commons. wikimedia. org/wiki/File: Diagrama_MESI. GIF#mediaviewer/File : Diagrama_MESI. GIF The MESI Protocol ISOTDAQ 2015 A Cache Line Can be in Four Possible States Modified The local processor has modified the cache line. This also implies it is the only copy in any cache Exclusive The cache line is not modified but known to not be loaded into any other processor’s cache Shared The cache line is not modified and might exist in another processor’s cache Invalid The cache line is invalid (unused) Giuseppe Avolio - CERN 15

Execution Reordering MESI is a complex protocol l l Details not needed for our purposes Focus just on one aspect l l When a CPU needs to write to a cache line which it does not hold, it has to message the other CPU and wait for the reply (containing the needed cache line) Latency! Add buffers l Write CPU 1 Inva lidat e le now k c A ent m dge Store buffer l l CPU 0 Stall l CPU 0 will not wait for the write action to be executed Invalidate queue l ISOTDAQ 2015 CPU 1 will answer to CPU 1 almost immediately and will queue the invalidate message Giuseppe Avolio - CERN Execution reordering!!! 16

Execution Reordering // The cache line containing // a resides on CPU 1 int a = 0; • a is not in the cache, write to the store buffer CPU 0 and transmits an invalidate message to CPU 1 // The cache line containing // b resides on CPU 0 int b = 0; • b not in the cache, ask to CPU 0 CPU 1 • the invalidate message received and queued // Executed by CPU 0 void foo() { a = 1; b = 1; } • b is in the cache, just write to it (b = 1) CPU 0 • send the value of b to CPU 1 // Executed by CPU 1 void bar() { while(b == 0) continue; assert(a == 1); } ISOTDAQ 2015 • value of b received from CPU 0 CPU 1 • while loop exits and the assertion fails CPU 1 sees the write to a to happen after the write to b Synchronization is needed!!! Giuseppe Avolio - CERN 17

Fixed Code // The cache line containing a resides on CPU 1 int a = 0; // The cache line containing b resides on CPU 0 int b = 0; // Executed by CPU 0 void foo() { a = 1; // Add a memory barrier: flush the store buffer before going on smb_mb(); b = 1; } // Executed by CPU 1 void bar() { while(b == 0) continue; // Add a memory barrier: wait for the invalidate queue to be empty smb_mb(); assert(a == 1); } ISOTDAQ 2015 Giuseppe Avolio - CERN 18

Few Notes l The computer does not execute the code you wrote! l l l Reordering is allowed just because it is extremely efficient! Adding proper synchronization let your code be independent from any code “transformation” or scheduling issue l l Compiler, processor (and caches) produce a better code! Sequential consistency or race condition free Memory barriers are very low level l Programming languages offer synchronization primitives at an higher level of abstraction ISOTDAQ 2015 Giuseppe Avolio - CERN 19

Multiple Sockets l Non-Uniform Memory Access (NUMA) l l Source: http: //www. ni. com/white-paper/11266/en/ ISOTDAQ 2015 Each CPU has its own dedicated memory CPUs are inter-connected via high speed links Messages between two different CPUs have higher latency than messages between two cores on the same socket Your multi-threaded program may need to be NUMA-aware l Beyond the scope of this lecture Giuseppe Avolio - CERN 20

Hints about how to write concurrent multi-threaded programs SOFTWARE ISOTDAQ 2015 Giuseppe Avolio - CERN 21

Memory Model l The execution of a program is strongly linked to the HW it runs on l l Do we really want our code to de dependent on any deep change in the HW model? l l l What if MESI is replaced by some different protocol What if additional buffers are added (like the store buffer and the invalidate queue? ) It has been realized that programming languages need a memory model l l Remind caches and MESI? The memory model can be considered as an abstraction of the HW layer The programmer is exposed only the programming language’s memory model Following the memory model’s rules, the code will be race condition free regardless of the architecture it is executed on Java and C++11 have a memory model ISOTDAQ 2015 Giuseppe Avolio - CERN 22

Memory Model l l A memory (consistency) model specifies the values that a shared variable read in a multithreaded program is allowed to return A memory model describes l l l How memory reads and writes may be executed by a processor relative to their program order How writes by one processor may become visible to other processors A memory model is an arbitration mechanism to determine how multiple threads access shared objects in memory ISOTDAQ 2015 Giuseppe Avolio - CERN 23

Memory Model l Three fundamental properties l Atomicity l Operations that are executed without interruption l Ordering l. A memory model determines what re-orderings are possible (relatively to program order) l Visibility l Determines when other threads will see changes made by the current thread ISOTDAQ 2015 Giuseppe Avolio - CERN 24

Organize Your Data You cannot ensure thread safety without understanding an object’s invariants and post-conditions. […] Encapsulating data within an object confines access to the data to the object’s methods, making it easier to ensure that the data is always accessed with the appropriate lock held. […] Confinement makes it easier to build thread-safe classes because a class that confines its state can be analyzed for thread safety without having to examine the whole program. (Brian Goetz, Java Concurrency in Practice) ISOTDAQ 2015 Giuseppe Avolio - CERN 25

Organize Your Data Design thread-safe classes Identify the variables that form the object’s state Identify the invariants that constrain the state variables Establish a policy for managing concurrent access to the object’s state ISOTDAQ 2015 Giuseppe Avolio - CERN 26

Mutexes Class Date { public: Date() : day(1), month(1), year(1970) {} l l void set(unsigned int d, unsigned int m, unsigned int y) { // Validity checks are omitted // The mutex is unlocked by the lock_guard’s destructor std: : lock_guard<std: : mutex> lk(m_mutex); m_day = d; m_month = m; m_year = y; } std: : array<unsigned int, 3> get() { std: : lock_guard<std: : mutex> lk(m_mutex ); return std: : array<int, 3 >{ {m_day, m_month, m_year} }; } // Would make sense to have separate getters/setters for // day, month and year? private: unsigned int m_day; unsigned int m_month; unsigned int m_year; std: : mutex m_mutex; // Why only one mutex? l Mutex stands for Mutual Exclusion When a thread locks a specific mutex, all other threads that try to lock the same mutex have to wait until the mutex is unlocked Self consistent view of the shared data Do it better: look for Read/Write mutexes }; ISOTDAQ 2015 Giuseppe Avolio - CERN 27

Deadlocks If thread A locks M 1 and tries to lock M 2, and thread B locks M 2 and tries to lock M 1, then both threads are locked forever Guidelines • Avoid nested locks • Avoid calling usersupplied code while holding a lock • Acquire locks in a fixed order // Solution provided in C++11 (nested locks) void lock. Multiple() { // std: : defer_lock tells the unique lock to // not acquire the lock on the mutex std: : unique_lock<std: : mutex> lk 1(m 1, std: : defer_lock); std: : unique_lock<std: : mutex> lk 2(m 2, std: : defer_lock); // std: : lock properly acquires the lock on the // mutexes avoiding deadlocks std: : lock(lk 1, lk 2); // When are the locks released? } ISOTDAQ 2015 Giuseppe Avolio - CERN 28

Condition Variables l std: : condition_variable cv; std: : mutex cv_m; int i = 0; void wait_thread() { // A condition variable is always associated to a mutex // that protects the access to the “condition” data std: : unique_lock<std: : mutex> lk(cv_m); // On notification the condition is checked and the lock // on the mutex is release or re-acquired atomically cv. wait(lk, []{return i == 1; }); std: : cerr << ". . . finished waiting. i == 1n"; } l A condition variable is associated with some event or other condition, and one or more threads can wait for that condition to be satisfied When the condition is verified, a thread can then notify one or more threads waiting on the condition variable void signal_thread() { std: : this_thread: : sleep_for(std: : chrono: : seconds(1 )); { std: : lock_guard<std: : mutex> lk(cv_m); i = 1; std: : cerr << "Notifying. . . n"; } // This notifies any waiting thread about a change in the “status” of the “condition” cv. notify_all(); } ISOTDAQ 2015 Giuseppe Avolio - CERN 29

Atomic Operations // Simple and completely unsafe counter public class Unsafe. Sequence { private int value = 0; public int get. Next() { // What if two threads execute this method concurrently? // Is the code broken even if the method is executed in two different moments? // Problems here are both atomicity and visibility return value++; } } Thread A Thread B ISOTDAQ 2015 value is 9 value is 10 9+1 value is 9 9+1 Giuseppe Avolio - CERN value is 10 30

Atomic Operations // Simple and completely safe counter public class Safe. Sequence { private Atomic. Integer value = new Atomic. Integer(); public int get. Next() { // Atomic types fix both the atomicity and visibility issues return value. increment. And. Get(); } } l An atomic operation is an indivisible operation l l Atomic variables are finer-grained and lighterweight than locks l l l Such an operation cannot be observed half-done by any thread The scope of contention is limited to a single variable Atomics are at least as fast as acquiring a lock The memory model defines the properties of atomic variables ISOTDAQ 2015 Giuseppe Avolio - CERN 31

CAS Operations // Safe counter using Compare And Swap (CAS) public class CASSafe. Sequence { private Atomic. Integer value = new Atomic. Integer(); l CAS is the building block of several lockless algorithms Be aware of the ABA problem! public int get. Next() { l int v; do { v = value. get(); } // compare. And. Swap’s arguments: // 1 - The expected old value of the variable // 2 - The new value of the variable // It returns the current value of the variable // The variable is updated to the new value only if its current // value matches the expected old value while(v != value. compare. And. Swap(v, v + 1)); return v + 1; } } ISOTDAQ 2015 Giuseppe Avolio - CERN 32

Lazy Initialization // Broken double-checked locking // Goal: create a single instance of an object in the most safe and efficient way public class Double. Checked. Locking { private static Resource resource; public static Resource get. Instance() { // Just check that the resource has not been created yet // Note that here no synchronization at all is used if(resource == null) { // If the resource does not exist yet, then acquire a lock // and create it. Since the lock is acquired we are sure that // no other thread is allowed to create an additional instance synchronized(Double. Checked. Locking. class) { if(resource == null) { resource = new Resource(); } } } return resource; } } // This widely used algorithm is broken, why? ISOTDAQ 2015 Giuseppe Avolio - CERN 33

Lazy Initialization - Fixed // Goal: create a single instance of an object in the most safe and efficient way public class Resource. Factory { private static class Resource. Holder { public final static Resource resource = new Resource(); } public static Resource get. Instance() { return Resource. Holder. resource; } } // The Java memory model defines properties of static variables // Note that no explicit locking is performed!!! // A very similar solution is also available in C++11 // Note: he previously shown double-checked algorithm can be fixed in java // using “volatile” (homework) ISOTDAQ 2015 Giuseppe Avolio - CERN 34

TASK BASED PROGRAMMING ISOTDAQ 2015 Giuseppe Avolio - CERN 35

Why Tasks Divide the application into abstract, discrete units of work • Simplifies program organization • Facilitates error recovery by providing natural transaction boundaries • Promotes concurrency by providing a natural structure for parallelizing work ISOTDAQ 2015 Giuseppe Avolio - CERN 36

A Thread per Task class Thread. Per. Task. Web. Server { public static void main(String[] args) throws IOException { final Server. Socket = new Server. Socket(80); while(true) { final Socket connection = socket. accept(); // A task is created for the new client connection Runnable task = new Runnable() { public void run() { // This is the actual code // handling the request handle. Request(connection) ; } }; // A new thread is started for each task new Thread(task). start(); } } } ISOTDAQ 2015 Giuseppe Avolio - CERN l Better than a serial implementation but: l l l Thread lifecycle overhead Resource consumption Stability 37

Using Thread Pools class Thread. Per. Task. Web. Server { private static final int NTHREADS = 100; private static final Executor exec = Executors. new. Fixed. Thread. Pool(NTHREADS); public static void main(String[] args) throws IOException { final Server. Socket = new Server. Socket(80); while(true) { final Socket connection = socket. accept(); // A task is created for the new client connection Runnable task = new Runnable() { public void run() { // This is the actual code // handling the request handle. Request(connection) ; } }; // Tasks are submitted to the thread pool // Usually put in some queue and then processes // by the worker threads execute(task); } } l l Thread pools help in efficiently exploiting the parallelism offered by the system Unlikely no thread pools in C++11 l Do it by yourself… } ISOTDAQ 2015 Giuseppe Avolio - CERN 38

Using Thread Pools l l Organize your tasks so that they are independent Consider singlethreaded pools l l Thread safety policies can be relaxed because tasks are not executed concurrently l Evaluate your thread pool policies l How many working threads l l Unlimited, fixed What about the working queue? l Fixed, unbounded Try to keep the tasks short in order to not cause starvation ISOTDAQ 2015 Giuseppe Avolio - CERN 39

(My) General Guidelines Simple Rules to Start with Multi-threading Programming • Start thinking in multi-threading since the very beginning of your project • Don’t share data • If you need to share then synchronize access to shared data • Evaluate data messaging as a different way of sharing • Usually requires copying memory • Properly organize your data • Write safe objects • Keep it as simple as possible • In other words: try the super-mega efficient-performant ultra-tricky solution only if needed • In other words: optimize only after measurements • Look for already existent solutions before writing a new-old thread algorithm or data structure • Measure, measure… ISOTDAQ 2015 Giuseppe Avolio - CERN 40

Conclusions Parallel programming is a lot of fun but it is not easy l Goal of this lecture was to give a broad overview of the parallel world l l l Just the top of the iceberg And remember l Correctness first!!! ISOTDAQ 2015 Giuseppe Avolio - CERN 41

For further studies REFERENCES ISOTDAQ 2015 Giuseppe Avolio - CERN 42

References Author Title Brian Goetz Java Concurrency in Practice Anthony Williams C++ Concurrency in Action - Practical Multithreading Bjarne Stroustrupt The C++ Programming Language - Fourth Edition Intel Guide for Developing Multithreaded Applications (web) Intel Threading Building Blocks (web) National Instruments Top Eight Features of the Intel Core i 7 Processors for Test, Measurement, and Control (web) Paul E. Mc. Kenney Memory Barriers: a Hardware View for Software Hackers (web) Ulrich Drepper What Every Programmer Should Know About Memory (web) ISOTDAQ 2015 Giuseppe Avolio - CERN 43

References Author Title Hans-J. Boehm Threads Cannot be Implemented as a Library (web) Scott Meyers and Andrei Alexandrescu C++ and the Perils of Double-Checked Locking (web) Herber Sutter atomic<> Weapons - The C++11 Memory Model and Modern Hardware (web) pragmateek. com Synchronization, memory visibility and leaky abstractions (web) The Java memory model (web) ISOTDAQ 2015 Giuseppe Avolio - CERN 44

You May Want to Look at… l Cache related issues False sharing l Cache misses l l More mutexe’s flavor Read/write l Recursive l Spin locks l l l Futures and promises How to properly interrupt tasks (and threads) ISOTDAQ 2015 Giuseppe Avolio - CERN 45