Barrier Synchronization Nir Shavit Multiprocessor Synchronization Spring 2003

Barrier Synchronization Nir Shavit Multiprocessor Synchronization Spring 2003 M. Herlihy & N. Shavit (c) 2003

Ideal Parallel Computation 0 0 0 1 1 1 9/9/2021 M. Herlihy & N. Shavit (c) 2003 2

Ideal Parallel Computation 2 2 2 1 1 1 9/9/2021 M. Herlihy & N. Shavit (c) 2003 3

Real-Life Parallel Computation 0 0 0 zzz… 9/9/2021 M. Herlihy & N. Shavit (c) 2003 1 1 4

Real-Life Parallel Computation 2 0 zzz… 1 1 Uh, oh 9/9/2021 M. Herlihy & N. Shavit (c) 2003 5

Barrier Synchronization 0 0 barrier 0 9/9/2021 M. Herlihy & N. Shavit (c) 2003 6

Barrier Synchronization barrier 1 1 1 9/9/2021 M. Herlihy & N. Shavit (c) 2003 7

Barrier Synchronization barrier Until every thread has left here No thread enters here 9/9/2021 M. Herlihy & N. Shavit (c) 2003 8

Why Do We Care? • Mostly of interest to – Scientific & numeric computation • Elsewhere – Garbage collection – Rare in systems programming 9/9/2021 M. Herlihy & N. Shavit (c) 2003 9

Barriers public class Barrier { int count; int size; public Barrier(int n){ this. size = this. count = n; } public void await() { if (count. Fetch. Dec()==1) { this. count = this. size; } else { while (this. count != 0) {} }}}} 9/9/2021 M. Herlihy & N. Shavit (c) 2003 10

Barriers public class Barrier { int size; Number threads int count; participating public Barrier(int n){ this. size = this. count = n; } public void await() { if (count. Fetch. Dec()==1) { this. count = this. size; } else { while (this. count != 0) {} }}}} 9/9/2021 M. Herlihy & N. Shavit (c) 2003 11

Barriers public class Barrier { int size; Number threads int count; not yet arrived public Barrier(int n){ this. size = this. count = n; } public void await() { if (count. Fetch. Dec()==1) { this. count = this. size; } else { while (this. count != 0) {} }}}} 9/9/2021 M. Herlihy & N. Shavit (c) 2003 12

Barriers public class Barrier { int size; Initialization int count; public Barrier(int n){ this. size = this. count = n; } public void await() { if (count. Fetch. Dec()==1) { this. count = this. size; } else { while (this. count != 0) {} }}}} 9/9/2021 M. Herlihy & N. Shavit (c) 2003 13

Barriers public class Barrier { int size; The method int count; public Barrier(int n){ this. size = this. count = n; } public void await() { if (count. Fetch. Dec()==1) { this. count = this. size; } else { while (this. count != 0) {} }}}} 9/9/2021 M. Herlihy & N. Shavit (c) 2003 14

Barriers public class Barrier { int size; If I’m last, reset int count; everything public Barrier(int n){ this. size = this. count = n; } public void await() { if (count. Fetch. Dec()==1) { this. count = this. size; } else { while (this. count != 0) {} }}}} 9/9/2021 M. Herlihy & N. Shavit (c) 2003 15

Barriers public class Barrier { int size; Wait for rest to int count; catch up public Barrier(int n){ this. size = this. count = n; } public void await() { if (count. Fetch. Dec()==1) { this. count = this. size; } else { while (this. count != 0) {} }}}} 9/9/2021 M. Herlihy & N. Shavit (c) 2003 16

Barriers public class Barrier { int count; int size; public Barrier(int n){ this. size = this. count = n; } What’s wrong with this protocol? public void await() { if (count. Fetch. Dec()==1) { this. count = this. size; } else { while (this. count != 0) {} }}}} 9/9/2021 M. Herlihy & N. Shavit (c) 2003 17

Reuse Barrier b = new Barrier(n); while ( mumble() ) { Do work(); repeat synchronize b. await() } 9/9/2021 M. Herlihy & N. Shavit (c) 2003 18

Barriers public class Barrier { int count; int size; public Barrier(int n){ Waiting for this. size = this. count = n; Phase 1 to finish } public void await() { if (count. Fetch. Dec()==1) { this. count = this. size; } else { while (this. count != 0) {} }}}} 9/9/2021 M. Herlihy & N. Shavit (c) 2003 19

Barriers public class Barrier { int count; Phase 1 int size; is so over public Barrier(int n){ Waiting for this. size = this. count = n; Phase 1 to finish } public void await() { if (count. Fetch. Dec()==1) { this. count = this. size; } else { while (this. count != 0) {} }}}} 9/9/2021 M. Herlihy & N. Shavit (c) 2003 20

Barriers public class Barrier { int count; Prepare for int size; phase 2 public Barrier(int n){ ZZZZZ…. this. size = this. count = n; } public void await() { if (count. Fetch. Dec()==1) { this. count = this. size; } else { while (this. count != 0) {} }}}} 9/9/2021 M. Herlihy & N. Shavit (c) 2003 21

Barriers public class Barrier { Waiting for int count; Phase 2 to finish int size; public Barrier(int n){ Waiting for this. size = this. count = n; Phase 1 to finish } public void await() { if (count. Fetch. Inc()==1) { this. count = this. size; } else { while (this. count != 0) {} }}}} 9/9/2021 M. Herlihy & N. Shavit (c) 2003 22

Basic Problem • One thread “wraps around” to start phase 2 • While another thread is still waiting for phase 1 • One solution: – Always use two barriers 9/9/2021 M. Herlihy & N. Shavit (c) 2003 23

Sense-Reversing Barriers public class Barrier { int count, size; boolean sense = true; public void await(boolean my. Sense) { if (count. Fetch. Dec()==1) { this. count = this. size; this. sense = my. Sense } else { while (this. sense != my. Sense) {} }}}} 9/9/2021 M. Herlihy & N. Shavit (c) 2003 24

Sense-Reversing Barriers public class Barrier { int count, size; boolean sense = true; Completed odd or even-numbered phase? public void await(boolean my. Sense) { if (count. Fetch. Dec()==1) { this. count = this. size; this. sense = my. Sense } else { while (this. sense != my. Sense) {} }}}} 9/9/2021 M. Herlihy & N. Shavit (c) 2003 25

Sense-Reversing Barriers public class Barrier { Thread working on odd or int count, size; even-numbered phase? boolean sense = true; public void await(boolean my. Sense) { if (count. Fetch. Dec()==1) { this. count = this. size; this. sense = my. Sense } else { while (this. sense != my. Sense) {} }}}} 9/9/2021 M. Herlihy & N. Shavit (c) 2003 26

Sense-Reversing Barriers public class Barrier { If I’m last, reverse the int count, size; boolean sense = true; sense on the way out public void await(boolean my. Sense) { if (count. Fetch. Dec()==1) { this. count = this. size; this. sense = my. Sense } else { while (this. sense != my. Sense) {} }}}} 9/9/2021 M. Herlihy & N. Shavit (c) 2003 27

Sense-Reversing Barriers public class Barrier { int count, size; boolean sense = true; Otherwise, wait for sense to flip public void await(boolean my. Sense) { if (count. Fetch. Dec()==1) { this. count = this. size; this. sense = my. Sense } else { while (this. sense != my. Sense) {} }}}} 9/9/2021 M. Herlihy & N. Shavit (c) 2003 28

Combining Tree Barriers 2 -barrier 9/9/2021 2 -barrier M. Herlihy & N. Shavit (c) 2003 29

Combining Tree Barriers 2 -barrier 9/9/2021 2 -barrier M. Herlihy & N. Shavit (c) 2003 30

Combining Tree Barrier public class CBarrier { int count, size; CBarrier parent; public void await(boolean my. Sense) { if (this. count. Fetch. Dec()==1){ if (this. parent != null) this. parent. await(); this. count = this. size; this. sense = my. Sense } else { while (this. sense != my. Sense) {} }}}} 9/9/2021 M. Herlihy & N. Shavit (c) 2003 31

Combining Tree Barrier public class CBarrier { int count, size; CBarrier parent; Parent barrier in tree public void await(boolean my. Sense) { if (this. count. Fetch. Dec()==1){ if (this. parent != null) this. parent. await(); this. count = this. size; this. sense = my. Sense } else { while (this. sense != my. Sense) {} }}}} 9/9/2021 M. Herlihy & N. Shavit (c) 2003 32

Combining Tree Barrier public class CBarrier { Thread working on odd or int count, size; even-numbered phase? CBarrier parent; public void await(boolean my. Sense) { if (this. count. Fetch. Dec()==1){ if (this. parent != null) this. parent. await(); this. count = this. size; this. sense = my. Sense } else { while (this. sense != my. Sense) {} }}}} 9/9/2021 M. Herlihy & N. Shavit (c) 2003 33

Combining Tree Barrier public class CBarrier { int count, size; CBarrier parent; Am I last? public void await(boolean my. Sense) { if (this. count. Fetch. Dec()==1){ if (this. parent != null) this. parent. await(); this. count = this. size; this. sense = my. Sense } else { while (this. sense != my. Sense) {} }}}} 9/9/2021 M. Herlihy & N. Shavit (c) 2003 34

Combining Tree Barrier public class CBarrier { Wait int count, size; CBarrier parent; on parent barrier, if any public void await(boolean my. Sense){ if (this. count. Fetch. Dec()==1){ if (this. parent != null) this. parent. await(); this. count = this. size; this. sense = my. Sense } else { while (this. sense != my. Sense) {} }}}} 9/9/2021 M. Herlihy & N. Shavit (c) 2003 35

Combining Tree Barrier public class CBarrier { Prepare int count, size; CBarrier parent; for next phase public void await(boolean my. Sense){ if (this. count. Fetch. Dec()==1){ if (this. parent != null) this. parent. await(); this. count = this. size; this. sense = my. Sense } else { while (this. sense != my. Sense) {} }}}} 9/9/2021 M. Herlihy & N. Shavit (c) 2003 36

Combining Tree Barrier public class CBarrier { Wake int count, size; CBarrier parent; up others at this level public void await(boolean my. Sense) { if (this. count. Fetch. Dec()==1){ if (this. parent != null) this. parent. await(); this. count = this. size; this. sense = my. Sense } else { while (this. sense != my. Sense) {} }}}} 9/9/2021 M. Herlihy & N. Shavit (c) 2003 37

Combining Tree Barrier public class CBarrier { I’m not int count, size; CBarrier parent; last, wait for release public void await(boolean my. Sense) { if (this. count. Fetch. Dec()==1){ if (this. parent != null) this. parent. await(); this. count = this. size; this. sense = my. Sense } else { while (this. sense != my. Sense) {} }}}} 9/9/2021 M. Herlihy & N. Shavit (c) 2003 38

Combining Tree Barrier • No sequential bottleneck – Fetch. Dec calls proceed in parallel • Low memory contention – Same reason • Cache behavior – Local spinning on bus-based architecture – Not so good for distributed 9/9/2021 M. Herlihy & N. Shavit (c) 2003 39

Remarks • Everyone spins on sense field – Local spinning on bus-based (good) – Network hot-spot on distributed architecture (bad) • Sequential bottleneck – Sequence of Fetch. Inc() calls • Not really scalable 9/9/2021 M. Herlihy & N. Shavit (c) 2003 40

Tournament Tree Barrier • If tree nodes have fan-in 2 – Don’t need to call Fetch. Dec() – Winner chosen statically • At level i – If i-th bit of id is zero, move up – Otherwise keep back 9/9/2021 M. Herlihy & N. Shavit (c) 2003 41

Tournament Tree Barriers winner 9/9/2021 loser winner M. Herlihy & N. Shavit (c) 2003 loser 42

Tournament Tree Barriers All flags blue 9/9/2021 M. Herlihy & N. Shavit (c) 2003 43

Tournament Tree Barriers Loser thread sets winner’s flag 9/9/2021 M. Herlihy & N. Shavit (c) 2003 44

Tournament Tree Barriers Loser spins on own flag 9/9/2021 M. Herlihy & N. Shavit (c) 2003 45

Tournament Tree Barriers Winner spins on own flag 9/9/2021 M. Herlihy & N. Shavit (c) 2003 46

Tournament Tree Barriers Winner sees own flag, moves up, spins 9/9/2021 M. Herlihy & N. Shavit (c) 2003 47

Tournament Tree Barriers Bingo! 9/9/2021 M. Herlihy & N. Shavit (c) 2003 48

Tournament Tree Barriers Sense-reversing: next time use blue flags 9/9/2021 M. Herlihy & N. Shavit (c) 2003 49

Tournament Barrier class TBarrier { boolean flag; TBarrier partner; TBarrier parent; boolean top; … } 9/9/2021 M. Herlihy & N. Shavit (c) 2003 50

Tournament Barrier class TBarrier { boolean flag; TBarrier partner; TBarrier parent; boolean top; … } 9/9/2021 Notifications delivered here M. Herlihy & N. Shavit (c) 2003 51

Tournament Barrier class TBarrier { boolean flag; TBarrier partner; TBarrier parent; boolean top; … } 9/9/2021 Other thead at same level M. Herlihy & N. Shavit (c) 2003 52

Tournament Barrier class TBarrier { boolean flag; TBarrier partner; TBarrier parent; boolean top; … } 9/9/2021 Parent (winner) or null (loser) M. Herlihy & N. Shavit (c) 2003 53

Tournament Barrier class TBarrier { boolean flag; TBarrier partner; TBarrier parent; boolean top; … } 9/9/2021 Am I the root? M. Herlihy & N. Shavit (c) 2003 54

Tournament Barrier void await(boolean my. Sense) { if (this. top) { return; } else if (this. parent != null) { while (this. flag[round] != my. Sense) {}; this. parent. await(my. Sense); this. partner. flag[round] = mysense; } else { this. partner. flag[round] = my. Sense; while (this. flag[round] != my. Sense) {}; }}} 9/9/2021 M. Herlihy & N. Shavit (c) 2003 55

Tournament Barrier void await(boolean my. Sense) { if (this. top) { The root, c’est moi return; } else if (this. parent != null) { while (this. flag[round] != my. Sense) {}; this. parent. await(my. Sense); this. partner. flag[round] = mysense; } else { this. partner. flag[round] = my. Sense; while (this. flag[round] != my. Sense) {}; }}} 9/9/2021 M. Herlihy & N. Shavit (c) 2003 56

Tournament Barrier void await(boolean my. Sense) { I am already if (this. top) { winner return; } else if (this. parent != null) { while (this. flag[round] != my. Sense) {}; this. parent. await(my. Sense); this. partner. flag[round] = mysense; } else { this. partner. flag[round] = my. Sense; while (this. flag[round] != my. Sense) {}; }}} 9/9/2021 M. Herlihy & N. Shavit (c) 2003 a 57

Tournament Barrier void await(boolean my. Sense) { Wait for partner if (this. top) { show up return; } else if (this. parent != null) { while (this. flag[round] != my. Sense) {}; this. parent. await(my. Sense); this. partner. flag[round] = mysense; } else { this. partner. flag[round] = my. Sense; while (this. flag[round] != my. Sense) {}; }}} 9/9/2021 M. Herlihy & N. Shavit (c) 2003 to 58

Tournament Barrier void await(boolean my. Sense) { Synchronize upstairs if (this. top) { return; } else if (this. parent != null) { while (this. flag[round] != my. Sense) {}; this. parent. await(my. Sense); this. partner. flag[round] = mysense; } else { this. partner. flag[round] = my. Sense; while (this. flag[round] != my. Sense) {}; }}} 9/9/2021 M. Herlihy & N. Shavit (c) 2003 59

Tournament Barrier void await(boolean my. Sense) { Notify partner if (this. top) { return; } else if (this. parent != null) { while (this. flag[round] != my. Sense) {}; this. parent. await(my. Sense); this. partner. flag[round] = mysense; } else { this. partner. flag[round] = my. Sense; while (this. flag[round] != my. Sense) {}; }}} 9/9/2021 M. Herlihy & N. Shavit (c) 2003 60

Tournament Barrier void await(boolean my. Sense) { My thread is such if (this. top) { loser return; } else if (this. parent != null) { while (this. flag[round] != my. Sense) {}; this. parent. await(my. Sense); this. partner. flag[round] = mysense; } else { this. partner. flag[round] = my. Sense; while (this. flag[round] != my. Sense) {}; }}} 9/9/2021 M. Herlihy & N. Shavit (c) 2003 a 61

Tournament Barrier void await(boolean my. Sense) { Tell partner I’m if (this. top) { return; } else if (this. parent != null) { while (this. flag[round] != my. Sense) {}; this. parent. await(my. Sense); this. partner. flag[round] = mysense; } else { this. partner. flag[round] = my. Sense; while (this. flag[round] != my. Sense) {}; }}} 9/9/2021 M. Herlihy & N. Shavit (c) 2003 here 62

Tournament Barrier void await(boolean my. Sense) { Wait for partner if (this. top) { release me return; } else if (this. parent != null) { while (this. flag[round] != my. Sense) {}; this. parent. await(my. Sense); this. partner. flag[round] = mysense; } else { this. partner. flag[round] = my. Sense; while (this. flag[round] != my. Sense) {}; }}} 9/9/2021 M. Herlihy & N. Shavit (c) 2003 to 63

Remarks • No need for read-modify-write calls • Each thread spins on fixed location – Good for bus-based architectures – Good for distributed architectures 9/9/2021 M. Herlihy & N. Shavit (c) 2003 64

Dissemination Barrier • At round i – Thread A notifies thread A+2 i (mod n) • Requires log n rounds 9/9/2021 M. Herlihy & N. Shavit (c) 2003 65

Dissemination Barrier +1 9/9/2021 +2 M. Herlihy & N. Shavit (c) 2003 +4 66

Dissemination Details • Use two copies of fields – Avoids interference • Use sense-reversing – Avoid reinitializing fields • Thread Arguments – Parity of round – Sense (flips when round becomes 0) 9/9/2021 M. Herlihy & N. Shavit (c) 2003 67

Dissemination Barrier public class DBarrier { private boolean flags[2][logn]; private DBarrier partners[2][logn]; } 9/9/2021 M. Herlihy & N. Shavit (c) 2003 68

Dissemination Barrier public class DBarrier { private boolean flags[2][logn]; private DBarrier partners[2][logn]; } Use two copies to avoid interference 9/9/2021 M. Herlihy & N. Shavit (c) 2003 69

Dissemination Barrier public class DBarrier { private boolean flags[2][logn]; private DBarrier partners[2][logn]; } One element per round 9/9/2021 M. Herlihy & N. Shavit (c) 2003 70

Dissemination Barrier public class DBarrier { private boolean flags[2][logn]; private DBarrier partners[2][logn]; } 9/9/2021 M. Herlihy & N. Shavit (c) 2003 Notify me here 71

Dissemination Barrier public class DBarrier { private boolean flags[2][logn]; private DBarrier partners[2][logn]; } Barrier for thread A + 2 i 9/9/2021 M. Herlihy & N. Shavit (c) 2003 72

Dissemination Barrier void await(int parity, boolean my. Sense) { for (int i=0; i<logn; i++) { this. partner[parity]. flags[i] = my. Sense; while (this. flags[i] != my. Sense) {} } } 9/9/2021 M. Herlihy & N. Shavit (c) 2003 73

Dissemination Barrier void await(int parity, boolean my. Sense) { for (int i=0; i<logn; i++) { this. partner[parity]. flags[i] = my. Sense; while (this. flags[i] != my. Sense) {} } } Parity of current phase 9/9/2021 M. Herlihy & N. Shavit (c) 2003 74

Dissemination Barrier void await(int parity, boolean my. Sense) { for (int i=0; i<logn; i++) { this. partner[parity]. flags[i] = my. Sense; while (this. flags[i] != my. Sense) {} } } Flips when parity becomes 0 9/9/2021 M. Herlihy & N. Shavit (c) 2003 75

Dissemination Barrier void await(int parity, boolean my. Sense) { for (int i=0; i<logn; i++) { this. partner[parity]. flags[i] = my. Sense; while (this. flags[i] != my. Sense) {} } } Notify thread A+2 i 9/9/2021 M. Herlihy & N. Shavit (c) 2003 76

Dissemination Barrier void await(int parity, boolean my. Sense) { for (int i=0; i<logn; i++) { this. partner[parity]. flags[i] = my. Sense; while (this. flags[i] != my. Sense) {} } } Wait for thread A-2 i 9/9/2021 M. Herlihy & N. Shavit (c) 2003 77

Remarks • Every thread spins in the same place – Good for distributed implementations • Works even if n not a power of 2 9/9/2021 M. Herlihy & N. Shavit (c) 2003 78

Clip Art 9/9/2021 M. Herlihy & N. Shavit (c) 2003 79