The Relative Power of Synchronization Methods Nir Shavit

The Relative Power of Synchronization Methods Nir Shavit Multiprocessor Synchronization Spring 2003 11/29/2020 © 2003 Herlihy and Shavit 2

Wait-Free Implementation • Every method call completes in finite number of steps • Implies no mutual exclusion (2) © 2003 Herlihy and Shavit

Wait-Free Constructions • Wait-free atomic registers – From safe registers • Two-threaded FIFO queue – From atomic registers – And indirectly from safe registers © 2003 Herlihy and Shavit

Rationale • We wanted atomic registers to implement mutual exclusion • So we couldn’t use mutual exclusion to implement atomic registers • But wait, there’s more! © 2003 Herlihy and Shavit

Unexpected Delay ? ? ? Sw app bac ed ou t ka t 11/29/2020(2) © 2003 Herlihy and Shavit 6

Heterogeneous Architectures ? ? ? Pentium 11/29/2020(1) ? ? ? Pentium © 2003 Herlihy and Shavit yawn 286 7

Fault-Tolerance ? ? ? 11/29/2020(2) ? ? ? © 2003 Herlihy and Shavit 8

Basic Questions • Might be a good idea in principle • But how do you do it – Systematically? – Correctly? – Efficiently? © 2003 Herlihy and Shavit

Concurrent FIFO Queue 11/29/2020 © 2003 Herlihy and Shavit 10

Two-Thread Wait-Free Queue public class Wait. Free. Queue { int head = 0, tail = 0; Item[QSIZE] items; public void enq(Item x) { while (tail-head == QSIZE); // busy-wait items[tail++ % QSIZE] = x; } public Item deq() { while (tail == head); // busy-wait return items[head++ % QSIZE]; }} 11/29/2020 © 2003 Herlihy and Shavit 11

What About Multiple Dequeuers? 11/29/2020 © 2003 Herlihy and Shavit 12

Grand Challenge Only new • Implement a FIFO queue aspect – – Wait-free Linearizable From atomic read-write registers Multiple dequeuers (1) © 2003 Herlihy and Shavit

Consensus • While you are ruminating on the grand challenge… • We will give you another puzzle – Consensus – Pretty important … © 2003 Herlihy and Shavit

Consensus: Each Thread has a Private Input 32 19 © 2003 Herlihy and Shavit 21

They Communicate 11/29/2020 © 2003 Herlihy and Shavit 16

They Agree on Some Thread’s Input 19 19 © 2003 Herlihy and Shavit 19

Formally: Consensus Consistent: all threads decide the same value Valid: the common decision value is some thread's input Wait-free: each thread decides after a finite number of steps © 2003 Herlihy and Shavit

Theorem: No Consensus from Registers if any Thread can Halt ? ? ? © 2003 Herlihy and Shavit

Proof Strategy • Assume otherwise • Reason about the properties of any such protocol • Derive a contradiction • Quod Erat Demonstrandum © 2003 Herlihy and Shavit

Wait-Free Computation A moves B moves • Either A or B “moves” • Moving means – Register read – Register write © 2003 Herlihy and Shavit

The Two-Move Tree Final states (2) Initial state © 2003 Herlihy and Shavit

Decision Values 1 0 0 1 © 2003 Herlihy and Shavit 1 1

Bivalent: Both Possible bivalent 1 0 0 1 © 2003 Herlihy and Shavit 1 1

Univalent: Single Value Possible univalent 1 0 0 1 © 2003 Herlihy and Shavit 1 1

1 -valent: Only 1 Possible 1 -valent 1 0 0 1 © 2003 Herlihy and Shavit 1 1

0 -valent: Only 0 possible 0 -valent 1 0 0 1 © 2003 Herlihy and Shavit 1 1

Summary • Wait-free computation is a tree • Bivalent system states – Outcome not fixed • Univalent states – Outcome is fixed – May not be “known” yet • 1 -Valent and 0 -Valent states © 2003 Herlihy and Shavit

Claim Some initial system state is bivalent If A starts with input 0 and B starts with input 1, then initially A should be able to decide 0 and B should be able to decide 1. Lets see why? © 2003 Herlihy and Shavit

A 0 -Valent Initial State 0 0 • All executions lead to decision of 0 (2) © 2003 Herlihy and Shavit (1)

A 0 -Valent Initial State 0 • Solo execution by A also decides 0 (1) © 2003 Herlihy and Shavit (1)

A 1 -Valent Initial State 1 1 • All executions lead to decision of 1 (2) © 2003 Herlihy and Shavit (1)

A 1 -Valent Initial State 1 • Solo execution by B also decides 1 (1) © 2003 Herlihy and Shavit (1)

A Univalent Initial State? 0 1 • Can all executions lead to the same decision? (2) © 2003 Herlihy and Shavit

State is Bivalent 1 0 • Solo execution by A must decide 0 11/29/2020 • Solo execution by B must decide 1 © 2003 Herlihy and Shavit 35

Critical States critical 0 -valent (3) 1 -valent © 2003 Herlihy and Shavit (3)

Critical States • Starting from a bivalent initial state • The protocol can reach a critical state – Otherwise we could stay bivalent forever – And the protocol is not wait-free © 2003 Herlihy and Shavit

From a Critical State c 0 -valent If A goes first, protocol decides 0 1 -valent If B goes first, protocol decides 1 © 2003 Herlihy and Shavit

Model Dependency • So far, memory-independent! • True for – – Registers Message-passing Carrier pigeons Any kind of asynchronous computation © 2003 Herlihy and Shavit

What are the Threads Doing? • Reads and/or writes • To same/different registers © 2003 Herlihy and Shavit

Possible Interactions A reads x A reads y x. read() y. read() x. write() y. write() x. read() ? ? y. read() ? ? x. write() ? ? y. write() ? ? © 2003 Herlihy and Shavit

Reading Registers A runs solo, decides 0 c 0 1 States look the same to A B reads x n o i A runs solo, t c i decides 1 d a r t n o C © 2003 Herlihy and Shavit

Possible Interactions x. read() y. read() x. write() y. write() x. read() no no y. read() no no x. write() no no ? ? y. write() no no ? ? © 2003 Herlihy and Shavit

Writing Distinct Registers A writes y c B writes x A writesioyn B writes x 0 1 d a r t n o C same The song remains the © 2003 Herlihy and Shavit t ic

Possible Interactions x. read() y. read() x. write() y. write() x. read() no no y. read() no no x. write() no no ? no y. write() no no no ? © 2003 Herlihy and Shavit

Writing Same Registers A writes x c B writes x A runs solo, decides 0 A writes x 0 States look the same to A 1 d a A runs solo, decides 1 r t n o C © 2003 Herlihy and Shavit t ic n io

That’s All, Folks! x. read() y. read() x. write() y. write() x. read() no no y. read() no no x. write() no no y. write() no no © 2003 Herlihy and Shavit

Theorem • It is impossible to solve consensus using read/write atomic registers – – Assume protocol exists It has a bivalent initial state Must be able to reach a critical state Case analysis of interactions • Reads vs others • Writes vs writes © 2003 Herlihy and Shavit

What Does Consensus have to do with Concurrent Objects? 11/29/2020 © 2003 Herlihy and Shavit 49

The Consensus Object public abstract class Consensus implements Consensus { private Object[] announce; Always announce n) { intentions public Consensus. Impl(int announce = new Object[n]; } public void propose(Object value) { announce[Thread. my. Index()] = value; } abstract public Object decide(); }} (4) © 2003 Herlihy and Shavit Figure out which one won (4)

Can FIFO Queue Implement A Consensus Object? proposed array 8 FIFO Queue with red and black balls © 2003 Herlihy and Shavit

A Consensus Protocol Preferred array FIFO Queue with red and black balls 8 Coveted red ball Dreaded black ball © 2003 Herlihy and Shavit

Protocol: Write Value to Array 0 11/29/2020 0 © 2003 Herlihy and Shavit 1 53

Protocol: Take Next Item from Queue 0 8 0 11/29/2020 1 © 2003 Herlihy and Shavit 54

Protocol: Take Next Item from Queue I got the 0 coveted 1 red ball, so I will decide my value I got the dreaded black ball, so I will decide the other’s value from the array 8 11/29/2020 © 2003 Herlihy and Shavit 55

Consensus Using FIFO Queue public class Queue. Consensus extends Consensus { Queue queue; Initialize public Queue. Consensus() { Queue queue = new Queue(); queue. enq(“red ball"); queue. enq(“black ball"); } public Object decide() { i won String status = queue. deq(); int i = Thread. my. Index(); if (status == “red ball") i lost return proposed[i]; else return proposed[1 -i]; } © 2003 Herlihy and Shavit

Why does this Work? • • If one thread gets the red ball Then the other gets the black ball Winner can take her own value Loser can find winner’s value in array – Because threads write array – Before dequeueing from queue © 2003 Herlihy and Shavit

Implication • We can solve 2 -thread consensus using only – A two-dequeuer queue, and – A couple of atomic registers © 2003 Herlihy and Shavit

Implications • Given – A consensus protocol from queue and registers • Assume there exists – A queue implementation from atomic registers • Substitution yields: n o i t c i – A wait-free consensus protocol from atomic registers d a tr n o c (1) © 2003 Herlihy and Shavit (1)

Corollary • It is impossible to implement – a two-dequeuer wait-free FIFO queue – from read/write memory. © 2003 Herlihy and Shavit

Consensus Numbers • An object has consensus number n – If it can be used • Together with atomic read/write registers – To implement n-thread consensus • But not (n+1)-thread consensus © 2003 Herlihy and Shavit

Consensus Numbers • Theorem – Atomic read/write registers have consensus number 1 © 2003 Herlihy and Shavit

Consensus Numbers • Consensus numbers are a useful way of measuring synchronization power • Theorem – If you can implement X from Y – And X has consensus number c – Then Y has consensus number at least c © 2003 Herlihy and Shavit

Synchronization Speed Limit • Conversely – If X has consensus number c – And Y has consensus number d < c – Then there is no way to construct a waitfree implementation of X by Y • This theorem will be very useful – Unforeseen practical implications! © 2003 Herlihy and Shavit

Earlier Grand Challenge • Snapshot means – Write any array element – Read multiple array elements atomically • What about the dual problem: – Write multiple array elements atomically – Scan any array elements • Call this problem multiple assignment © 2003 Herlihy and Shavit

Multiple Assignment Theorem • No protocol – Using only read/write registers • Implements multiple assignment • Weird or what? – Single write/multiple read OK – Multi write/multiple read impossible – In fact: Multi write/single read also impossible!!! (1) © 2003 Herlihy and Shavit (1)

Proof Strategy • Given a 3 -element array – A writes atomically to slots 0 and 1 – B writes atomically to slots 1 and 2 – Any thread can scan any set of locations • Then we can solve 2 -consensus – So multiple assignment has consensus number at least 2 – No read/write register implementation © 2003 Herlihy and Shavit (1)

Initially A Writes to 0 and 1 B 11/29/2020 Writes to 1 and 2 © 2003 Herlihy and Shavit 68

Thread A wins if it sees A B 11/29/2020 © 2003 Herlihy and Shavit 69 (1)

Thread A wins if it sees A B 11/29/2020(1) © 2003 Herlihy and Shavit 70

Thread A loses if it sees A B 11/29/2020(1) © 2003 Herlihy and Shavit 71

Multi-Consensus class Multi. Consensus { Multi. Assign multi; Write/Scan locations public Object decide() { int i = Thread. my. Index(); // my index int j = 1 - i; // other index multi. assign(i, input, i+1, input); Object[] result = multi. scan(); if (result[(j + 1) % 3] == null || (result[j] == result[(j + 1) % 3])) return proposed[i]; Figure out else return proposed[j]; which one won } (4) © 2003 Herlihy and Shavit (4)

Summary • If a thread can assign atomically to 2 out of 3 array locations • Then we can solve 2 -consensus • Implying – No wait-free implementation – From read/write registers © 2003 Herlihy and Shavit

Read-Modify-Write • Method takes 2 arguments: – Variable x – Function f • Method call: – Returns value of x – Replaces x with f(x) © 2003 Herlihy and Shavit

Read-Modify-Write public abstract class RMW { private int value; Return prior value public void synchronized rmw(function f) { int prior = this. value; this. value = f(this. value); return prior; } Apply function f() } (1) © 2003 Herlihy and Shavit

Example: Read public abstract class RMW { private int value; public void synchronized read() { int prior = this. value; this. value = this. value; return prior; } f(v)=v is the identity function } (1) © 2003 Herlihy and Shavit

Example: test&set public abstract class RMW { private int value; public synchronized void TAS() { int prior = this. value; this. value = 1; return prior; } } F(v)=1 is constant function (1) © 2003 Herlihy and Shavit

Example: fetch&inc public abstract class RMW { private int value; public synchronized void FAI() { int prior = this. value; this. value = this. value+1; return prior; } f(v)=v+1 is increment function } (1) © 2003 Herlihy and Shavit

Example: fetch&add public abstract class RMW { private int value; public synchronized void FAA(int x) { int prior = this. value; this. value = this. value+x; return prior; } F(v, x)=v+x is the addition function } (1) © 2003 Herlihy and Shavit

Example: swap public abstract class RMW { private int value; public synchronized void swap(int x) { int prior = this. value; this. value = x; return prior; } F(v, x) = x is a constant function } (1) © 2003 Herlihy and Shavit

Example: compare&swap public abstract class RMW { private int value; public synchronized void CAS(int old, int new) { int prior = this. value; if (this. value == old) this. value = new; return prior; } f(old, new) is a complex function } (1) © 2003 Herlihy and Shavit

Definition • A RMW is non-trivial if there exists a value v Such that v ≠ f(v) • read is trivial • FAA is not © 2003 Herlihy and Shavit

Theorem • Any non-trivial RMW object has consensus number at least 2 • Implies no wait-free implementation of RMW registers from read/write registers • Hardware RMW instructions not just a convenience © 2003 Herlihy and Shavit

Reminder • All objects that inherit from Consensus have – propose method – which just stores input into this. announce[i] © 2003 Herlihy and Shavit

Proof Initialized to v public class RMWConsensus implements Consensus { private RMW r; Am I first? public Object decide() { int i = Thread. my. Index(); Yes, return if (r. rmw(f) == v) my input return this. announce[i]; else return this. announce[1 -i]; }} 11/29/2020(4) © 2003 Herlihy and Shavit No, return 85 other’s input

Proof • We have displayed – A two-thread consensus protocol – Using any non-trivial RMW object © 2003 Herlihy and Shavit

Interfering RMW • Let F be a set of functions such that for all fi and fj, either – They commute: fi(fj(v))=fj(fi(v)) – They overwrite: fi(fj(v))=fi(v) • Claim: Any such set of RMW objects has consensus number exactly 2 © 2003 Herlihy and Shavit

Examples • Test-and-Set f(v)=1 Overwrite fi(fj(v))=fi(v) • Swap f(v, x)=x Overwrite fi(fj(v))=fi(v) • Fetch-and-inc f(v)=v+1 Commute fi(fj(v))= fj(fi(v)) © 2003 Herlihy and Shavit

Meanwhile Back at the Critical State A about to apply f. A B about to apply f. B c 0 -valent 1 -valent © 2003 Herlihy and Shavit

Maybe the Functions Commute A applies f. A c B applies f. B A applies f. A C runs solo 0 -valent 0 1 © 2003 Herlihy and Shavit 1 -valent

Maybe the Functions Commute A applies f. A c B applies f. B These states look the same to C B applies f. B A applies f. A C runs solo 0 -valent 0 1 © 2003 Herlihy and Shavit 1 -valent

Maybe the Functions Overwrite A applies f. A c B applies f. B A applies f. A C runs solo 0 C runs solo 1 0 -valent © 2003 Herlihy and Shavit 1 -valent

Maybe the Functions Overwrite These states look the same to C A applies f. A c B applies f. B A applies f. A C runs solo 0 C runs solo 1 0 -valent © 2003 Herlihy and Shavit 1 -valent

Impact • Many early machines provided these “weak” RMW instructions – Test-and-set (IBM 360) – Fetch-and-add (NYU Ultracomputer) – Swap (Original SPARCs) • We now understand their limitations – But why do we want consensus anyway? © 2003 Herlihy and Shavit

Example: compare&swap public abstract class RMW { private int value; public void CAS(int old, int new) { int prior = this. value; if (this. value == old) this. value = new; return prior; } } (1) © 2003 Herlihy and Shavit

CAS Has Unbounded Consensus Number Initialized to -1 public class RMWConsensus implements Consensus { private RMW r; Am I first? public Object decide() { int i = Thread. my. Index(); Yes, return int j = r. CAS(-1, i); my input if (j == -1) return this. announce[i]; else return this. announce[j]; }} No, return (4) © 2003 Herlihy and Shavit other’s input

The Consensus Hierarchy 1 Read/Write Registers, Snapshots… 2 T&S, F&I, Swap…. . . ∞ CAS, … © 2003 Herlihy and Shavit

Multiple Assignment • Atomic k-assignment • Solves consensus for 2 k-2 threads • Every even consensus number has an object (can be extended to odd numbers) © 2003 Herlihy and Shavit

Lock-Free Implementations • Infinitely often some method call completes in a finite number of steps • Stalinistic real world systems approach • Implies no mutual exclusion (2) © 2003 Herlihy and Shavit

Lock-Free Implementations • Lock-free = Wait-free for finite executions • Lock-free consensus is just as impossible • All the results we presented hold for lock-free algorithms also. (2) © 2003 Herlihy and Shavit

There is More: Universality • Consensus is universal • From n-thread consensus – – – Wait-free/Lock-free Linearizable n-threaded Implementation Of any sequential object © 2003 Herlihy and Shavit