Dynamic Storage Reconfiguration Alexander Spiegelman Idit Keidar Agenda

Dynamic Storage Reconfiguration Alexander Spiegelman, Idit Keidar

Agenda • Reconfigurable atomic storage – Motivation – Definition • Solutions – Expired configuration oracle – Traversing possible configurations – Using a generic Sp. Sn abstraction – Existing algorithms as Sp. Sn implementations

Distributed Shared Storage server client • Reliable replicated storage • Using unreliable components • Asynchrony - tolerate unpredictable network delays 3

Distributed Shared Storage 101 • Servers – fault-prone • Clients – ephemeral, infinitely many • Servers emulate read/write registers to clients – ABD [Attiya, Bar-Noy and Dolev] read/write 1 2 3 4 5

ABD Register Emulation Write to majority 1 2 3 4 5

ABD Register Emulation 1 2 3 4 5

ABD Register Emulation Read from majority 1 2 3 4 5

ABD Register Emulation 1 2 3 4 Every two majorities intersect 5

Motivation for Reconfiguration • ABD tolerates minority failures 1 2 3 4 5 • No more failures are allowed! – need to be able to remove faulty and add correct servers

But, Be Careful! Remove(1, 2) Add(6, 7) 1 2 3 4 Remove(1, 2) Add(8, 9) 5

But, Be Careful! 6 7 4 3 5 8 9

But, Be Careful! Split Brain! 6 7 4 3 5 • Client have to coordinate 8 9

Agenda • Reconfigurable atomic storage – Motivation – Definition • Solutions – Expired configuration oracle – Traversing possible configurations – Using a generic Sp. Sn abstraction – Existing algorithms as Sp. Sn implementations • Liveness with endless reconfigurations

Model • Infinite set of clientsΠ – any client can fail by crashing • Infinite set of potential servers Φ – failures are restricted – generalize majority to dynamic model (soon) • Communication – asynchronous

Configuration • Finite set of servers & history how we got to it • Changes = {+, -} XΦ – E. g. , change can be +s 12 or –s 7 • Configuration = finite subset of Changes – e. g. , {+s 1, +s 2, -s 2 +s 3, -s 4, +s 12} • Initial configuration C 0 is known to all clients – e. g. , {+s 1, +s 2, +s 3, +s 4, +s 5}

Configuration’s Membership • Servers that were added and not removed – i. e. , {s | +s ∈ conf ∧ -s ∉ conf } – e. g. , Initial membership is {s 1, s 2, s 3, s 4, s 5} – by slight abuse of notation, we sometimes refer to C. membership as C configuration C C. membership +s 1 +S 3 +s 5 - s 2 +s 4 - s 5 S 3 s 4

Install event • Configurations can be installed – i. e. , install(C 0) – Intuitively, indicates that from now on we expect to be able to perform operations in the new configuration • We will use it to define the failure model

Changing the Configuration • Add reconfig(C, P) to the API – Changes configuration C by applying changes in P – Returns new installed configuration C’ – May install more configurations – C becomes superseded • C is either C 0 or a configuration returned by an earlier reconfig (assumption on usage) • Proposal P is a set of changes – E. g. , {+s 12, -s 7, -s 2} • By convention, reconfig({}, C 0) returns C 0 at time 0

Changing the Configuration (sequential example) C 0 reconfig(C 0, {+s 4, -s 1+}) installed +s 1 +s 2 +S 3 When reconfig returns C 1, C 0 is superseded C 1 +s 1 - s 1 +s 2 +s 4 +S 3

Reconfiguration Properties • reconfig(C, P) returns C’ s. t. – C’ was installed – C’ reflects P: C’ C P – Validity: if C’’ was installed, then for every c in C’’ there was reconfig(*, P’’) invocation s. t. c ∈ P’’

Reconfiguration Properties +s 1 • For example consider reconfig(C 0, {+s 4}) +s 2 – Concurrent with reconfig(C 0, {+s 5, - s 1}), C 1 +s 1 C 2 C 0 +S 3 C 4 +s 1 +s 4 +s 2 - s 1 +s 2 +s 5 +S 3 - s 1 +s 4 +S 3 • Both operations can install • reconfig(C 0, {+s 4}) can return all

Failure Model • For ABD, need live majority – but in which configuration? • Availability: A configuration C is available if a majority of C. members is alive – Rationale: can emulate register using ABD • We require reconfigurability: – Every installed configuration C must be available until it superseded (a reconfig returns C’ ⊂ C) – Rationale: Administrator may take down machines once a reconfiguration removing them returns

Shared Memory Abstraction • We can see each configuration as a collection of (low-level) MWMR registers • emulated by the servers in the configuration’s membership (ABD) • accessible by clients if the configuration is available • all the registers can be read in one access c c X. read() Y. write(v) s s s

Problem: Dynamic Storage • Emulate MWMR register • Using a dynamic set of servers • May fail subject to reconfigurability • Clients invoke operations – read(), write(v), and reconfig(C, P) – sequential read/write specification: a read returns the last written value – reconfig(C, P) returns C’ that is installed, reflects P, and satisfies validity

Agenda • Reconfigurable atomic storage – Motivation – Definition • Solutions – Expired configuration oracle – Traversing possible configurations – Using a generic Sp. Sn abstraction – Existing algorithms as Sp. Sn implementations

Tracking Available Configurations C 0 +s 1 +s 2 +S 3

Tracking Available Configurations C 0 +s 1 +s 2 +S 3 C’ reconfig +s 4, +s 5 +s 1 +s 4 +s 2 +s 5 +S 3

Tracking Available Configurations C 0 +s 1 +s 2 C’ reconfig +s 4, +s 5 C’ client We need to help clients +s 4 +s 2 +s 5 +S 3 May be unavailable (reconfigurability) +s 1

Expired Configuration Oracle • Necessary for liveness • Distributed • Notified of superseded configurations – Using expire(config, next) event • Informs clients of next configurations

Tracking Available Configurations C 0 +s 1 +s 2 +S 3 C’ reconfig +s 4, +s 5 Expire(C 0, C 1) May be unavailable (reconfigurability) C’ client +s 1 +s 4 +s 2 +s 5 +S 3

Expire: A Closer Look • Client invokes expire(config, next) – config is expired, next is newer • Different clients may expire the same configuration with different nexts • A client that accesses an unavailable configuration – May either succeed OR – Receive an expired message with some next configuration OR – Stuck forever if the configuration was not previously expired

Expire Event c 1 C 0

Expire Event Reconfig(C 0, P 1) c 1 C 0

Expire Event Reconfig(C 0, P 1) Install(C 1) c 1 C 1=C 0∪P 1 C 0

Expire Event expi re(C 0, C 1 ) Reconfig(C 0, P 1) Install(C 1) expire(C 0, C 1) c 1 C 1=C 0∪P 1 C 0

Expire Event Reconfig(C 0, P 2) Reconfig(C 0, P 1) Install(C 1) expire(C 0, C 1) c 2 c 1 C 1=C 0∪P 1 C 0

Expire Event Reconfig(C 0, P 2) Install(C 2) Reconfig(C 0, P 1) Install(C 1) expire(C 0, C 1) c 2 c 1 C 1=C 0∪P 1 C 0 C 2=C 0∪P 2

Expire Event exp ire( C 0 , C 2 ) Reconfig(C 0, P 2) Install(C 2) expire(C 0, C 2) Reconfig(C 0, P 1) Install(C 1) expire(C 0, C 1) c 2 c 1 C 1=C 0∪P 1 C 0 C 2=C 0∪P 2

Expire Event C 1 C 2 c 3 c 4 Reconfig(C 0, P 2) Install(C 2) expire(C 0, C 2) Reconfig(C 0, P 1) Install(C 1) expire(C 0, C 1) c 2 c 1 C 1=C 0∪P 1 C 0 Expire is dangerous! C 2=C 0∪P 2 Unavailable Clients must coordinate.

Agenda • Reconfigurable atomic storage – Motivation – Definition • Solutions – Install/expire oracle – Traversing possible configurations – Using a generic Sp. Sn abstraction – Existing algorithms as Sp. Sn implementations

Dynamic Atomic Storage Read/Write Naïve Solution • When there is no reconfig in progress, emulate read/write using ABD on latest configuration returned by reconfig • What happens during reconfiguration? Tomorrow Technion servers will be down for maintenance from 5: 30 am to 6: 45 am Virtually Yours, Moshe Barak

We Don’t Want to Stop the World • Where do we read/write during reconfiguration? • Track installed non-superseded configurations – Read in all – Write in the latest • How we do it? • What about reconfig?

Consensus Can Help With Reconfig • Rambo – uses consensus to agree on the next conf C 0

Consensus Can Help With Reconfig • Rambo – uses consensus to agree on the next conf – Reconfig: C 0

Consensus Can Help With Reconfig • Rambo – uses consensus to agree on the next conf – Reconfig: Propose(P 1) c 1 C 0 Propose(P 2) c 2

Consensus Can Help With Reconfig • Rambo – uses consensus to agree on the next conf – Reconfig: c 1 C 0 Decide(P’) c 2

Consensus Can Help With Reconfig • Rambo – uses consensus to agree on the next conf – Reconfig: Install(C 1) expire(C 0, C 1) c 1 C 0 Decide(P’) c 2 C 1=C 0∪P’

Consensus Can Help With Reconfig • Rambo – uses consensus to agree on the next conf – Reconfig: Install(C 1) expire(C 0, C 1) c 1 C 0 Decide(P’) c 2 C 1=C 0∪P’ C 2

Traversing Configuration DAG • Asynchronous solutions traverse DAG – Dyna. Store [Aguilera, Keidar, Malkhi and Shraer] – Smart. Merge[Jehl, Vitenberg and Meling] – Parsimonious Sp. Sn [Gafni and Malkhi]

Traversing Configuration DAG

Traversing Configuration DAG C 1 C 2 C 3 Clients install the configurations they propose.

Traversing Configuration DAG C 1 C 2 C 3 Then, propose them in the initial configuration,

Traversing Configuration DAG C 1, C 3 C 2, C 3 C 1 C 2 C 3 and learn what configurations were proposed

Traversing Configuration DAG C 1, C 3 C 2, C 3 C 1 C 2 C 3 Continue the traverse to all the learned configurations. • In each configuration a client proposes the union of all the configurations it traversed so far

Traversing Configuration DAG C 1, C 3 C 2, C 3 C 1 C 2 C 3 For correctness, must ensure traversal intersection

Traversing Configuration DAG C 1, C 3 C 2, C 3 C 1 C 2 C 3

Traversing Configuration DAG C 1, C 3 C 2, C 3 Orange accesses C 1: Proposes (and installs) C 1∪C 3 Learns C 1∪C 3 C 1 C 2 C 3

Traversing Configuration DAG C 1, C 3 C 2, C 3 C 1∪C 3 C 2∪C 3 Green accesses C 2: Proposes (and installs) C 2∪C 3 Learns C 2∪C 3 C 1∪C 3 C 2∪C 3

Traversing Configuration DAG C 1, C 3 C 2, C 3 C 1∪C 3 C 2∪C 3 Green accesses C 3: Proposes C 2∪C 3 Learns C 2∪C 3 C 1∪C 3 C 2∪C 3

Traversing Configuration DAG C 1, C 3 C 2, C 3 C 1∪C 3 C 2∪C 3 Orange accesses C 3: Proposes C 1∪C 3 Learns C 2∪C 3, C 1∪C 3 C 2∪C 3

Traversing Configuration DAG C 1, C 3 C 2, C 3 C 1∪C 3 C 2∪C 3 Red accesses C 3: Proposes C 3 Learns C 2∪C 3, C 1∪C 3 C 2∪C 3

Traversing Configuration DAG C 1, C 3 C 2, C 3 C 1∪C 3 C 2∪C 3 C 1∪ C 2∪C 3 Red accesses C 1∪C 3: Proposes C 1∪ C 2∪ C 3 Learns. C 1∪ C 2∪ C 3 C 1∪C 3 C 2∪C 3 C 1∪ C 2∪C 3

Traversing Configuration DAG C 1, C 3 C 2, C 3 C 1∪C 3 C 2∪C 3 C 1∪ C 2∪C 3 Orange accesses C 1∪C 3: Proposes C 1∪C 2∪C 3 Learns C 1∪C 2∪C 3 C 1∪C 3 C 2∪C 3 C 1∪ C 2∪C 3

Traversing Configuration DAG C 1, C 3 C 2, C 3 C 1∪C 3 C 2∪C 3 C 1∪ C 2∪C 3 Orange accesses C 2∪C 3: Proposes C 1∪C 2∪C 3 Learns C 1∪C 2∪C 3 C 1∪C 3 C 2∪C 3 C 1∪ C 2∪C 3

Traversing Configuration DAG C 1, C 3 C 2, C 3 C 1∪C 3 C 2∪C 3 C 1∪ C 2∪C 3 Red accesses C 2∪C 3: Proposes C 1∪C 2∪C 3 Learns C 1∪C 2∪C 3 C 1∪C 3 C 2∪C 3 C 1∪ C 2∪C 3 C 1� C 3 C 2� C 3 C 1� C 2� C 3

Traversing Configuration DAG C 1, C 3 C 2, C 3 C 1∪C 3 C 2∪C 3 C 1∪ C 2∪C 3 Green accesses C 2∪C 3: Propose C 2∪ C 3 Learns C 1∪ C 2∪ C 3 C 1∪C 3 C 2∪C 3 C 1∪ C 2∪C 3

Traversing Configuration DAG C 1, C 3 C 2, C 3 C 1∪C 3 C 2∪C 3 C 1∪ C 2∪C 3 Orange accesses C 1∪C 2∪C 3 : Proposes C 1∪C 2∪C 3 Learns C 1∪C 2∪C 3 No more configurations: Returns C 1∪C 2∪C 3 C 1∪C 3 C 2∪C 3 C 1∪ C 2∪C 3

Traversing Configuration DAG C 1, C 3 C 2, C 3 C 1∪C 3 C 2∪C 3 C 1∪ C 2∪C 3 Green accesses C 1∪C 2∪C 3 : Proposes C 1∪C 2∪C 3 Learns C 1∪C 2∪C 3 No more configurations: Returns C 1∪C 2∪C 3 C 1∪C 3 C 2∪C 3 C 1∪ C 2∪C 3

Traversing Configuration DAG C 1, C 3 C 2, C 3 C 1∪C 3 C 2∪C 3 C 1∪ C 2∪C 3 Red accesses C 1∪C 2∪C 3 : Proposes C 1∪C 2∪C 3 Learns C 1∪C 2∪C 3 No more configurations: Returns C 1∪C 2∪C 3 C 1∪C 3 C 2∪C 3 C 1∪ C 2∪C 3

Traversing Configuration DAG C 1, C 3 C 2, C 3 C 1∪C 3 C 2∪C 3 C 1∪ C 2∪C 3 Before returning, clients expire all configuration in their Dag with the last one C 1∪C 3 C 2∪C 3 C 1∪ C 2∪C 3

Traversing Configuration DAG C 1, C 3 C 2, C 3 C 1∪C 3 C 2∪C 3 C 1∪C 3 Orange DAG C 2∪C 3 C 1∪C 3 C 2∪C 3 C 1∪ C 2∪C 3

Traversing Configuration DAG C 1, C 3 C 2, C 3 C 1∪C 3 C 2∪C 3 C 1∪C 3 Green DAG C 2∪C 3 C 1∪C 3 C 2∪C 3 C 1∪ C 2∪C 3

Traversing Configuration DAG C 1, C 3 C 2, C 3 C 1∪C 3 C 2∪C 3 C 1∪C 3 Red DAG C 2∪C 3 C 1∪C 3 C 2∪C 3 C 1∪C 2∪C 3

Traversing Configuration DAG C 1, C 3 C 2, C 3 C 1∪C 3 C 2∪C 3 C 1∪ C 2∪C 3 C 1∪C 3 C 2∪C 3 All DAGs must intersect in every level C 1∪C 2∪C 3

Recall: No agreement • Clients may return different configurations in the intersection path C 1∪C 3 C 2∪C 3 C 1∪C 2∪C 3

Recall: No agreement • Clients may return different configurations in the intersection path • For example: C 1∪C 3 C 2∪C 3 C 1, C 3 C 2, C 3 C 1∪C 3 C 2∪C 3 Red accesses 3: Proposes C 3 Learns C 3 No more configurations: Returns C 3

But, Lattice Agreement • Configurations in the same path in the DAG are related by containment C 1∪C 3 C 2∪C 3 C 1∪C 2∪C 3

(Simple) Lattice Agreement • Each process starts with an initial value and learns values that satisfies the following: – Validity: A value learnt by a process is a union of some set of initial values that includes its own initial value. – Consistency: Values learnt by any two processes related by containment. – Stability. A process can learn at most one value.

Agenda • Reconfigurable atomic storage – Motivation and definition – Failure model • Solutions – Expired configuration oracle – Traversing possible configurations – Using a generic Sp. Sn abstraction – Existing algorithms as Sp. Sn implementations

Sp. Sn Abstraction • The core task clients solve in every configuration – A multi-input, multi-output function • Uses ABD to implement a collection of registers • Usage: call Sp. Sn in each configuration to obtain next configuration to traverse Sp. Sn ABD Sp. Sn

Sp. Sn API and Guarantees • Participants input their proposals • The outputs are sets of subsets of proposals • Intersection: One subset appears in all outputs – for example: {{A}} {{A}, {A, B}, {A, B, C}} {{A}, {A, C}, {A, B, C}} {{A}, {B}} {{A}, {B}, {C}}

Generic Traverse Using Sp. Sn • Start with: – speculation = { current } – P= proposal • track every configuration C in speculation: – propose P to C. Sp. Sn – add the output to speculation – P speculation union • if C expired by next – speculation = {next} Jump forward • finally – for every C in speculation , expire(C, P)

Great! But How To Build An Atomic Register? • All operations traverse the DAG – They are very similar • reconfig transfers the latest written value to the last configuration before expiring the configurations in the DAG • read/write help reconfig to unite the DAG, read from every configuration in the DAG and write to the last one

Great! But How To Build An Atomic Register? • Start with: – speculation = { current } – P= proposal • track every configuration C in speculation: – propose P to C. Sp. Sn – add the output to speculation – P speculation union • if C expired by next – speculation = {next} Same in all operations • finally – for every C in speculation , expire(C, final proposal)

reconfig(proposal) • Start with: – speculation = { current } – P= proposal – <v, ts> = < , 0> • track every configuration C in speculation: – speculation = speculation ∪ C. Sp. Sn(P) – <v’, ts’> C. read() – if ts’ > ts, then <v, ts> <v’, ts’> – if C is the last not tracked in speculation • C. write(<v, ts>) • speculation = speculation ∪ C. Sp. Sn(P) – P speculation union • For every C in speculation expire(C, P), and Return P

read() • Start with: – speculation = { current } – P= – <v, ts> = < , 0> • track every configuration C in speculation: – speculation = speculation ∪ C. Sp. Sn(P) – <v’, ts’> C. read() – if ts’ > ts, then <v, ts> <v’, ts’> – if C is the last not tracked in speculation • C. write(<v, ts>) • speculation = speculation ∪ C. Sp. Sn(P) – P speculation union • Return v

write(value) • Start with: – speculation = { current } – P= – <v, ts> = < , 0> • track every configuration C in speculation: – speculation = speculation ∪ C. Sp. Sn(P) – <v’, ts’> C. read() – if ts’ > ts, then <v, ts> <v’, ts’> – if C is the last not tracked in speculation • C. write(<value, ts+1>) • speculation = speculation ∪ C. Sp. Sn(P) – P speculation union • Return ok

Agenda • Reconfigurable atomic storage – Motivation and definition – Failure model • Solutions – Expired configuration oracle – Traversing possible configurations – Using a generic Sp. Sn abstraction – Existing algorithms as Sp. Sn implementations

Recasting Dynamic Algorithms in Terms of Sp. Sn Implementation • We can extract each algorithm’s mechanism to implement Sp. Sn • Implementation differ in the complexity cost and the size of the obtained DAG. – We measure complexity in the number w/r accesses – The DAG size depends on the number of concurrent reconfig and w/r operations.

Dynamic Algorithms in Terms of Sp. Sn For m concurrent clients, n of which perform reconfigurations Algorithm Sp. Sn Implementation DAG size Sp. Sn Cost RAMBO if (proposal != null) write consensus(proposal) return collect n Consensus Dyna. Store if (proposal != null) write proposal return weak snapshot collect min(m 2, 2 n) O(1) collects Smart. Merge Pre-computation: assumes prop lattice. Agreement(proposal) lattice. Agreement never fails (not Sp. Sn: same as Dyna. Store reconfigurable) n O(1) collects Parsimonious Sp. Sn n O(n) collects multi-round commit –adopt

Dyna. Store Uses Weak Snapshot • Array of registers - one for each client • Each client 1. If proposal !=null, write proposal to its register P

Dyna. Store Uses Weak Snapshot • Array of registers - one for each client • Each client 1. If proposal !=null, write proposal to its register 2. collects all the registers 3. If collect is not empty, collect again All clients that return non-empty set see the first written proposal P P One proposal appears in all outputs

Dyna. Store Uses Weak Snapshot • Cost: 3 operations – One write and two collects • DAG size: – 2 n possible subsets of proposals (power set) that can be installed – Each client installs at most n configurations – Therefore: • min(m 2, 2 n)

Smart. Merge Uses Lattice Agreement • Assumes lattice Agreement never fails (not reconfigurable) • Clients first propose in lattice Agreement and then continue with the output as Dyna. Store • Recall – Validity: A value learnt by a process is a union of some set of initial values that includes its own initial value. – Consistency: Values learnt by any two processes related by containment.

Smart. Merge Uses Lattice Agreement • Cost: O(n) – Generalized lattice agreement [Rajamani, Rajan, Ramalingam and Vaswani] • Dag size • The output of lattice agreement is related by containment • Therefore, the size of the power set is bounded by n – At most n possible installed configurations • Thus, the DAG size is at most n

Parsimonious Sp. Sn • Use multiple rounds of commit-adopt – Each round O(1) operations – At least n round – Complexity: O(n) • All the returned subset are related by containment – Therefore, the DAG size is at most n • Reconfigurable – I will skip the details

Dynamic Atomic Storage: Conclusion • Reconfiguration is essential for system longevity • Definition as shared memory object – Read/write/reconfig operations • Failure model: reconfigurability – Configuration remains available until superseded • Generic implementation – Install new configurations – Traverse all installed non-superseded configurations in read/write/reconfig operations – Use Sp. Sn abstraction – Different solutions implement Sp. Sn in different ways

Thank You