Logic and Lattices for Distributed Programming Neil Conway
Logic and Lattices for Distributed Programming Neil Conway, William R. Marczak, Peter Alvaro, Joseph M. Hellerstein UC Berkeley David Maier Portland State University
Distributed Programming: Key Challenges Asynchrony Partial Failure
Dealing with Disorder Enforce global order – Paxos, Two-Phase Commit, GCS, … – “Strong Consistency” Tolerate disorder – Programmer must ensure correct behavior for many possible network orders – “Eventual Consistency” • Typical goal: replicas converge to same final state
Dealing with Disorder Enforce global order – Paxos, Two-Phase Commit, GCS, … – “Strong Consistency” Tolerate disorder – Programmer must ensure correct behavior for many possible network orders – “Eventual Consistency” • Typical goal: replicas converge to same final state
Goal: Make it easier to write programs on top of eventual consistency
This Talk 1. Prior Work – Convergent Modules (CRDTs) – Monotonic Logic (CALM) 2. Bloom. L 3. Case Study
Client 0 Write: Read: {Alice, Bob, Bob} Carol} Students {Alice, Bob, Carol} {Alice, Bob} How to resolve? Write: Read: {Alice, Bob, Bob} Dave} Client 1 Students {Alice, Bob, Bob} Dave}
Problem Replicas perceive different event orders Goal Same final state at all replicas Solution Use commutative operations (“merge functions”)
Client 0 Students {Alice, Bob, Carol, Dave} Merge = Set Union Client 1 Students {Alice, Bob, Carol, Dave}
Commutative Operations • Common design pattern • Formalized as CRDTs: Convergent and Commutative Replicated Data Types – Shapiro et al. , INRIA (20092012) – Based on join semilattices
Lattices h. S, t, ? i is a bounded join semilattice iff: – S is a set – t is a binary operator (“least upper bound”) • Associative, commutative, and idempotent • Induces a partial order on S: x ·S y if x t y = y • Informally, “merge function” for elements of S – ? is the “least” element in S • 8 x 2 S: ? t x = x 12
Time Set (LUB = Union) Increasing Integer (LUB = Max) Boolean (LUB = Or)
Read: {Alice, Bob, Carol, Dave} Client 0 Read: {<Alice, Bob>} Write: {<Alice, Bob>, <Carol, Dave>} Students {Alice, Bob, Carol, Carol} Dave} Teams {<Alice, Bob>, {<Alice, Bob>} <Carol, Dave>} Replica Synchronization Remove: {Dave} Client 1 Students {Alice, Bob, Carol} {Alice, Bob, Carol, Dave} Teams {<Alice, Bob>, {<Alice, Bob>} <Carol, Dave>}
Read: {Alice, Bob, Carol} Students {Alice, Bob, Carol, Dave} {Alice, Bob, Carol} Client 0 Read: {<Alice, Bob>} Teams {<Alice, Bob>} Replica Nondeterministic Synchronization Outcome! Remove: {Dave} Students {Alice, Bob, Carol} {Alice, Bob, Carol, Dave} Client 1 Teams {<Alice, Bob>}
Problem: Composition of CRDTs can result in non-determinism
Possible Solution: Encapsulate all distributed state in a single CRDT Hard to design, verify, and test Doesn’t scale with application size
Goal: Design a language that allows safe composition of CRDTs
Solution: … Datalog? • Concurrent work: distributed programming using Datalog – P 2 (2006 -2010) – Bloom (2010 -2012) • Monotonic logic: building block for convergent distributed programs
Monotonic Logic • As input set grows, output set does not shrink – “Retraction-free” • Order independent • e. g. , map, filter, join, union, intersection Non-Monotonic Logic • New inputs might retract previous outputs • Order sensitive • e. g. , aggregation, negation
Monotonicity and Determinism Agents learn strictly more knowledge over time Different learning order, same final outcome Result: Program is deterministic!
Consistency As Logical Monotonicity CALM Analysis 1. All monotone programs are deterministic 2. Simple syntactic test for monotonicity Result: Whole-program static analysis for eventual consistency
Problem: CALM only applies to programs over growing sets Version Numbers Timestamps Threshold Tests
Quorum Vote • A coordinator accepts votes from agents • Count # of votes – When Count(Votes) > k, send “success” message
Quorum Vote • A coordinator accepts votes from agents • Count # of votes – When Count(Votes) > k, send “success” message Aggregation is non-monotonic!
Bloom. L Whole program analysis Flexible types (any lattice) CRDTs Limited scope (single object) Flexible types (any lattice) CALM Whole program analysis Limited types (only sets)
Bloom. L Constructs Organization Communication State Computation Collection of agents Message passing Lattices Functions over lattices
Monotone Functions f : S T is a monotone function iff 8 a, b 2 S : a ·S b ) f(a) ·T f(b) 28
Time Monotone function from set increase-int size() Set (LUB = Union) Monotone function from increase-int boolean >= 5 Increasing Integer (LUB = Max) Boolean (LUB = Or)
Quorum Vote in Bloom. L QUORUM_SIZE = 5 RESULT_ADDR = "example. org" class Quorum. Vote include Bud Annotated Ruby class Communication interfaces state do channel : vote_chn, [: @addr, : voter_id] Program state channel : result_chn, [: @addr] Lattice state declarations lset : votes lmax : vote_cnt lbool : got_quorum Accumulate votes end Monotonic CALM into set Monotone function: set ! max Monotone function: max ! bool bloom do votes <= vote_chn {|v| v. voter_id} Program vote_cnt <= votes. size got_quorum <= vote_cnt. gt_eq(QUORUM_SIZE) result_chn <~ got_quorum. when_true { [RESULT_ADDR] } Merge function for set lattice end Threshold test on bool (monotone) end logic 30
Bloom. L Features • Generalizes logic programming to lattices – Integration of relational-style queries and functions over lattices – Efficient incremental evaluation scheme • Library of built-in lattices – Booleans, increasing/decreasing integers, sets, multisets, maps, … • API for defining custom lattices
Case Studies Key-Value Store – Object versioning via vector clocks – Quorum replication Replicated Shopping Cart – Using custom lattice types to encode domain-specific knowledge
Case Studies Key-Value Store – Object versioning via vector clocks – Quorum replication Replicated Shopping Cart – Using custom lattice types to encode domain-specific knowledge
Case Study: Shopping Carts 34
Case Study: Shopping Carts 35
Case Study: Shopping Carts 36
Case Study: Shopping Carts 37
Perspectives on Shopping • CRDTs – Individual server replicas converge • Bloom – Checkout is non-monotonic requires distributed coordination • Built-in Bloom. L lattice types – Checkout is not a monotone function of any of the built-in lattices
Observation: Once a checkout occurs, no more shopping actions can be performed
Observation: Each client knows when a checkout can be processed “safely”
Monotone Checkout OPS = [1, 2, 3] Complete OPS = [2, 3] Incomplete OPS = [1, 2] Incomplete OPS = [1] Incomplete OPS = [2] Incomplete OPS = [3] Incomplete 41
Monotone Checkout 42
Monotone Checkout 43
Monotone Checkout 44
Monotone Checkout 45
Shopping Takeaways • Checkout summary is a monotone function of client’s activities • Custom lattice type captures applicationspecific notion of “forward progress” – “Unsafe” state hidden behind ADT interface
Recap 1. How to build eventually consistent systems – Write disorderly programs 2. Disorderly state – Lattices 3. Disorderly computation – Monotone functions over lattices 4. Bloom. L – Type system for deterministic behavior – Support for custom lattice types
Thank You! http: //www. bloom-lang. net
Backup Slides
Strong Consistency in Industry “… there was a single overarching theme within the keynote talks… strong synchronization of the sort provided by a locking service must be avoided like the plague… [the key] challenge is to find ways of transforming services that might seem to need locking into versions that … can operate correctly without locking. ” -- Birman et al. , “Toward a Cloud Computing Research Agenda” (LADIS, 2009) 50
Bloom Operational Model 51
Quorum Vote in Bloom QUORUM_SIZE = 5 RESULT_ADDR = "example. org" class Quorum. Vote include Bud Annotated Ruby class Communication state do channel : vote_chn, [: @addr, : voter_id] channel : result_chn, [: @addr] table : votes, [: voter_id] scratch : cnt, [] => [: cnt] end Program state Persistent Storage Transient Storage Not (set) monotonic! Count votes Accumulate votes bloom do votes <= vote_chn {|v| [v. voter_id]} Program cnt <= votes. group(nil, count(: voter_id)) result_chn <~ cnt {|c| [RESULT_ADDR] if c >= QUORUM_SIZE} end Send message when quorum reached logic 52
Built-in Lattices Name Description ? atb lbool Threshold test lmax Sample Monotone Functions false a∨ b Increasing number 1 max(a, b ) gt(n) ! lbool +(n) ! lmax -(n) ! lmax lmin Decreasing number − 1 min(a, b) lt(n) ! lbool lset Set of values ; a[b intersect(lset) ! lset product(lset) ! lset contains? (v) ! lbool size() ! lmax lpset Non-negative set ; a[b sum() ! lmax lbag Multiset of values ; a[b mult(v) ! lmax +(lbag) ! lbag lmap Map from keys to lattice values empty map when_true() ! v at(v) ! any-lat intersect(lmap) ! lmap 53
Failure Handling Great question! 1. Monotone programs handle transient faults very well – Deterministic simple logging – Commutative, idempotent simple recovery 2. Future work: “controlled non-determinism” – Timeout code is fundamentally non-deterministic – But we still want mostly deterministic programs
Handling Non-Monotonicity … is not the focus of this talk Basic alternatives: 1. Nodes agree on an event order using distributed coordination (e. g. , Paxos) 2. Allow non-deterministic outcomes • If needed, compensate and apologize
- Slides: 55