PerformanceOptimal Readonly Transactions Haonan Lu Siddhartha Sen Wyatt

Performance-Optimal Read-only Transactions Haonan Lu★ Siddhartha Sen✢, Wyatt Lloyd★ ★Princeton University, ✢Microsoft Research 1

Distributed Storage Systems Enable Today’s Web Services Load Page Web Read Mia’s Page Jack Friend Jack Mia Storage Write Friend Lists 2

Distributed Storage Systems Reads Dominate Workloads Load Page Web Mia’s Page Reads Jack Friend Jack Writes Mia Jack’s Page Storage Friend Lists 3

Distributed Storage Systems Simple Reads Are Insufficient Web Unfriend Mia New page Read Jack Load Page New Page Mia’s Page Jack’s Page New Page Read Storage Friends ✔ Friend Lists Unfriended 4

Read-Only Transactions • A group of simple reads sent in parallel • Do not write data – Writes are allowed in the system • Coordinate a consistent view across shards Coordination overhead causes higher latency and lower throughput 5

Goal: Read-only transaction performance as close as possible to simple reads 6

Goal: Read-only transaction performance as close as possible to simple reads We answer: • What does optimal performance mean for read-only transactions? • When is optimal performance achievable? • How can we design performance-optimal read-only transactions? 7

Performance Factors Engineering vs. Algorithmic Coordination Algorithmic Properties • Focus on the algorithmic properties due to coordination Engineering Factors • Equally impact simple reads and read-only transactions • Abstract engineering factors by comparing to simple reads … Batching Networking Hardware 8

Performance Factors Algorithmic Properties Blocking R R Simple Read Page Friends 9

Performance Factors Algorithmic Properties Blocking Messages R R Simple Read Page Friends 10

Performance Factors Algorithmic Properties Timestamp Blocking Messages R R Metadata Simple Read Page Timestamp Friends 11

Performance Factors Coordination Is Algorithmic Properties More Messages Coordination Blocking Overhead Metadata Simple Read 12

Read-Only Transactions Optimal Performance Algorithmic Properties Blocking N Metadata Messages O C Simple Read Performance-optimal Read-only Transactions (N, O, C) Simple Read 13

Non-Blocking Reads • Do not wait on external events – Distributed locks, timeouts, messages, etc. • Lower latency – Avoid any time spent blocking • Higher throughput – Avoid CPU cost of context switches 14

One-Round Communication • One-round on-path reads – Succeed in one round, i. e. , no retries • No off-path messages – Required by reads but off the critical path • Lower latency – Avoids time for extra on-path messages • Higher throughput – Avoids CPU cost of processing extra messages 15

Constant Metadata • Metadata – Information used to find a consistent view – Timestamps, transaction IDs, etc. • Size of metadata remains constant regardless of contention • Higher throughput – Avoids CPU cost of processing extra data 16

Performance-optimal read-only transactions are NOC: Non-blocking messages that complete in One-round with Constant metadata 17

Strict Serializability • The strongest consistency model – Writing applications made easy • Requires a total order + real-time order Jack New Page Done New Add Page Done Read Mia New Page Friends Read ✔ Mia Friends Mia ✔ 18

The NOCS Theorem: Impossible for read-only transaction algorithms to achieve performance-optimality [N, O, C] and strict serializability [S] 19

Proof Intuition of NOCS unstable Svr-1 Finalized Write Svr-2 Svr-3 Coordination Free Unfinalized Write Coordination Required Svr-4 now 20

Proof Intuition of NOCS unstable Svr-1 Svr-2 ? Svr-3 ? Must give up either N, O, or C Svr-4 ROTXN now 21

NOC Designs By the NOCS Theorem Process-order Serializability Our new design: PORT Weak Strong Strict Serializability Causal Read Committed My. SQL Cluster Weak Consistency 22

Design Insight Capturing the Stable Frontier stable unstable Svr-1 Svr-2 Svr-3 Svr-4 Stable Frontier (SF) now 23

Version Clock • A type of logical clock – Specialized for distributed storage systems • Treat reads and writes differently – Enable optimizations for reads and writes • Capture the stable frontier 24

PORT Overview Jack Web Client Storage Server 25

PORT Overview Jack Key A Version Clock [AX]0 [AY]1 [AZ]2 26

PORT Overview Jack Version Stamp (VS) 1 Version Clock Key A [AX]0 [AY]1 [AZ]2 VS 27

Write in PORT Jack Write 2 1 Version clocks tick on writes A : = AY VS = 2 “Done” Key A [AX]0 [AY]2 28

Read in Port Jack Read 2 1 No tick on reads A=? VS = 2 A = AY Key A [AX]0 [AY]2 [AZ]5 29

Read Promotion Ensures a Total Order Jack Read 2 1 A=? VS = 2 Key A [AX]0 [ ? ]2 30

Read Promotion Ensures a Total Order Jack Read 2 1 A=? VS = 2 A = AX Key A [AX]0 [A Immutable X]1 [AX]2 31

Read Promotion Ensures a Total Order Mia Write 2 1 A : = AY VS = 2 “Done” Key A [AX]0 2 [AY]3 32

Track Stable Frontier SF Map ? SF = 3 SFA = 3 SFB = 3 SFC = 5 Mia Read/Write 2 3 1 Advance to stable frontier SFA = 3 Key A [AX]0 2 [AY]3 33

Read-Only Transaction Logic SF Map SF = 3 SFA = 3 SFB = 3 SFC = 5 Jack 3 1 ad e R ? = A 3 = VS Key A [AX]0 [AY]3 [AZ]7 Re ad B= VS ? =3 Key B [BX]0 [BY]1 [BZ]3 34

Read-Only Transaction Logic SF Map SF = 3 SFA = 3 SFB = 3 SFC = 5 Jack 3 1 ad e R ? = A 3 = VS Key A [AX]0 [AY]3 [AZ]7 Re ad B= VS ? =3 Key B [BX]0 [BY]1 [BZ]3 35

Read-Only Transaction Logic SF Map SF = 3 3 SFA = 7 SFB = 3 SFC = 5 Jack 3 1 FA AY = A B= , S =7 Key A [AX]0 [AY]3 [AZ]7 B Z , S F B =3 Key A [BX]0 [BY]1 [BZ]3 36

PORT Is NOC • Reading at the stable frontier ensures reads are non-blocking (N) • Client pre-determined snapshot with VS ensures one-round communication (O) • One VS per read request ensure constant metadata (C) 37

PORT Systems • Scylla-PORT – Base system: Scylla. DB (non-transactional) • Highly optimized sensitive to overhead – NOC + Process-ordered serializability – Supports simple writes (not write transactions) • Eiger-PORT – Base system: Eiger (N, O, C) • Existing read-only and write transactions – NOC + Causal consistency – Supports write transactions 38

Evaluation of Scylla-PORT • To understand – Overhead in latency and throughput compared to simple reads – Performance advantages compared to other protocols, e. g. , OCC. • Experiment configuration – YCSB benchmark with customized parameters for skew and read-to-write ratios – Evaluated latency, throughput, scalability, freshness 39

Latency-Throughput Uniform, 5% Writes Scylla-OCC Scylla-PORT Scylla. DB Higher Throughput Lower Latency 40

Latency-Throughput Zipf = 0. 99, 5% Writes Scylla-OCC Scylla-PORT Scylla. DB 8% 41

Conclusion • Performance-optimal read-only transactions: NOC • The NOCS Theorem for read-only transactions – Impossible to have all of the NOCS properties • The design of PORT – NOC with the strongest consistency to date • Scylla-PORT – Minimum performance overhead compared to simple reads – Significantly outperforms the standard OCC 42