Seneca Remote Mirroring Done Write Minwen Ji Alistair

Seneca: Remote Mirroring Done Write Minwen Ji , Alistair Veitch and John Wilkes HP Labs 27 February 2021 1

Motivations: Reliability and Availability • 2 out of 5 enterprises that experience a disaster go out of business within 5 years [Gartner Report] • Outages cost >$250 K/hour (25%) or >$5 M/hour (4%) [Eagle Rock Online Survey] 2

Our Contributions • A taxonomy of the design choices for remote mirroring • An asynchronous protocol that is designed to handle many kinds and sequences of failures • Checking the correctness of the protocol using I/O automata-based simulation 3

Remote Mirroring Overview App Mirroring Module App Local Remote Primary Secondary Wide Area Network Primary Log App Mirroring Module Secondary Log Competing goals: High performance, low cost, and low data loss 4

Design Choices • Synchronous vs. asynchronous – Propagate update to mirror before vs. after write request returned to application • Divergence: zero bounded, op/byte/time bounded, resource bounded, unbounded – Amount of data allowed to be out-of-sync between mirrors • As-is logging vs. write coalescing – Store all versions vs. a subset of versions of overwritten data in log 5

Seneca’s Choices • Synchronous vs. asynchronous – Low data loss vs. smooth traffic and high performance • Divergence: zero bounded, op/byte/time bounded, resource bounded, unbounded – Low data loss vs. smooth traffic and high availability • As-is logging vs. write coalescing – Little secondary log space vs. low primary log space and low traffic 6

A Taxonomy 4 Async. Bitmap 3 Async. Coalesce 2 Async 1 Sync Perf + Cost – Loss + Avail+ Cost – Loss + 7

Evaluation of Seneca’s Choices Metrics: Impact of asynchronous propagation and write coalescing on WAN traffic and log space Traces Cello 2002 SAP RDW Open. Mail Capacity Length Mean Write Rate 1. 44 TB 24 hours 0. 78 MB/s 4 TB 15 mins 1. 95 MB/s 500 GB 1. 4 hours 0. 34 MB/s 640 GB 1 hour 1. 70 MB/s 8

Simulation Results Mean Traffic vs. Coalescing Interval Traffic CDF w/ Coalescing On/Off Log Size vs. Coalescing Interval • • • Mean traffic: 5 -40% reduction with write coalescing allowed within 30 sec intervals 95 th percentile usage: reduced from 93% of 4 T 3 lines to 85% of 3 T 3 lines Log space: 100 GB log will cover a network outage for 14 -81 hours 9

How To Get Things Right • Hard problems: – Rolling disasters • Primary fails => secondary inconsistent => system inaccessible – Failover dilemmas • Primary fails before propagation • Secondary takes over and continues to update • Old primary returns • Our approach: – Finite state machines 10

Local Seneca State Remote Seneca State Primary State Secondary State 11

Checking Correctness • Simulation – Started with Input/Output Automata (a model checking language) – Constrained random walks in the state space – Implemented in C • Correctness criteria – Coverage, safety and liveness • Latest results – Detected and fixed many non-trivial implementation bugs in a relatively short time – Average failure injections before a bug is detected: 16435 – Mean Time Between Failures for the protocol proper: 4100 years – The latest bug took 1. 77 M writes, 75. 9 K failures, 22. 4 K recoveries and 6. 6 M internal events to detect 12

Summary • A taxonomy of design space for remote mirroring • Evaluation of Seneca’s design choices • A finite state machine description of the Seneca remote mirroring protocol • Checking the correctness of Seneca with simulations 13