Attested AppendOnly Memory Making Adversaries Stick to their

Attested Append-Only Memory: Making Adversaries Stick to their Word Byung-Gon Chun (ICSI) October 15, 2007 Joint work with Petros Maniatis (Intel Research, Bekeley), Scott Shenker, and John Kubiatowicz (UC Berkeley) 1

Context 2

Context 3

Centralized Server Alice inc (1) 11 Bob dec (1) 10 11 Counter 10 Alice: inc(1) Counter 11 Bob: dec(1) Counter 10 4

Centralized Server Alice: inc(1) Counter 11 Alice Bob inc (1) Bob: dec(1) Counter 10 11 10 Counter dec (1) Bob: dec(1) Counter 9 9 Alice: inc(1) Counter 10 Linearizability: 1. A serial schedule of operations 2. External order Liveness: making progress 5

Replicated Servers Linearizability and liveness Alice 1110 inc (1) S 1: 11 S 2: 11 S 3: 1 1 Bob 3 11 10 1 2 4 10 11 Byzantine Fault Tolerant Replicated Systems e. g. , PBFT[TOCS 02] BFT algorithms can tolerate up to 1/3 faulty replicas. e. g. , f out of 3 f+1 If the fault assumption is violated, there is no guarantee! 6

Servers Equivocating to Servers Sequence 101 Alice: inc(1) Alice 3 inc (1) 1 10 11 dec (9) Bob 2 4 10 9 Sequence 101 Bob: dec(1) 7

Servers Equivocating to Clients Sequence 101 Alice: inc(1) Alice S 3: 11 3 1 11 S 1: 11 S 2: 11 Bob 9 S 1: 9 S 2: S 4: 9 2 4 9 Sequence 101 Bob: dec(1) 8

Questions • Does preventing equivocation help at all? – Can we improve upon the 1/3 Byzantine fault bound? • How do we prevent equivocation? – Is there any minimal system support? 9

Talk Outline • Introduction and Motivation • Attested Append-Only Memory (A 2 M) • A 2 M Protocols • Evaluation • Conclusion 10

High-level View Service 1 3 2 4 Application + Protocol Equivocation guard Non-equivocation 11

Attested Append-Only Memory (A 2 M) • A set of numbered logs • Each log entry contains – Sequence number – Stored value – Crypto digest of entire log • lookup / end – Get a log entry – Attest (sequence number, value, history digest) – Attest freshness – Attest the end of log • append / advance – Cannot overwrite 12

An A 2 M Usage Pattern Replicas need to agree on msg in sequence number n A 2 M Sending lookup(n) replica <n, h(msg)> append(h(msg)) msg, <n, h(msg)> Result: the sending replica is forced to say the same msg for n 13

A 2 M Implementation Scenarios Third-party service Software isolation Remote Local Faulty app Faulty operator Faulty app Virtual machine Local Faulty app Virtual machine monitor Trusted hardware Local Faulty app Faulty operator 14

Talk Outline • Introduction and Motivation • Attested Append-Only Memory (A 2 M) • A 2 M Protocols • Evaluation • Conclusion 15

A 2 M protocols • A 2 M State Machine Replication – A 2 M-PBFT-EA • A 2 M-Storage (SUNDR-like) • A 2 M-Q/U 16

Background: PBFT • Assumptions – Byzantine faults – Secure cryptography – Weak synchrony • Guarantee linearizability and liveness with up to f faults out of 3 f+1 replicas • Three phase protocol • View change 17

Background: PBFT Execution Agreement Primary s 1 Preprepare Prepare Commit Quorum = 3 s 2 Execute req, resp s 3 Reply s 4 time [1, a] Request Quorum: matching messages from different replicas Client 1 [1, b] Client 2 18

A 2 M-PBFT-E(Execution) Primary s 1 Preprepare Prepare Commit Quorum = 3 s 2 Execute Request log req, resp, <seq, req, hist> s 3 Reply s 4 time Request Client 1 Attested by A 2 M 19

Intuition Client 1 Client 2 S 1: req 1, resp 1, <n, h(req 1), h(hist 1)> S 2: req 1, resp 1, <n, h(req 1), h(hist 1)> S 3: req 1, resp 1, <n, h(req 1), h(hist 1)> S 1: req 2, resp 2, <n, h(req 2), h(hist 2)> S 2: req 2, resp 2, <n, h(req 2), h(hist 2)> S 4: req 2, resp 2, <n, h(req 2), h(hist 2)> 20

Liveness Problems of A 2 M-PBFT-E Quorum = 3 Primary s 1 Preprepare Prepare Commit Execute s 2 s 3 req, resp, <seq, req, hist> Reply s 4 time Request Client 1 Attested by A 2 M 21

A 2 M-PBFT-EA(Execution+Agreement) Primary s 1 Preprepare Prepare Commit Quorum = 3 Execute s 2 s 3 req, resp, <seq, req, hist> Reply s 4 time Request Client 1 Attested by A 2 M 22

A 2 M-PBFT-EA (2 f + 1 replicas) Primary s 1 Preprepare Prepare Commit Quorum = 2 Execute s 2 s 3 req, resp, <seq, req, hist> Reply time Request Client 1 Attested by A 2 M 23

Intuition PBFT (3 f + 1) Quorum 1 (2 f + 1) Quorum 2 (2 f + 1) f+1 A 2 M-PBFT-EA (2 f + 1) Quorum 1 (f + 1) Quorum 2 (f + 1) 1 1 A 2 M 1 non-faulty replica 24

A 2 M-PBFT-EA (Three phase) Primary s 1 Preprepare Prepare Commit Quorum = 2 Execute s 2 s 3 req, resp, <seq, req, hist> Reply time Request Client 1 Attested by A 2 M 25

A 2 M-PBFT-EA (Two phase) Primary s 1 Prepare Commit Execute s 2 s 3 req, resp, <seq, req, hist> Reply Request time Client 1 Attested by A 2 M 26

Other Results • A 2 M-PBFT-EA View change • A 2 M-Storage: achieve linearizability in an untrusted single-server system • A 2 M-Q/U: require 4 f+1 replicas (instead of 5 f+1 replicas) to tolerate f faults 27

Talk Outline • Introduction and Motivation • Attested Append-Only Memory (A 2 M) • A 2 M Protocols • Evaluation • Conclusion 28

Protocol Trade-offs 1/3 1/2 PBFT 3 f+1 2/3 A 2 M-PBFT-EA 29

Evaluation Setup • Implemented A 2 M-PBFT-E and A 2 M-PBFT-EA • A 2 M protocols use signatures or MACs for authentication • Four replicas in a LAN. Each replica has its own A 2 M. • Microbenchmarks – Null operation with various request or response sizes • Macrobenchmarks: NFS – Software package compilation 30

Macro-benchmarks: NFS -S -PBFT -A 2 MPBFT-E (sig) -A 2 MPBFT-E (MAC) Step Copy Uncompress Untar Configure Make Clean Total (seconds) 0. 219 1. 015 2. 322 12. 748 7. 241 0. 180 0. 709 3. 027 4. 448 12. 412 7. 461 0. 298 1. 026 4. 378 6. 826 19. 173 9. 778 0. 640 23. 725 28. 355 (0%) 41. 821 (47. 5%) 0. 728 3. 103 4. 553 12. 659 7. 500 0. 312 -A 2 MPBFTEA (3 phase) (sig) (MAC) 2. 141 8. 601 12. 896 26. 181 11. 379 0. 742 0. 763 3. 236 4. 669 13. 040 7. 510 0. 311 28. 854 61. 940 (1. 8%) (118. 4%) 29. 528 (4. 1%) 31

Broader Implications Trustworthy system Small trusted primitives Untrusted Trustworthy system Untrusted + Untrusted • What small trusted primitives to put to make systems better – e. g. , trusted logical clocks for weak consistency guarantees – e. g. , network interface card attestation • More classes of components with different fault characteristics – trusted, semi-trusted, untrusted 32

Conclusions • Present A 2 M, a small trusted, log-based primitive – Simple and easily implementable – Prevent equivocation • Improve fault tolerance by forcing servers to commit to a single history of operations – Improve fault bounds of BFT state machine replication – Achieve linearizability in an untrusted single-server system – The benefits are achieved with small performance overhead • A 2 M has broader implications on structuring trustworthy systems 33

Thank you! Questions? SOSP 2007 34

Related Work • Weaken the guarantee – fork* consistency [NSDI 07] – fork consistency [OSDI 04] • Standard trusted hardware like TPM – does not improve the fault bound • Auditing – Peer. Review [SOSP 07], CATS [FAST 07] • Shared file servers – SUNDR[OSDI 04], Ivy [OSDI 02], Plutus[FAST 03] • Separating agreement from execution • Symmetric faults – hybrid fault model • Group communication 35