Speicher Securing LSMbased KV Stores using Shielded Execution

Speicher Securing LSM-based KV Stores using Shielded Execution Maurice Bailleu, Jörg Thalheim, Pramod Bhatotia Christof Fetzer TU Dresden Michio Honda NEC Labs Kapil Vaswani Microsoft Research

Storage security in the cloud Untrusted infrastructure Storage or query operations User Data How do we ensure security of data in the cloud?

Trusted computing Trusted Execution Environments (TEEs): Hardware extensions for trusted computing, e. g. Intel SGX and ARM Trust. Zone Shielded execution: Run-time framework based on TEE to provide strong security, e. g. HAVEN [OSDI’ 14] and SCONE [OSDI’ 16] Can we also use shielded execution for securing legacy storage systems? Address space Secure memory region (or enclave) Shielded application TEE Intel SGX or ARM Trust. Zone

Research GAP Unfortunately NOT! At least not directly ; -) • Shielded execution is mainly designed for securing (volatile) in-memory computation • However, storage systems require securing non-volatile state, i. e. , persistent state on an untrusted storage medium across reboot, crash, or migration Research challenge: How to extend the trust beyond the "secure but stateless enclave" to the "untrusted storage in stateful setting"?

Our contribution A secure persistent LSM-based KV store for the untrusted computing infrastructure Security properties: • Confidentiality • Unauthorized data access is prohibited • Integrity • Unauthorized change to data is detected • Freshness • Stale state of data is detected (rollback / forking attacks)

Outline • Motivation • Challenges • Design • Evaluation

Challenge #1: Enclave physical memory • KV stores keep data in-memory for fast operations • Unfortunately, EPC is limited (only ~94 Mi. B available) • To support larger memory region, SGX supports secure paging • However, EPC paging incurs high overheads (2 -2000 X) Enclave physical memory (EPC) > 128 Mi. B Redesigned in-memory LSM data structure (Mem. Table) to overcome the enclave physical memory limitation 128 Mi. B EPC paging

Challenge #2: Untrusted storage • Persistent KV stores persist data on untrusted storage (SSDs) • Trust of the enclave does not naturally extends to untrusted storage • Further, the security properties need to valid across system reboot, crash, or migration Trusted enclave (Volatile memory Region) Untrusted storage Redesigned on-disk LSM data structures (SSTable and log files) to extend the trust beyond the enclave memory

Challenge #3: I/O syscall • Storage systems issue frequent I/O syscalls • Thread executing the syscall need to exit the enclave • Enclave exit operations are expensive since they require TLB flushing, security checks, etc. Trusted enclave I/O call Exit enclave to issue the syscall Designed a direct I/O library for shielded execution based on SPDK for fast I/O without exiting enclave

Challenge #4: Trusted counter • To protect system against rollback or forking attacks, we need trusted monotonic counters • SGX counters are extremely slow (250 ms) and wear out in a couple of days! • Unsuitable for modern KV stores SGX trusted counters Designed an asynchronous trusted counter interface taking advantage of persistency guarantees of modern KV stores

Outline • Motivation • Challenges • Design • Background • System components • Algorithms • Evaluation

Background: LSM-based KV store Level 1 SST Level 2 SST …. Log files Main memory Mem. Table Level 0 …. MANIFEST Persistent storage (SSD) SST …. Write Ahead Log (WAL)

Outline • Motivation • Challenges • Design • Background • System components • Algorithms • Evaluation

System overview Host memory Trusted enclave memory Trusted counter Storage engine (Rocks. DB) Mem. Table DMA I/O lib Speicher controller SSD Operating system SSTable Log files

Mem. Table Enclave (Key part) Host (Value part) Value … K H P Key Hash Ptr Value Split the Mem. Table into two parts: (1) Key part --> stored in the enclave (2) Value part --> stored in the untrusted host memory

SSTable KV KV … KV Block #1 … Merkle tree Block #n H 1 KV KV … Hn Storing the hashes of the blocks allows to make integrity checks for every KV pair, while still allowing fast lookup.

Log files: WAL and Manifest Record 1 Trusted Counter Value 1 Append #1 Hash 1 R 2 C 2 H 2 Append #2 . . . Log files guarantee freshness of the data based on the trusted counter

Asynchronous monotonic counter Unstable period Time Synchronous Async point increment () Synchronous point Async increment () Expected time Time to persist data in modern KV stores can overlap with the unstable period

Outline • Motivation • Challenges • Design • Background • System components • Algorithms • Evaluation See the paper for more algorithms

Algorithm #1: Put (K, V)

Algorithm #1: Put 4. Get stable counter and expected time 6. Return success and expected time 3. Increment trusted counter 2. Write record to WAL 5. Write to Mem. Table 1. Put request Enclave Trusted Counter Storage engine Speicher controller Client Host Memory Mem. Table I/O lib DMA SSD SST, Log files

Algorithm #2: Get (K)

Algorithm #2: Get 1. Get Request 2. Search in Mem. Table 3. Search in SST files 4. Return Value Enclave Trusted Counter Storage engine Speicher controller Client Host Memory Mem. Table I/O lib DMA SSD SST, Log files

Outline • • Motivation Challenges Design Evaluation

Evaluation • Questions: 1. What is the performance of the direct I/O library? 2. What is the performance overhead of SPEICHER? See the paper for more results • Experimental setup: • Intel Xeon E 3 -1270 v 5 (3. 60 GHz, 4 cores, 8 hyper-threads) -- Skylake w/ SGX • Intel DC P 3700 SSD (400 GB, PCIe x 4) -- w/ SPDK

Q 1 : Throughput of the I/O lib Throughput [GB/s] 1. 4 1. 2 Speicher Native 1 0. 8 0. 6 0. 4 0. 2 0 2 4 Blocksize [Ki. B] 8 Our I/O library performance at par to native SPDK 16

Relative Overhead Q 2: Overheads of Speicher 35 30 25 20 15 10 5 0 Lower is better 90/10 80/20 Workload [R/W] 100/0 Reasonable overhead, which results mostly from en-/decryption of KV pairs!

Summary Speicher: A secure LSM based KV-Store Security properties: confidentiality + integrity + freshness Challenge: How to extend the trust beyond the "secure but stateless enclave" to the "untrusted storage in stateful setting"? Contributions Secure LSM data structure Implementation based on Direct I/O library Asynchronous counters Algorithms for secure storage and query operations Reasonably overheads

Backup!

Native Mem. Table Memory K V Key Value 30

Native SSTable KV KV … KV 31

Native log files: WAL and MANIFEST Record 1 Append #1 Record 2 Append #2 . . .