PMTEST Testing Persistent Memory Applications Samira Khan STORAGE

PMTEST Testing Persistent Memory Applications Samira Khan

STORAGE MEMORY CPU TWO-LEVEL STORAGE MODEL Ld/St DRAM FILE I/O VOLATILE FAST BYTE ADDR NONVOLATILE SLOW BLOCK ADDR 2

STORAGE MEMORY CPU TWO-LEVEL STORAGE MODEL Ld/St DRAM FILE I/O NVM PCM, STT-RAM VOLATILE FAST BYTE ADDR NONVOLATILE SLOW BLOCK ADDR Non-volatile memories combine characteristics of memory and storage 3

VISION: UNIFY MEMORY AND STORAGE CPU NVM PERSISTENT MEMORY Ld/St Provides an opportunity to manipulate persistent data directly in memory Avoids reading and writing back data to/from storage 4

CHALLENGE: NEED ALL STORAGE SYSTEM SUPPORTS APPLICATION OS/SYSTEM Ld/St MEMORY STORAGE FILE I/O Crash Consistency NVM APPLICATION OS/SYSTEM Ld/St PERSISTENT MEMORY Overhead in OS/storage layer overshadows the benefit of nanosecond access latency of NVM 5

CHALLENGE: NEED ALL STORAGE SYSTEM SUPPORTS APPLICATION OS/SYSTEM Ld/St MEMORY STORAGE FILE I/O Crash Consistency NVM APPLICATION Ld/St PERSISTENT MEMORY Not the operating system, Application layer is responsible for crash consistency in PM 6

NON-VOLATILE MEMORY PERSISTENT MEMORY Programming Persistent Memory Applications CHALLENGE: PM Programming is Hard! Requirements and Key Ideas PMTEST: Interface and SIGMETRICS’ 14 Mechanism ASPLOS’ 19 Results and Conclusion 7

PERSISTENT MEMORY PROGRAMMING • Support for crash consistency have two fundamental guarantees • Durability: writes become persistent in PM • Ordering: one write becomes persistent in PM before another Core Volatile Cache Durability Guarantee: writeback data from cache Flush A Persistent PM-DIMM 8

PERSISTENT MEMORY PROGRAMMING • Support for crash consistency have two fundamental guarantees • Durability: writes become persistent in PM • Ordering: one write becomes persistent in PM before another Core Volatile Cache Persistent B A PM-DIMM Ordering Guarantee: Write A before B Writeback A Barrier Writeback B 9

PERSISTENT MEMORY PROGRAMMING PM Programming Expert • Uses low-level primitives • Understands the hardware • Understands the algorithm Normal • Uses a high-level interface • Does not need to know details of hardware or algorithm Two different ways to program persistent applications 10

PERSISTENT MEMORY PROGRAMMING (LOW-LEVEL) • Hardware provides low-level primitives for crash consistency • Exposes instructions for cache flush and barriers • sfence, clwb from x 86 • dc cvap from ARM • Academic proposals, e. g. , ofence, dfence. x 86 clwb sfence PM-DIMM ARM dc cvap dsb PM-DIMM New Instr PM-DIMM [Kiln’ 13, Thy. NVM’ 15, DPO’ 16, JUSTDOLogging’ 16, ATOM’ 17, HOPS’ 17, etc. ] 11

PROGRAMMING USING LOW-LEVEL PRIMITIVES 1 void list. Append(item_t new_val) { 2 node_t* new_node = new node_t(new_val); 3 new_node->next = head; 4 head = new_node; 5 persist_barrier(); Writes 6 } to PM can reorder Create new_node Update head pointer Writeback updates Head In cache new_node is lost after failure Inconsistent linked list 12

PROGRAMMING USING LOW-LEVEL PRIMITIVES 1 void list. Append(item_t new_val) { 2 node_t* new_node = new node_t(new_val); 3 new_node->next = head; persist_barrier(); 4 head = new_node; 5 persist_barrier(); 6 } Enforce writeback before changing head Head In In cache PM Ensuring crash consistency with low-level is HARD! Consistent linkedprimitives list 13

PERSISTENT MEMORY PROGRAMMING PM Programming Expert • Uses low-level primitives • Understands the hardware • Understands the algorithm Normal • Uses a high-level interface • Does not need to know details of hardware or algorithm 14

PERSISTENT MEMORY PROGRAMMING (HIGH-LEVEL) • Libraries provide transactions on top of low-level primitives • Intel’s PMDK • Academic proposals Atomic. Begin { Append a new node; } Atomic. End; Uses logging mechanisms to atomically commit the updates [NV-Heaps’ 11, Mnemosyne’ 11, ATLAS’ 14, REWIND’ 15, NVL-C’ 16, NVThreads’ 17 LSNVMM’ 17, etc. ] 15

PROGRAMMING USING TRANSACTIONS 1 void List. Append(item_t new_val) { 2 TX_BEGIN { 3 node_t *new_node = make. Node(new_val); 4 TX_ADD(list. head, sizeof(node_t*)); 5 List. head = new_node; 6 List. length++; 7 } TX_END 8 } Create new_node backup head Update length is not backed up before update! 16

PROGRAMMING USING TRANSACTIONS 1 void List. Append(item_t new_val) { 2 TX_BEGIN { 3 node_t *new_node = make. Node(new_val); 4 TX_ADD(list. head, sizeof(node_t*)); 5 List. head = new_node; 6 List. length++; TX_ADD(list. length, sizeof(unsigned)); 7 } TX_END Backup length 8 } before update Ensuring crash consistency with transactions is still HARD! 17

PERSISTENCE MEMORY PROGRAMMING IS HARD PM Programming Expert • Uses low-level primitives • Understands the hardware • Understands the algorithm Normal • Uses a high-level interface • Does not need to know details of hardware or algorithm Both expert and normal programmers can make mistakes 18

PERSISTENT MEMORY PROGRAMMING IS HARD Detect crash consistency bugs We need a tool to detect crash consistency bugs! 19

NON-VOLATILE MEMORY PERSISTENT MEMORY Programming Persistent Memory Applications CHALLENGE: PM Programming is Hard! Requirements and Key Ideas PMTEST: Interface and SIGMETRICS’ 14 Mechanism ASPLOS’ 19 Results 20

REQUIREMENTS OF THE TOOL Flexible PM Libraries Kernel Modules Existing HW Custom Programs Future HW and Models [PMDK, NV-Heaps’ 11, Mnemosyne’ 11, ATLAS’ 14, REWIND’ 15, NVL-C’ 16, NVThreads’ 17 LSNVMM’ 17, etc. ] [PMFS’ 14, BPFS’ 09, NOVA’ 16, NOVA-Fortis’ 17, Strata’ 17, SCMFS’ 11 etc. ]store, [DPO’ 16, HOPS’ 17, etc. ] E. g. , custom database, key-value etc. ARM, [x 86, etc. ] 21

PMTEST KEY IDEAS: FLEXIBLE • Many different programming models and hardware primitives available PM Program Call library PMDK Library write, sfence, clwb x 86 PM Program PM Kernel Module Call library Mnemosyne Library write, dc cvap, dsb ARM write, sfence, clwb x 86 The challenge is to support different hardware and software models 22

PMTEST KEY IDEAS: FLEXIBLE Operations that maintain crash consistency are similar: ordering and durability guarantees PM Program Call library PMDK Library write, sfence, clwb x 86 PM Program PM Kernel Module Call library Mnemosyne Library write, dc cvap, dsb ARM write, sfence, clwb x 86 Our key idea is to test for these two fundamental guarantees which in turn can cover all hardware-software variations 23

PMTEST KEY IDEAS: FAST • Prior work [Yat’ 14] uses exhaustive testing O(n!) n sfence write A write B write C. . . sfence write B write A write C. . . sfence write C write B write A. . . sfence write A write C write B. . . sfence write B write C write A. . . sfence write C write A write B. . . sfence Recoverable? Exhaustive testing is time consuming and not practical 24

PMTEST KEY IDEAS: FAST • Reduce test time by using only one dynamic trace Runtime Trace Persistent Memory Application sfence write C write B write A. . . sfence Recoverable? A significant improvement over O(n!) testing 25

PMTEST KEY IDEAS: FAST • PMTest infers the persistence interval from PM operation trace The interval in which a write can possibly become persistent write A clwb A sfence write B clwb B sfence Trace A A persists before B B Timeline A disjoint interval indicates that no re-ordering in the hardware will lead to a case where A does not persist before B 26

PMTEST KEY IDEAS: FAST • PMTest infers the persistence interval from PM operation trace The interval in which a write can possibly become persistent write A write B clwb A sfence clwb B sfence Trace A Interleaving A may NOT persist before B B Timeline An overlapping interval indicates that there is a case where A does not persist before B 27

PMTEST KEY IDEAS: FAST • PMTest infers the persistence interval from PM operation trace The interval in which a write can possibly become persistent write A write B clwb A sfence clwb B A persists before B? sfence Trace A B No Timeline Querying the trace can detect any violation in ordering and durability guarantee at runtime 28

NON-VOLATILE MEMORY PERSISTENT MEMORY Programming Persistent Memory Applications CHALLENGE: PM Programming is Hard! Requirements and Key Ideas PMTEST: Interface and SIGMETRICS’ 14 Mechanism ASPLOS’ 19 Results and Conclusion 29

PMTEST OVERVIEW Testing Annotation Persistent Memory Application Offline Checking Rules PMTesting Results Online 30

PMTEST OVERVIEW Testing Annotation Persistent Memory Application Offline Checking Rules PMTesting Results Online 31

PMTEST INTERFACE PMTest Expert • Assertion-like low-level interface • Check behavior vs. specification Normal • High-level interface • Minimize programmer’s effort • Automatically inject low-level checkers PMTest provides two different interfaces 32

PMTEST LOW-LEVEL INTERFACE • Two low-level checkers • is. Ordered. Before(A, size. A, B, size. B) Checks whether A is persisted before B (Ordering guarantee) • Is. Persisted(A, size. A) Checks whether A has been written back to PM (Durability guarantee) 33

PMTEST LOW-LEVEL INTERFACE • Two low-level checkers • is. Ordered. Before(A, size. A, B, size. B) Checks whether A is persisted before B (Ordering guarantee) • Is. Persisted(A, size. A) Checks whether A has been written back to PM (Durability guarantee) • Help check if implementation meets specification for • Programs/kernel modules based on low-level primitives • PM libraries 34

EXAMPLE void hash. Map. Remove() { . . . remove(buckets->bucket[hash]); count--; Check if count has been persisted before rebuilding persist_barrier(); . . . hashmap_rebuild(); is. Ordered. Before(&count, sizeof(unsigned), &hashmap, sizeof(hashmap)); is. Persisted(&hashmap, size); } Check if all updates have been persisted in rebuilding PMTest helps the programmers to reason about the code *This example is inspired by hashmap_atomic from PMDK 35

PMTEST LOW-LEVEL INTERFACE • Two low-level checkers • is. Ordered. Before(A, size. A, B, size. B) Check whether A is persisted before B (Ordering guarantee) • Is. Persisted(A, size. A) Check whether A has been written back to PM (Durability guarantee) • Help check if implementation meets specification for • Programs/kernel modules based on low-level primitives • PM libraries • Further enables high-level checkers to automate testing 36

PMTEST HIGH-LEVEL INTERFACE • Currently provides high-level checkers for PMDK transactions • Automatically detects crash consistency bugs void List. Append(item_t new_val) { TX_CHECKER_START; //Start of TX checker TX_BEGIN { node_t *new_node = make. Node(new_val); TX_ADD(list. head, sizeof(node_t*)); List. head = new_node; List. length++; } TX_END TX_CHECKER_END; //End of TX checker } Automatically check if there is a backup before update Automatically check if all updates have been persisted * This example does not include initialization and communication with PMTest 37

PMTEST HIGH-LEVEL INTERFACE • Currently provides high-level checkers for PMDK transactions • Automatically detects crash consistency bugs • If all updates have been persisted at the end of the transaction • If there is a backup before update during the transaction • Automatically detects performance bugs • Redundant log/backup • Duplicated writeback/flush operations (for all programs) High-level checkers minimize programmer’s effort 38

PMTEST OVERVIEW Testing Annotation Persistent Memory Application Offline Checking Rules PMTesting Results Online 39

PMTEST CHECKING MECHANISM for (. . . ) { TX_CHECKER_START; TX_BEGIN; . . . TX_END; TX_CHECKER_END; PMTest_SEND_TRACE; } At Runtime . . . write A write B clwb B sfence TX_END Auto inject low-level checkers for high-level checkers Checking Engine Result: A is not persistent! PM Trace The checking engine tests the trace 40

CHECKING ENGINE ALGORITHM • Infer the persistence interval in which a write can become persistent • Check the interval against the low-level checkers sfence write A clwb A write B sfence is. Ordered. Before A B is. Persist B PM Trace A B A and B can be persisted any time sfence Time B may not persist Persistence Interval Our interval-based check enables faster testing 41

NON-VOLATILE MEMORY PERSISTENT MEMORY Programming Persistent Memory Applications CHALLENGE: PM Programming is Hard! Requirements and Key Ideas PMTEST: Interface and SIGMETRICS’ 14 Mechanism ASPLOS’ 19 Results and Conclusion 42

Platform METHODOLOGY CPU: 8 -core Skylake 2. 1 GHz, OS: Ubuntu 14. 04, Linux kernel 4. 4 Memory: 64 GB DDR 4 NVM: 64 GB Battery-backed NVDIMM Workloads Micro-benchmarks (from PMDK) • C-Tree • B-Tree • RB-Tree • Hash. Map Real-world workloads • PM-optimized file system • Intel’s PMFS (kernel module) • PM-optimized database • Redis (PMDK Library) • Memcached (Mnemosyne Library) Baselines • No testing tool • With Intel’s Pmemcheck (only for PMDK-based programs) 43

. . . H as h. M ee BTr (Transactio n) R re BT -T C e 8 6 4 2 0 re e Speedup vs. Pmemcheck MICRO-BENCHMARK PMTest is 7. 1 X faster than Pmemcheck 44

22 X slowdown with Pmemcheck 2. 0 1. 5 e Av er ag FS PM is ed +LRU +OLTP +Filebench M M +Memslap +YCSB R ed ch em ca ch ed 1. 0 em ca PMTest Overhead REAL-WORLD WORKLOADS PMTest has < 2 X overhead in real-world workloads 45

BUG DETECTION • Validated with • 42 synthetic bugs injected to microbenchmarks • 3 existing bugs from commit history • New bugs found • 1 crash consistency bug in PMDK applications • 1 performance bug in PMFS • 1 performance bug in PMDK applications 46

CONCLUSION • It is hard to guarantee crash consistency in persistent memory applications PMTest pmtest. persistentmemory. org • Our tool PMTest is fast and flexible • Flexible: Supports kernel modules, custom PM programs, transaction-based programs • Fast: Incurs < 2 X overhead in real-workload applications • PMTest has detected 3 new bugs in PMFS and PMDK applications 47