Optimizing ZFS for Block Storage Will Andrews Justin

Optimizing ZFS for Block Storage Will Andrews, Justin Gibbs Spectra Logic Corporation

Talk Outline • Quick Overview of ZFS • Motivation for our Work • Three ZFS Optimizations – COW Fault Deferral and Avoidance – Asynchronous COW Fault Resolution – Asynchronous Read Completions • • • Validation of the Changes Performance Results Commentary Further Work Acknowledgements

ZFS Feature Overview • File System/Object store + Volume Manager + RAID • Data Integrity via RAID, checksums stored independently of data, and metadata duplication • Changes are committed via transactions allowing fast recovery after an unclean shutdown • Snapshots • Deduplication • Encryption • Synchronous Write Journaling • Adaptive, tiered caching of hot data

Simplified ZFS Block Diagram Presentation Layer ZFS POSIX Layer ZFS Volumes Lustre Objects and Caching Data Management Unit Layout Policy Storage Pool Allocator CAM Target Layer File, Block, or Object Access TX Management & Spectra Optimizations Object. Here Coherency Configuration & Control Volumes, RAID, zfs(8), zpool(8) Snapshots, I/O Pipeline

ZFS Records or Blocks • ZFS’s unit of allocation and modification is the ZFS record. • Records range from 512 B to 128 KB. • Checksum for each record are verified when the record is read to ensure data integrity. • Checksums for a record are stored in the parent record (indirect block, or DMU node) that reference it, which are themselves checksummed.

Copy-on-Write, Transactional, Semantics • ZFS never overwrites a currently allocated block – A new version of the storage pool is built in free space – The pool is atomically transitioned to the new version – Free space from the old version is eventually reused • Atomicity of the version update is guaranteed by transactions, just like in databases.

ZFS Transactions • Each write is assigned a transaction. • Transactions are written in batches called “transaction groups” that aggregate the I/O into sequential streams for optimum write bandwidth. • TXGs are pipelined to keep the I/O subsystem saturated – Open TXG: Current version of Objects. Most changes happen here. – Quiescing TXG: Waiting for writers to finish changes to in-memory buffers. – Synching TXG: buffers being committed to disk.

Copy on Write In Action überblock Root of Storage Pool DMU Node Root of an Object (file) In d Ob irec jec t L t E ink xp ag an e f sio or n Write Indirect Block Data Block Data Block

Tracking Transaction Groups Time DMU Buffer ject b O t n Curre rsion Ve Open TXG Quiescing TXG Syncing TXG Dirty Record Data Record Data • DMU Buffer (DBUF): Metadata for ZFS blocks being modified • Dirty Record: Syncher information for committing the data.

Performance Demo

Performance Analysis When we write an existing block, we must mark it dirty… void dbuf_will_dirty(dmu_buf_impl_t *db, dmu_tx_t *tx) { int rf = DB_RF_MUST_SUCCEED | DB_RF_NOPREFETCH; ASSERT(tx->tx_txg != 0); ASSERT(!refcount_is_zero(&db->db_holds)); DB_DNODE_ENTER(db); if (RW_WRITE_HELD(&DB_DNODE(db)->dn_struct_rwlock)) rf |= DB_RF_HAVESTRUCT; DB_DNODE_EXIT(db); (void) dbuf_read(db, NULL, rf); (void) dbuf_dirty(db, tx); }

Doctor, it hurts when I do this… • Why does ZFS Read on Writes? – ZFS records are never overwritten directly – Any missing old data must be read before the new version of the record can be written – This behavior is a COW Fault • Observations – Block consumers (Databases, Disk Images, FC LUN, etc. ) are always overwriting existing data. – Why read data in a sequential workload when you are destined to discard it? – Why force the writer to wait to read data?

Optimization #1 Deferred Copy On Write Faults How Hard Can It Be? Fam ous Wor Last ds

DMU Buffer State Machine (Before) Read Issued READ Read Complete UNCACHED Truncate CACHED Copy Complete Full Block Write FILL Teardown EVICT

DMU Buffer State Machine (After)

Tracking Transaction Groups Time Open TXG DMU Buffer UNCACHED Dirty Record

Tracking Transaction Groups Time Open TXG DMU Buffer PARTIAL|FILL Dirty Record Data

Tracking Transaction Groups Time Open TXG DMU Buffer PARTIAL Dirty Record Data

Tracking Transaction Groups Time DMU Buffer PARTIAL Open TXG Quiescing TXG Dirty Record Data

Tracking Transaction Groups Time Open TXG DMU Buffer PARTIAL Quiescing TXG Syncing TXG Dirty Record Data Record Data Syncer Processes Record

Tracking Transaction Groups Time Open TXG DMU Buffer READ Quiescing TXG Syncing TXG Dirty Record Data Record Data Syncer Processes Record Read Buffer Dispatch Synchronous Read

Tracking Transaction Groups Time Open TXG DMU Buffer READ Quiescing TXG Syncing TXG Dirty Record Data Record Data Syncer Processes Record Merge Read Buffer Synchronous Read Returns

Tracking Transaction Groups Time DMU Buffer CACHED Open TXG Quiescing TXG Syncing TXG Dirty Record Data Record Data

Optimization #2 Asynchronous Fault Resolution

Issues with Implementation #1 • Syncer stalls due to synchronous resolve behavior. • Resolving reads that are known to be needed are delayed. – Example: a modified version of the record is created in a new TXG • Writers should be able to cheaply start the resolve process without blocking. • The syncer should operate on multiple COW faults in parallel.

Complications • Split Brain – ZFS record can have multiple personality disorder • Example: Write, truncate, write again, all in flight at the same time with a resolving read. – Term reflects how dealing with this issue made us feel. • Chaining syncer’s write to the resolving read – This read may have been started in advance of syncer processing due to a writer noticing that resolution is necessary.

Optimization #3 Asynchronous Reads

Thread Blocking Semantics Callback Semantics ZFS – Block Diagram Presentation Layer ZFS Posix Layer ZFS Volumes Lustre Objects and Caching Data Management Unit Layout Policy Storage Pool Allocator CAM Target Layer Configuration & Control zfs(8), zpool(8)

Asynchronous DMU I/O • Goal: Get as much I/O in flight as possible • Uses Thread Local Storage (TLS) – Avoid lock order reversals – Avoid modifications in APIs just to pass down a queue. – No lock overhead due to it being per-thread • Refcounting while issuing I/Os to make sure callback is not called until entire I/O completes

Results

Bugs, bugs… Deadlocks n o i t rup Sl r o c a Dat Missed e eepin g hold py o c b o t s vents ing no n-slee pable locks t n e m u g Page faults ng ar o r w Unpro tecte Bad comments d crit ing t ical se n u o c t ref ctions c e r r o Inc ks a e l ory Mem Insufficient interlocking Invalid state machine transitions Split brain conditions Disclaimer: This is not a complete list.

Validation • ZFS has many complex moving parts • Simply thrashing a ZFS is not a sufficient test – Many hidden parts make use of the DMU layer and are not directly involved in data I/O or at all • Extensive modifications of the DMU layer require thorough verification – Every object in ZFS uses the DMU layer to support its transactional nature

Testing, testing… • Many more asserts added • Solaris Test Framework ZFS test suite – Extensively modified to (mostly) pass on Free. BSD – Has ~300 tests, needs more • ztest: Unit (ish) test suite – Element of randomization requires multiple test runs – Some test frequencies increased to verify fixes • xdd: Performance tests – Finds bugs involving high workloads

Cleanup & refactoring • DMU I/O APIs rewritten to allow issuing async IOs, minimize hold/release cycles, & unify API for all callers • DBUF dirty restructured – Now looks more like a checklist than an organically grown process – Broken apart to reduce complexity and ease understanding of its many nuances

Performance results almost • It goes 3 -10 X faster! Without breaking^anything! • Results that follow are for the following config: – – RAIDZ 2 of 4 2 TB SATA drives on 6 Gb LSI SAS HBA Xen HVM Dom. U w/ 4 GB RAM, 4 cores of 2 GHz Xeon 10 GB ZVOL, 128 KB record size Care taken to avoid cache effects

1 Thread Performance Results 450 400 350 300 250 200 150 100 50 0 Aligned 128 K Sequential Write (MB/s) Aligned 16 K Sequential Write Unaligned 128 K Sequential (MB/s) Write (MB/s) Before After 16 K Random Write (IOPS) 16 K Random Read (IOPS)

10 Thread Performance Results 400 350 300 250 200 150 100 50 0 Aligned 128 K Sequential Write Aligned 16 K Sequential Write Unaligned 128 K Sequential (MB/s) Write (MB/s) Before After 16 K Random Write (IOPS) 16 K Random Read (IOPS)

Commentary • Commercial consumption of open source works best when it is well written and documented – Drastically improved comments, code readability • Community differences & development choices – Sun had a small ZFS team that stayed together – Free. BSD has a large group of people who will frequently work on one area and move on to another – Clear coding style, naming conventions, & test cases are required for long-term maintainability

Further Work • Apply deferred COW fault optimization to indirect blocks – Uncached metadata still blocks writers and this can cut write performance in half • Required indirect blocks should be fetched asynchronously • Eliminate copies and allow larger I/O cluster sizes in the SPA clustered I/O implementation • Improve read prefetch performance for sequential read workloads • Hybrid RAIDZ and/or more standard RAID 5/6 transform • All the other things that have kept Kirk working on file systems for 30 years.

Acknowledgments • Sun’s original ZFS team for developing ZFS • Pawel Dawidek for the Free. BSD port • High. Cloud Security for the Free. BSD port of the STF ZFS test suite • Illumos for continuing open source ZFS development • Spectra Logic for funding our work

Questions? Preliminary Patch Set: http: //people. freebsd. org/~will/zfs/