DADA Dynamic Allocation of Disk Area Jayaram Bobba

DADA – Dynamic Allocation of Disk Area Jayaram Bobba Vivek Shrivastava Department of Computer Sciences, University of Wisconsin Madison

Outline l Introduction l Existing Framework Implementation Issues Results Conclusions l l Department of Computer Sciences, University of Wisconsin Madison

Problem l Disk Volumes l l l Allocate physical space on creation Multiple Disk Volumes on a physical disk Unused space Disk space not a premium What If? Department of Computer Sciences, University of Wisconsin Madison

Motivation l Virtual Machine Environment l l l Storage Area Networks l l Multiple OS may create pressure on disk space Access patterns may warrant dynamic relocation of disk blocks Storage area is a bottleneck resource Dynamic Physical Allocation (LFS style) l Group writes from various volumes into a single physical write. Department of Computer Sciences, University of Wisconsin Madison

Related Work l HP Auto. RAID l l Dynamically change redundancy levels Hot Data – Mirrored Cold Data – RAID 5 Dynamic migration of data Department of Computer Sciences, University of Wisconsin Madison

Outline l Introduction l Existing Framework l Implementation Issues Results Conclusions l l l Department of Computer Sciences, University of Wisconsin Madison

Existing Framework l Logical Volume Manager (LVM) l l l User space tool that enables creation of logical volumes of a logical partition. Supports resizing of volumes. Device Mapper (DM) l Kernel driver that provides a level of indirection in address translation Department of Computer Sciences, University of Wisconsin Madison

LVM design l l l Physical Volume (PV) Volume Group (VG) Logical Volume (LV) Physical Extent (PE) Logical Extent (LE) Department of Computer Sciences, University of Wisconsin Madison

LVM design Physical Device Department of Computer Sciences, University of Wisconsin Madison

LVM design PV PE Department of Computer Sciences, University of Wisconsin Madison

LVM design LV 1 PV PE Department of Computer Sciences, University of Wisconsin Madison

LVM design LE LV 1 LV 2 PV PE Department of Computer Sciences, University of Wisconsin Madison

LVM design LE VG LV 1 LV 2 PV PE Department of Computer Sciences, University of Wisconsin Madison

LVM – DM interaction LVM 2 libdevmapper Userspace Kernel Device Mapper Department of Computer Sciences, University of Wisconsin Madison

Outline l Introduction Existing Framework l Implementation l Issues Results Conclusions l l l Department of Computer Sciences, University of Wisconsin Madison

Modifications l A trap mechanism from kernel driver to userlevel LVM code. l Support for multiple segments in LV l Support for virtual mappings in driver Department of Computer Sciences, University of Wisconsin Madison

Initial State LVM daemon LVM User space Kernel space Device Mapper Unallocated Disk Space Department of Computer Sciences, University of Wisconsin Madison

Volume Group Creation LVM daemon LVM User space Kernel space Device Mapper PE (4 MB) Unallocated Disk Space 25 Extents Department of Computer Sciences, University of Wisconsin Madison

Current Implementation LVM daemon Create LV LVM User A 32 MB User space Kernel space Device Mapper Free Disk Space Department of Computer Sciences, University of Wisconsin Madison

Current Implementation LVM daemon LVM User A Map 0 -7 extents for user A User space Kernel space Device Mapper Free Disk Space Department of Computer Sciences, University of Wisconsin Madison

Current Implementation LVM daemon User A LVM Create LV 16 MB User space Kernel space Device Mapper Free Disk Space Department of Computer Sciences, University of Wisconsin Madison User B

Current Implementation LVM daemon User A LVM Create LV 16 MB map 8 -11 extents for B User space Kernel space Device Mapper Free Disk Space Department of Computer Sciences, University of Wisconsin Madison User B

Current Implementation LVM daemon User A LVM User B map 8 -11 extents for B User space Kernel space Device Mapper Free Disk Space Department of Computer Sciences, University of Wisconsin Madison

Current Implementation LVM daemon User A LVM Read/write User space User B Read/write Kernel space Device Mapper Free Disk Space Department of Computer Sciences, University of Wisconsin Madison

Volume Group Creation LVM daemon LVM User space Kernel space Device Mapper PE (4 MB) Unallocated Disk Space 25 Extents Department of Computer Sciences, University of Wisconsin Madison

Modified Implementation LVM daemon Create LV 1 LVM User A 32 MB User space Kernel space Device Mapper Free Disk Space Department of Computer Sciences, University of Wisconsin Madison

Modified Implementation LVM daemon LVM User A Virtually map 0 -7 extents for user A User space Kernel space Device Mapper Free Disk Space Department of Computer Sciences, University of Wisconsin Madison {0 -7 error}

Modified Implementation LVM daemon User A LVM Create LV 16 MB User space Kernel space Device Mapper Free Disk Space Department of Computer Sciences, University of Wisconsin Madison {LV A 0 -7 error}

Modified Implementation LVM daemon User A LVM Create LV 2 16 MB User B Virtually map 0 -3 extents for B User space Kernel space Device Mapper {LV A 0 -7 error} {LV B 0 -3 error} Free Disk Space Department of Computer Sciences, University of Wisconsin Madison

Modified Implementation LVM daemon User A LVM Write 6 MB User space Kernel space Device Mapper {LV A 0 -7 error} {LV B 0 -3 error} Free Disk Space Department of Computer Sciences, University of Wisconsin Madison

Modified Implementation LVM daemon User A LVM Write 6 MB trap User space User B Kernel space Device Mapper {LV A 0 -7 error} {LV B 0 -3 error} Free Disk Space Department of Computer Sciences, University of Wisconsin Madison

Modified Implementation LVM daemon On Demand allocation of disk area LVM Write 6 MB User A User B trap User space Kernel space Device Mapper {LV A 0 -7 error} {LV B 0 -3 error} Free Disk Space Department of Computer Sciences, University of Wisconsin Madison

Modified Implementation LVM daemon On Demand allocation of disk area LVM Write 6 MB Allocate 0 -1 extents (8 MB) LV A User B User space Kernel space Device Mapper {LV A 0 -7 error} {LV B 0 -3 error} Free Disk Space Department of Computer Sciences, University of Wisconsin Madison

Modified Implementation LVM daemon On Demand allocation of disk area LVM User A Write 6 MB User B Return User space Kernel space Device Mapper {LV A 0 -1 linear, 2 -7 error} {LV B 0 -3 error} Free Disk Space Department of Computer Sciences, University of Wisconsin Madison

Modified Implementation LVM daemon User A LVM Write User B 10 MB User space Kernel space Device Mapper {LV A 0 -1 linear, 2 -7 error} {LV B 0 -3 error} Free Disk Space Department of Computer Sciences, University of Wisconsin Madison

Modified Implementation LVM daemon User A LVM Write 10 MB trap User space User B Kernel space Device Mapper {LV A 0 -1 linear, 2 -7 error} {LV B 0 -3 error} Free Disk Space Department of Computer Sciences, University of Wisconsin Madison

Modified Implementation LVM daemon On Demand allocation of disk area User A LVM Write 10 MB trap User space User B Kernel space Device Mapper {LV A 0 -1 linear, 2 -7 error} {LV B 0 -3 error} Free Disk Space Department of Computer Sciences, University of Wisconsin Madison

Modified Implementation LVM daemon On Demand allocation of disk area LVM LV B Allocate 8 -11 extents (12 MB) User A User B Write 10 MB User space Kernel space Device Mapper {LV A 0 -1 linear, 2 -7 error} {LV B 0 -2 linear, 3 error} Free Disk Space Department of Computer Sciences, University of Wisconsin Madison

Outline l Introduction Existing Framework Implementation l Issues l Results Conclusions l l l Department of Computer Sciences, University of Wisconsin Madison

Issues LVM implementation not efficient LVM 2 libdevmapper Userspace Kernel Device Mapper Department of Computer Sciences, University of Wisconsin Madison

Issues LVM implementation not efficient LVM 2 libdevmapper Userspace Costly kerneluser space crossing Kernel Device Mapper Department of Computer Sciences, University of Wisconsin Madison

Issues LVM implementation not efficient LVM 2 libdevmapper Userspace Device Mapper Kernel Department of Computer Sciences, University of Wisconsin Madison Normal function call

More Issues l On a trap l How much to allocate? l l l Where to allocate? l l l On demand Pre-allocate Temporal locality Spatial locality Free unused space Department of Computer Sciences, University of Wisconsin Madison

How much? l On-demand Allocate only as many blocks as needed l l Pre-allocate physical space for blocks likely to be accessed in the future. l l l How much? Where? Department of Computer Sciences, University of Wisconsin Madison

How much? l l Tradeoff – Performance vs. Disk Space Depends on the workload l l l I/O on critical path Asynchronous I/O Multithreaded Applications Department of Computer Sciences, University of Wisconsin Madison

Pre-Allocation Think: Filesystem Block Allocation l Dumb Allocation l Stride Allocation l Multi-Strided Allocation Department of Computer Sciences, University of Wisconsin Madison

Allocation Strategy LVM daemon On Demand allocation of disk area LVM Allocate 2 extents (8 MB) User A Write trap User space Kernel space Device Mapper Free Disk Space Department of Computer Sciences, University of Wisconsin Madison {0 -1 linear}{2 -7 error}

Dumb Allocation LVM daemon On Demand allocation of disk area LVM Allocate 2 extents (8 MB) User A Write 6 MB trap User space Kernel space Device Mapper Free Disk Space Department of Computer Sciences, University of Wisconsin Madison {0 -1 linear}{2 -7 error}

Strided Allocation (Motivation) Department of Computer Sciences, University of Wisconsin Madison

Stride Allocation LVM daemon On Demand allocation of disk area LVM Allocate 4 extents (16 MB) User A Write trap User space Kernel space Device Mapper Free Disk Space {0 -1 linear}{2 -7 error} Pre. Allocated Extent Department of Computer Sciences, University of Wisconsin Madison

Outline l Introduction Existing Framework Implementation Issues l Results l Conclusions l l l Department of Computer Sciences, University of Wisconsin Madison

Results l Micro. Benchmarks l l l mkfs Asynchronous io (aio) based workload Trace based analysis l Trace collected by HP Research Labs used in the D-GRAID paper Department of Computer Sciences, University of Wisconsin Madison

Micro Benchmark : mkfs Does the # traps taken affect performance ? Department of Computer Sciences, University of Wisconsin Madison

Micro Benchmark : Asynchronous I/O Does the # traps taken affect performance ? Department of Computer Sciences, University of Wisconsin Madison

Trace Based Analysis Dumb Allocation – What is the tradeoff in choosing extent size? Department of Computer Sciences, University of Wisconsin Madison

Trace Based Analysis Strided Allocation – What is the tradeoff in choosing extent size? Department of Computer Sciences, University of Wisconsin Madison

Trace Based Analysis : Dumb vs. Stride (ext 2) Department of Computer Sciences, University of Wisconsin Madison

Trace Based Analysis : Dumb vs. Stride (ext 3) Department of Computer Sciences, University of Wisconsin Madison

Outline l Introduction Existing Framework Implementation Issues Results l Conclusions l l Department of Computer Sciences, University of Wisconsin Madison

Conclusions l l Tradeoff between performance penalty and space Useful in environments where disk space is the bottleneck Techniques like Pre Allocation can be used to reduce the performance penalty. For practical implementation, LVM must be placed in kernel address space Department of Computer Sciences, University of Wisconsin Madison

Acknowledgments l l Vijayan Prof. Remzi Department of Computer Sciences, University of Wisconsin Madison