First magnetic disks the IBM 305 RAMAC 2

First magnetic disks, the IBM 305 RAMAC (2 units shown) introduced in 1956. One platter shown top right. A RAMAC stored 5 million characters on 50 24 -inch diameter platters. Two access arms moved up and down to select a disk, and in/out to select a track. Right: a variety of disk drives: 8”, 5. 25”, 3. 5”, 1. 8” and 1”.

Storage Anselmo Lastra The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

Outline • Magnetic Disks • RAID • Advanced Dependability/Reliability/Availability • I/O Benchmarks, Performance and Dependability • Conclusion 3 The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

Disk Figure of Merit: Areal Density • Bits recorded along a track ♦ Metric is Bits Per Inch (BPI) • Number of tracks per surface ♦ Metric is Tracks Per Inch (TPI) • Disk metric is bit density per unit area ♦ Metric is Bits Per Square Inch: Areal Density = BPI x TPI 4 The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

Historical Perspective • 1956 IBM Ramac — early 1970 s Winchester ♦ For mainframe computers, proprietary interfaces ♦ Steady shrink in form factor: 27 in. to 14 in. • Form factor and capacity drives market more than performance • 1970 s developments ♦ 5. 25 inch floppy disk form factor ♦ Emergence of industry standard disk interfaces • Early 1980 s: PCs and first generation workstations • Mid 1980 s: Client/server computing ♦ Centralized storage on file server ♦ Disk downsizing: 8 inch to 5. 25 ♦ Mass market disk drives become a reality • industry standards: SCSI, IPI, IDE • 5. 25 inch to 3. 5 inch drives for PCs; End of proprietary interfaces • 1990 s: Laptops => 2. 5 inch drives • 2000 s: 1. 8” used in media players (1” microdrive didn’t do as well) 5 The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

Current Disks • Caches to hold recently accessed blocks • Microprocessor and command buffer to enable reordering of accesses 6 The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

Future Disk Size and Performance • Continued advance in capacity (60%/yr) and bandwidth (40%/yr) • Slow improvement in seek, rotation (8%/yr) • Time to read whole disk Year Sequentially Randomly (1 sector/seek) 1990 4 minutes 6 hours 2000 12 minutes 1 week(!) 2006 56 minutes 3 weeks (SCSI) 2006 171 minutes 7 weeks (SATA) • Cost has dropped by 100, 000 since 1983 7 The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

Arrays of Small Disks? • Katz and Patterson asked in 1987: • Can smaller disks be used to close gap in performance between disks and CPUs? Conventional: 4 disk 3. 5” 5. 25” 10” designs Low End 14” High End Disk Array: 1 disk design 3. 5” 8 The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

Advantages of Small Form Factor Disk Drives Low cost/MB High MB/volume High MB/watt Low cost/Actuator Cost and Environmental Efficiencies 9 The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

Replace Small Number of Large Disks with Large Number of Small Disks! (1988 Disks) IBM 3390 K IBM 3. 5" 0061 x 70 Capacity 20 GBytes 320 MBytes 23 GBytes 97 cu. ft. 11 cu. ft. 9 X 0. 1 cu. ft. Volume 3 KW 1 KW 3 X 11 W Power 120 MB/s 8 X 1. 5 MB/s Data Rate 15 MB/s 3900 IOs/s 6 X 55 I/Os/s I/O Rate 600 I/Os/s 250 KHrs ? ? ? Hrs 50 KHrs MTTF $250 K $150 K $2 K Cost Disk Arrays have potential for large data and I/O rates, high MB per cu. ft. , high MB per KW, but what about reliability? 10 The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

Array Reliability • Reliability of N disks = Reliability of 1 Disk ÷ N 50, 000 Hours ÷ 70 disks = 700 hours Disk system MTTF: Drops from 6 years to 1 month! • Arrays (without redundancy) too unreliable to be useful! Hot spares support reconstruction in parallel with access: very high media availability can be achieved 11 The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

Redundant Arrays of (Inexpensive) Disks • Files are "striped" across multiple disks • Redundancy yields high data availability ♦ Availability: service still provided to user, even if some components failed • Disks will still fail • Contents reconstructed from data redundantly stored in the array Capacity penalty to store redundant info Bandwidth penalty to update redundant info 12 The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

RAID 0 • Performance only • No redundancy • Stripe data to get higher bandwidth • Latency not improved 13 The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

Redundant Arrays of Inexpensive Disks RAID 1: Disk Mirroring/Shadowing recovery group • Each disk is fully duplicated onto its “mirror” Very high availability can be achieved • Bandwidth sacrifice on write: Logical write = two physical writes • Reads may be optimized • Most expensive solution: 100% capacity overhead • (RAID 2 not interesting, so skip) 14 The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

Redundant Array of Inexpensive Disks RAID 3: Parity Disk 10010011 11001101 10010011. . . logical record P 1 1 0 1 Striped physical 1 0 records 0 0 P contains sum of 0 1 other disks per stripe 0 1 mod 2 (“parity”) 1 0 If disk fails, subtract 1 1 P from sum of other disks to find missing information 15 1 0 0 0 1 1 0 1 The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

RAID 3 • Sum computed across recovery group to protect against hard disk failures, stored in P disk • Logically, a single high capacity, high transfer rate disk: good for large transfers • Wider arrays reduce capacity costs, but decreases availability • 33% capacity cost for parity if 3 data disks and 1 parity disk 16 The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

Inspiration for RAID 4 • RAID 3 relies on parity disk to discover errors on Read • But every sector has an error detection field • To catch errors on read, rely on error detection field vs. the parity disk • Allows independent reads to different disks simultaneously 17 The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

Redundant Arrays of Inexpensive Disks RAID 4: High I/O Rate Parity Insides of 5 disks Example: small read D 0 & D 5, large write D 12 -D 15 18 D 0 D 1 D 2 D 3 P D 4 D 5 D 6 D 7 P D 8 D 9 D 10 D 11 P D 12 D 13 D 14 D 15 P D 16 D 17 D 18 D 19 P D 20 D 21 D 22 D 23 P . . . Disk Columns. . Increasing Logical Disk Address Stripe The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

Inspiration for RAID 5 • RAID 4 works well for small reads • Small writes (write to one disk): ♦ Option 1: read other data disks, create new sum and write to Parity Disk ♦ Option 2: since P has old sum, compare old data to new data, add the difference to P (2 reads, 2 writes) • Small writes are limited by Parity Disk: Write to D 0, D 5 both also write to P disk 19 D 0 D 1 D 2 D 3 P D 4 D 5 D 6 D 7 P The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

Redundant Arrays of Inexpensive Disks RAID 5: High I/O Rate Interleaved Parity Independent writes possible because of interleaved parity Example: write to D 0, D 5 uses disks 0, 1, 3, 4 20 D 1 D 2 D 3 P D 4 D 5 D 6 P D 7 D 8 D 9 P D 10 D 11 D 12 P D 13 D 14 D 15 P D 16 D 17 D 18 D 19 D 20 D 21 D 22 D 23 P . . Disk Columns. . . Increasing Logical Disk Addresses The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

Downside of Disk Arrays: Cost of Small Writes RAID-5: Small Write Algorithm 1 Logical Write = 2 Physical Reads + 2 Physical Writes D 0' new data D 0 D 1 D 2 D 3 old data (1. Read) P old (2. Read) parity + XOR (3. Write) 21 D 0' D 1 (4. Write) D 2 D 3 P' The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

RAID 6: Recovering from 2 failures • Like the standard RAID schemes, it uses redundant space based on parity calculation per stripe • Idea is that operator may make mistake and swap wrong disk, or 2 nd disk may fail while replacing 1 st • Since it is protecting against a double failure, it adds two check blocks per stripe of data. ♦ If p+1 disks total, p-1 disks have data; assume p=5 22 The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

Summary of RAID Techniques • Disk Mirroring, Shadowing (RAID 1) Each disk is fully duplicated onto its "shadow" Logical write = two physical writes 100% capacity overhead • Parity Data Bandwidth Array (RAID 3) Parity computed horizontally Logically a single high data bw disk • High I/O Rate Parity Array (RAID 5) 1 0 0 1 1 1 0 0 1 1 0 0 1 0 Interleaved parity blocks Independent reads and writes Logical write = 2 reads + 2 writes 23 The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

HW Failures in Real Systems: Tertiary Disks • A cluster of 20 PCs in seven 7 -foot high, 19 -inch wide racks with 368 8. 4 GB, 7200 RPM, 3. 5 -inch IBM disks. The PCs are P 6 -200 MHz with 96 MB of DRAM each. 24 The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

Does Hardware Fail Fast? 4 disks that failed in Tertiary Disk Author says that almost all disk failures began as transient failures. The operator had to decide when to replace. 25 The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

Internet Archive • Good section in text about Internet Archive ♦ In 2006, over a petabyte of disk (1015 bytes) ♦ Growing at 20 terabytes (1012) per month ♦ Now says ~ 3 PB • Each PC was 1 GHz Via, 512 MB, dissipates 80 W • Each node had 4 500 GB drives • 40 nodes/rack ♦ Petabyte takes 12 racks • PC cost $500, each disk $375, 40 -port Ethernet switch $3000 26 The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

Capricorn PS now • • • AMD Athlon 64 (x 2) 4 SATA disks (1 -4 TB) 92 Watts/node 40 nodes/rack So 160 TB/rack • 24 k. W/petabyte ♦ So 576 k. Wh/day, ~17, 000/mo ♦ Ave. U. S. house used 920 k. Wh a month in 2006 ♦ Best housed in KY? 27 The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

Drives Today • Just ordered simple RAID enclosure (just levels 0, 1) ♦ $63 • Two 1 TB SATA drives ♦ $85/ea 28 The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

Summary • Disks: Areal Density now 30%/yr vs. 100%/yr in 2000 s • Components often fail slowly • Real systems: problems in maintenance, operation as well as hardware, software 29 The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL