Managing the Unimaginable A Practical Approach to Petabyte

Managing the Unimaginable: A Practical Approach to Petabyte Data Storage Randy Cochran, Infrastructure Architect, IBM Corporation, hcochran@us. ibm. com TOD -1366 - Information on Demand Infrastructure

Data Storage is Getting Out-of-Hand Are storage demands starting to overpowering you? 1

Most Research Firms Agree § “It is projected that just four years from now, the world’s information base will be doubling in size every 11 hours. ” (“The toxic terabyte; How data-dumping threatens business efficiency”, Paul Coles, Tony Cox, Chris Mackey, and Simon Richardson, IBM Global Technology Services white paper, July 2006) § “Our two-year terabyte CAGR of 52% is 3 ppt (percentage points) below rolling four quarter results of 55%. ” ("Enterprise Hardware: 2007 -08 storage forecast & views from CIOs", Richard Farmer and Neal Austria, Merrill Lynch Industry Overview, 03 January 2007) § “With a 2006– 2011 CAGR nearing 60%, there is no lack in demand for storage…” ("Worldwide Disk Storage Systems 2007– 2011 Forecast: Mature, But Still Growing and Changing", Research Report # IDC 206662, Natalya Yezhkova, Electronics. ca Publications, May 2007) § “According to The. Info. Pro…. . the average installed capacity in Fortune 1000 organizations has jumped from 198 TB in early 2005 to 680 TB in October 2006. …. . TIP found that capacity is doubling every 10 months. ” (Info. Stor Magazine, Kevin Komiega, October 19, 2006) 2

What’s Driving Petabyte Level Storage? The “Perfect Storm” General Increase in demand New digital data technologies More regulatory requirements Better protection from litigation Proliferation of Sophisticated applications Disaster Recovery plans Declining storage media costs A desire for greater storage efficiency Storage technical skills scarcity A growing understanding of retained data’s business value According to IDC, between 2006 and 2010 information added annually to the digital universe will increase more than six fold from 161 to 988 exabytes. 3

Just How Big is a Petabyte? Petabyte storage had been around for years – online Petabyte storage has not. “Ninety-two percent of new information is stored on magnetic media, primarily hard disks. ” “How Much Information 2003”, UC Berkeley's School of Information Management and Systems 4

How Big is That in Human Terms? According to Britannica. com the U. S. Library of Congress contains approximately 18 million books, 2. 5 million recordings, 12 million photographs, 4. 5 million maps, and more than 54 million manuscripts. 5

Why is Petabyte Storage a Challenge? § Areas Impacted by Petabyte Storage: • • • Content and File Management Application & Database Characteristics Storage Management Architectural Design Strategy Performance and Capacity SAN Fabric Design Backup and Recovery Methods Security System Complexity Compliance with Regulatory Requirements Operational Policies and Processes Maintenance Requirements 6

Content and File Management 7

Management Starts With Data Classification § Data Classification Assumptions • Not all data is created equal • The business value of data changes over time • Performance can be improved by re-allocating data to an optimized storage configuration • The value of most business data is not fixed; it is expected to change over time § Understanding the business value of data is a crucial in designing an effective data management strategy Which data has a greater value to the business - a client’s purchase record, or a memo about last year’s phone system upgrade? 8

Data Classification Example There are no universally accepted standard definitions for Tier Levels. 9

Control Your File Content § Implement file aging § Set data retention periods § Eliminate low value data • • • Clean out old backup files Eliminate outdated information Deploy de-duplication technology Reduce storage of low value data Locate and purge corrupt files § Crack down on unauthorized storage usage § Periodically review log files and archive or delete obsolete information 10

Application and Database Characteristics 11

Know Your Application and Database Needs § Know your applications needs • • • User expectations Workload complexity Read or write intensity Sequential files usage IOPS dependence Stripe size optimization Throughput requirements Service prioritization Growth expectations § Don’t allow databases to “lock up” vast amounts of storage 12

Applications Will Drive Storage Requirements § Applications characteristics will drive storage decisions • Value to the business • Number of users • Usage patterns Ø Ø • • • Steady Bursty Cyclical Variable 7 x 24 or 9 x 5 access Domestic or global access Distributed or self-contained High or low security data Architectural constraints § Significant performance gains (or losses) can be achieved by matching requirements to storage characteristics 13

Storage Management 14

Large Storage Systems Must Be Managed § Information Lifecycle Management (ILM) § Hierarchical Storage Management (HSM) § Storage Resource Management (SRM) § Storage Virtualization "Enterprises can achieve better and more targeted utilization of resources by first establishing the value of their information assets and then using storage management software to execute the policies that define how resources are utilized. " Noemi Greyzdorf, research manager, Storage Software, IDC 15

Information Lifecycle Management “(ILM is) the process of managing business data throughout its lifecycle from conception until disposition across different storage media, within the constraints of the business process. ” (courtesy of Veritas Corporation, Nov. 2004) ILM is not a commercial product, but a complete set of products and processes for managing data from its initial inception to its final disposition. 16

Information Lifecycle Management § Information has business value • • It’s value changes over time It ages at different rates It has a finite life-cycle As data ages its performance needs change § Some Information is subject to different security requirements, due to government regulatory or legal enforcements § Outdated information has different disposal criteria § A combination of processes and technologies that determine how information flows through a corporate environment § Encompasses management of information from its creation until it becomes obsolete and is destroyed 17

“Best Practices” for ILM Implementations § § § § § Know exactly where information is stored Be able to retrieve information quickly and efficiently Limit access to only those who need to view data Create policies for managing and maintaining data Do not destroy important documents Avoid keeping multiple copies of the same data Retain information only until it is no longer useful Destroy outdated files on a regular basis Document all processes and keep them up-to-date 18

Hierarchical Storage Management “HSM is a policy-based data storage management system that automatically moves data between highcost and low-cost storage media, without requiring the knowledge or involvement of the user. ” (courtesy of http: //searchstorage. tedchtarget. com) IBM has been involved in providing HSM solutions for over 30 -years and offer a wide variety of products with automated data movement capabilities. 19

File Access Activity Over Time 20

Hierarchical Storage Management 10% 20% 70% Archive § HSM Concepts • • • Only 10%-15% of most data is actively accessed The business value of data changes over time Between 80% and 90% of all stored data is inactive High performance storage (FC disks) are expensive Lower performance media (tape, optical platters, and SATA disk) are comparatively inexpensive 21

Hierarchical Storage Management $$$$ $$ $ § HSM Concepts (cont. ) • Enterprise class storage is not required for all data • Policies can be set to establish the proper frequency for transitioning aging data to less expensive media • HSM allows optimal utilization of expensive disk storage • Low cost, high density disks consume fewer resources • Overall storage system performance may improve 22

IBM Products with HSM Capabilities § General Parallel File System (GPFS) § IBM Content Manager for Multiplatforms § Tivoli Storage Manager HSM for Windows § Tivoli Storage Manager for Space Management (AIX) § SAN File System (SFS) § DFSMShsm (Mainframe) § High Performance Storage System (HPSS) 23

Storage Resource Management “Storage Resource Management (SRM) is the process of optimizing the efficiency and speed with which the available drive space is utilized in a storage area network (SAN). Functions of an SRM program include data storage, data collection, data backup, data recovery, SAN performance analysis, storage virtualization, storage provisioning, forecasting of future needs, maintenance of activity logs, user authentication, protection from hackers and worms, and management of network expansion. An SRM solution may be offered as a stand-alone product, or as part of an integrated program suite. ” (Definition Courtesy of http: //searchstorage. techtarget. com) IBM’s primary tool for Storage Resource Management is their Total. Storage Productivity Center suite of tools for disk, data, fabric, and replication. 24

Storage Resource Management Functions 25

Storage Virtualization § Virtualization “The act of integrating one or more (back end) services or functions with additional (front end) functionality for the purpose of providing useful abstractions. Typically virtualization hides some of the back end complexity, or adds or integrates new functionality with existing back end services. Virtualization can be nested or applied to multiple layers of a system. ” (Definition Courtesy of http: //www. snia. org/education/dictionary) Virtualization allows most of the complexity of a storage infrastructure to be hidden from the user. 26

Virtualization Makes Storage One Large Pool § Virtualization Characteristics • Makes storage configuration details invisible to the user • Improves overall manageability of the system • Aggregates isolated storage “islands” into a unified view • Facilitates greater flexibility and scalability • Optimizes utilization of storage capacity • Provides the ability to move data on-the-fly • Improves storage subsystems flexibility • Allows rapid re-allocation of storage resources • Improves performance by providing another layer of caching • May provide additional functionality for the SAN 27

Architectural Design Strategy 28

Key Architectural Design Considerations § § § Resource Consumption Storage Economics RAID Allocation Performance Objectives Other Design Issues The integrity of the architectural design will determine the overall performance, stability, economic efficiency, manageability and future scalability of the system. 29

Power Consumption vs. Storage Capacity These disks all have very similar power consumption requirements, even though the largest one features 28 times the capacity of the smaller one. In addition, each disk will require approximately 0. 4 -0. 6 watts of electrical power to cool each BTU of heat produced. ** National retail price of electricity per Kw. H from “Power, Cooling, Space Efficient Storage”, page 2, ESG white paper, Enterprise Strategy Group, July. 2007. 30

Comparing Storage Subsystem Power Costs Significant power savings may be realized by redistributing data to the appropriate type and size of disk drive. 31

Comparing Storage Subsystem Cooling Costs Additional power savings may be realized from the reduced cooling requirements provided by high capacity, lower wattage disk drives. 32

Comparing Storage Floor-Space Cost The DS 4800 and DS 4200 storage subsystems include the required number of disk expansion trays mounted in standard equipment racks. 33

How Do the Costs Add Up? Tiered Storage Approach Traditional Approach Everything on DS 8300 s DS 8300 DS 4800 DS 4200 s with SATA Disk Savings: $614, 935 / yr. 34

A Look at Older Disk Subsystem Efficiency Storing 100 TB of data on more modern storage subsystems results in 50% less power consumption, a 53% reduction in BTUs per hr. , and a reduction in required floor space of 38%. In addition, a DS 8300 system has over 7 x the throughput of the ESS 800. 35

Why is Tiered Storage Important? § Maps data’s business value to disk characteristics § Places data on storage appropriate to its usage § Incorporates lower cost disks § Reduces resource usage (power, cooling, etc. ) § Matches user access needs to storage characteristics § Capitalizes on higher capacity disk drive technology § Increases overall performance of the system 36

A Typical Tiered Storage Architecture DS 8300 DS 4800 DS 4200 s with SATA Disk TS 3500 Tape Library Business Critical Business Important Business Standard High Performance / Very High Availability Good Performance / High Availability Average Performance / Standard Availability Normally a tiered storage strategy is based on data’s business value. Reference / Historical Near-line or Off-line 37

Choosing the Right Controller Frame 38

Choosing the Right Disk Characteristics 39

Comparing Disk Drive Attributes 40

The Cost Impact of Adding Disk Trays Note: Calculations based on 146 GB, 10 K RPM Drives 41

Tiered Storage Design Pros and Cons § Advantages • • • Lower initial purchase price Higher capacity per square foot Reduced power consumption Decreased requirement for cooling Increased equipment flexibility Potentially a higher performance solution § Disadvantages • Inherently a more complex architecture • Greater up-front effort to design and implement • Requires advanced storage design skills and knowledge 42

RAID Selection Decision Drivers § Application or Database characteristics • Read/write mix • Dependency on IOPS § RAID Performance characteristics • • § § Appropriate RAID level Number of disks per array Stripe size Available bandwidth Configuration rules and recommendations Loss from data parity and hot sparing Disk failure probability RAID parity rebuild times 43

Loss from Mirroring, Striping, and Sparing RAID 10 = Mirror plus Stripe RAID 1 = Mirror Only 44

Loss from RAID 5 Parity and Sparing Note: The second tray has one 2+P array to allow for one spare drive per two trays. Note: The second tray has one 6+P array to allow for one spare drive per two trays. Note: Each tray has one spare drive per tray. 45

Other Architectural Considerations § Compatibility § High availability § Architectural robustness § Flexibility and scalability § Stability of the technology § Vendor’s financial standing § Well defined product line roadmap § Support for industry standards 46

Performance and Throughput 47

Storage Subsystem Performance Drivers § Business objectives and user expectations § Applications and database characteristics § Server characteristics § SAN fabric characteristics § Storage controller characteristics § Caching characteristics § Configuration characteristics § Disk latency characteristics "We can't solve problems by using the same kind of thinking we used when we created them. " Albert Einstein 48

Storage Performance Enhancers § Data Mover – Reassigning data transfer tasks to a specialized “engine” reduces the workload on the host processing system. § Search Engines – Systems dedicated to executing searches in vast amounts of stored data to satisfy specific requests. § Directory Services – Stores and organizes information about resources and data objects. § High Speed Interconnections – Dedicated “behind the scenes” networks dedicated to the transfer of large amounts of data. § Autonomic Computing – must have an ability to reconfigure itself under varying and possibly unpredictable conditions. 49

Other Storage Performance Tips 50

SAN Fabric Network 51

SAN Fabric Overview § SAN Fabric is the interconnecting structure between associated servers and storage devices § Proper fabric design will directly impact on: • • • Performance Availability Equipment cost Manageability Maintainability § Communications protocol can be either Fibre Channel, Ethernet, or a combination of both § Ability of the fabric to scale is critical § Monitoring of SAN fabric traffic is a necessity 52

Designing a High Performance Fabric § Select an optimal SAN fabric topology • • Mesh Waterfall Core / Edge Fat tree Butterfly Torus Hypercube Hybrid § Ensure the design is driven by business application requirements § Keep it as simple as possible for manageability 53

Common SAN Fabric Examples 54

SAN Fabric Design Considerations § SAN Fabric design issues: • • • • Throughput requirements Potential bottlenecks Port speed / port count Port subscription rate Maximum hop count Redundancy for High Availability Modularity (flexibility/scalability) Future upgradeability Complexity vs. overall manageability Isolation vs. unification Wide Area Network interconnections Power consumption and footprint Component cost 55

Backup and Recovery 56

Backup and Recovery Challenges § Size of the backup window § Ability to recover files § The time to recover files § Integrity of the data backups § Required frequency of backups § Stored data retention period • Functional • Legal § Available bandwidth for backup resources § Media deterioration over time § Technological obsolescence 57

The Traditional Storage Backup Approach 1. 0 PB of Storage TS 3500 Tape Library (192) of the newest LTO 4 tape drives running at a maximum native transfer rate of 120 MB/sec. would need at least 13 -hours to back up 1. 0 PB of data. 58

Fast Tape Drives can Saturate the Network One LTO-4 tape drive run in native mode is 20% faster than the effective usable bandwidth of Gigabit Ethernet! Four LTO-4 tape drives run at a 2: 1 compressed mode will take most of the usable bandwidth of 10 Gbps Ethernet! 59

Large Systems Backup Approaches § Point-in-Time Copies and Replication • Snapshots Ø Ø Ø Most popular method of large storage backup Snapshots create an exact copy of the source data Once a bitmap is created, storage access can resume While copying, new data is written to both source and target Requires minimal downtime for production systems • Replication Ø Replication creates a mirror image of data over distance Ø May be synchronous (consistent) or asynchronous (lagging) Ø Synchronous is distance-limited, asynchronous is not 60

Point-in-Time Copy 61

Data Replication Structures 62

Other Large Systems Backup Approaches § Object-based Backups • • Backs up only new blocks that have changed Copies only files it has never seen before Inserts pointers if file exists somewhere else Provides instant recoveries by presenting a mountable volume § Delta-Block Incremental Backups • • Evaluates changed data by breaking a file down into discrete blocks Block-for-block comparison of a modified file with an existing file When a difference is detected it extracts a copy of that block only Usually copies a number of blocks, but not the entire file § Continuous Data Protection (CDP) • • Copies blocks to the backup system as soon as they change Stores data in a log that allows recovery from any point in time Performs fast recoveries by restoring only blocks that have changed Provides instant recoveries by presenting a mountable volume 63

Security Requirements 64

Impact of Petabyte Storage on Security § Traditional distributed access control techniques are designed for smaller systems with general or random workloads § Petabyte storage may service tens of thousands of clients and hundreds of storage devices § Storage design must be capable of supporting I/O patterns that are highly parallel and very bursty by nature § Security solutions must be kept highly scalable to keep up with storage growth patterns 65

Impact of Petabyte Storage on Security (Cont. ) § Authentication and authorization requirements can dramatically impact server performance § Performance could be further reduced if data is encrypted § Traditional security protocols perform poorly because they do not scale well § The number of security operations is closely tied to the number of devices and requests 66

Regulatory Compliance 67

The Challenge Of Regulatory Compliance § What’s driving storage regulatory legislation? • Corporate fraud and illegal practices • Increased emphasis on the security of personal data • The threat of terrorist activities • The global impact of the Internet • Increased reliance on stored data for defense against litigation • Increased business dependence on electronic communications (e-mail, digital voice messaging, instant messaging, Vo. IP, etc. ) 68

Regulatory Requirements Continue to Grow § According to the Storage Networking Industry Association (SNIA) there are over 20, 000 regulations worldwide addressing the storage of data § The number of government regulatory requirements increases every year • There is little chance this upward trend will reverse itself in the future • Regulatory guidelines do not dictate how you should maintain your data, only what the expected outcome should be. § If you do business overseas, you must also be aware of the applicable foreign regulatory requirements 69

Common Regulatory Compliance Goals § Most regulatory requirements are based upon: • Security: Maintain data in a secure environment • Efficiency: Rapid location and retrieval of data • Legibility: Recovered documents must be in a readable format that is clear and concise • Authenticity: The recovered document must be verifiable as the original • Validation: Documentation process must be available for review by a neutral third party Regulatory compliance becomes more challenging as storage subsystems grow in size and complexity. 70

Regulatory Legislation Examples § § § Sarbanes-Oxley HIPAA USA Patriot Act Gramm-Leach-Bliley Act FRCP CFR 240 17 a(f) NASD 3010 and 3110 21 CFR Part 11 (FDA) DOD 5015. 2 California Senate Bill 1386 Florida Sunshine Law § § § PCI ISO 17799 CFR Title 18 (FERC) E-SIGN EU Directive 95/46/EC Basel II NARA GRS 20 CFR Title 47, Part 42 NASD 2711/NYSE Rule 472 JCAHO FPC 65 COOP compliance 71

Maintenance Requirements 72

Disk Drive Reliability Misconceptions § Actual disk failure rate is usually higher than published – Vendors indicate a. 58% -. 88% failure rate – Actual field usage suggests a 1. 5% - 3. 5% (or greater) failure rate § Field studies show no appreciable difference in reliability between SCSI, FC, and SATA drives § Heat and high duty cycles do not appear to have as detrimental of an effect on disk life as once thought 73

Disk Drive Reliability Misconceptions (Cont. ) § Infant Mortality doesn’t appear to be a significant issue for newly installed disks § Disks exhibit a fairly linear rate of failure over time, which is contrary to the standard “bathtub” model § Self-Monitoring, Analysis and Reporting Technology (S. M. A. R. T. ) diagnostics appear to predict only about 50% of all disk failures 74

Disk Failures Are an Expected Occurrence Using the disk count from our previous Traditional vs. Tiered models, it’s easy to see that disk failures will occur on a regular basis. 75

Other Considerations 76

Other Issues to Consider § § § § Design for minimal human intervention Maintain extensive monitoring of the environment Exercise control over information propagation Architect for maximum performance and throughput Ensure robustness and high availability Configure for scalability and flexibility Develop well defined SLA objectives Implement a structured support operation 77

Emerging Technologies 78

Emerging Technologies to Watch § Thin Provisioning § Data de-duplication § SAS Interface § Infini. Band § NPIV (N_Port ID Virtualization) § Large-platter storage technologies § 2. 5” disk technologies § Solid state disk drives § Virtualized file systems (i. e. - ZFS, SOFS) § Grid Storage 79

Summary 80

Some Parting Thoughts § Online multi-petabyte storage is a reality § Data will double every two to three years § Storage media cost-per-GB will continue to decline § Storage Operational management is a growing issue § Governmental regulations will increase over time § New technologies will demand additional storage § Experienced storage designers and administrators will grow increasingly harder to find § Scarce data center resources (bandwidth, floor space, power, cooling, etc. ) will become more expensive § A carefully designed architecture is the key to efficient storage operations 81