ISTORE An Introspective Storage Architecture for Network Service

  • Slides: 15
Download presentation
ISTORE: An Introspective Storage Architecture for Network Service Applications Aaron Brown, David Oppenheimer, Kimberly

ISTORE: An Introspective Storage Architecture for Network Service Applications Aaron Brown, David Oppenheimer, Kimberly Keeton, Randi Thomas, Jim Beck, John Kubiatowicz, and David Patterson http: //iram. cs. berkeley. edu/istore 1999 Winter IRAM Retreat Slide 1

Agenda • • Overview of ISTORE: Motivation and Architecture Hardware Details and Prototype Plans

Agenda • • Overview of ISTORE: Motivation and Architecture Hardware Details and Prototype Plans Software Architecture Discussion and Feedback Slide 2

Motivation • Emerging paradigm of network-based services – today: e-commerce, online database services, search

Motivation • Emerging paradigm of network-based services – today: e-commerce, online database services, search engines, and web servers – tomorrow: above, plus thin-client/PDA infrastructure support • These services have unique demands – vast storage needs (large operational and historical DSS databases) – high access rates – mix of transactional (ACID) and non-transactional accesses – easy incremental scaling of storage and processing * minimal administration burden » easy setup, scaling, maintenance, repair, tuning, . . . » automatic dynamic adaptation to failures, workload changes, . . . Slide 3

Scaleable Storage Appliances • Best architecture to meet these demands is a scaleable self-maintaining

Scaleable Storage Appliances • Best architecture to meet these demands is a scaleable self-maintaining storage appliance – appliance: a single-function device optimized to provide one application or service » consists of software and hardware constructed with that service in mind » interface to WAN/LAN via standard high-level protocols » internal implementation hidden – storage appliance: integrates an application with its data storage in one unit – scaleable appliance: incrementally scaleable with automatic integration of new resources (e. g. , disks) – self-maintaining appliance: runs autonomously, without human intervention, once initially configured • Examples: e-commerce, decision-support DB, PDA proxy, video server, or online voting appliances Slide 4

Today’s Appliances • Some storage appliances exist today: – – – Network Appliance NFS

Today’s Appliances • Some storage appliances exist today: – – – Network Appliance NFS and proxy servers SNAP! Server file server Qube web server Sonic. Wall firewall. . . • But none are truly self-maintaining, and most are not scaleable Slide 5

Why Aren’t There Appliances That Match Our Criteria? 1) Self-maintenance is tricky and application-dependent

Why Aren’t There Appliances That Match Our Criteria? 1) Self-maintenance is tricky and application-dependent * can’t rely on human administrators to configure, monitor, and tune system for its application – failure management » devices must fail fast without interrupting service » failures should not require immediate human intervention – performance management » system must adapt to changes in workload or access patterns – system upgrades and scaling » new hardware resources must be automatically incorporated without interruption of service » new devices should be used immediately to improve performance or repair prior failures Slide 6

More Appliance Challenges 2) Appliance designs are inextricably tied to single applications – each

More Appliance Challenges 2) Appliance designs are inextricably tied to single applications – each service or application requires a custom-designed appliance » new systems software must be written, tested, debugged » often can share hardware, but entire system must be retested » especially difficult to design new self-maintaining appliances – performance and manageability suffer when an appliance is used for other than its intended purpose » e. g. , if workload doesn't quite match appliance design goals » example: using a file-server appliance as the storage subsystem for a database application Slide 7

The ISTORE Solution: An Introspective Meta-Appliance • A meta-appliance is a generic hardware and

The ISTORE Solution: An Introspective Meta-Appliance • A meta-appliance is a generic hardware and software platform that can be specialized into a singleapplication, focused appliance • An introspective system uses hardware and software monitoring to tune itself and to achieve selfmaintenance • The ISTORE meta-appliance is a hardware/software architecture for building scaleable self-maintaining storage appliances Slide 8

ISTORE Hardware Innovations 1) ISTORE uses intelligent hardware – ISTORE is a shared-nothing distributed

ISTORE Hardware Innovations 1) ISTORE uses intelligent hardware – ISTORE is a shared-nothing distributed system of intelligent device bricks interconnected by an intelligent chassis CPU, memory, NI Intelligent Chassis Device Intelligent Device Brick – intelligent device brick: a device (e. g. , a disk) plus a fast embedded CPU, memory, and NI (e. g. , an IRAM) – intelligent chassis: scaleable, redundant, fast network + UPS Slide 9

Benefits of ISTORE’s Hardware Architecture • Decentralized processing (shared-nothing) – system can withstand partial

Benefits of ISTORE’s Hardware Architecture • Decentralized processing (shared-nothing) – system can withstand partial failure • Intelligent devices monitor their own “health, ” test themselves, manage failures, collect applicationspecified performance data, and execute application worker code – provides the foundation for self-maintenance and self-tuning • Plug & play, hot-swappable bricks ease configuration, scaling – hardware is specialized/scaled by selecting an appropriate collection of devices » disks, DRAMs, WAN/LAN interfaces, . . . Slide 10

ISTORE Software Innovations 2) ISTORE’s system software is designed to deliver self-maintenance and allow

ISTORE Software Innovations 2) ISTORE’s system software is designed to deliver self-maintenance and allow for easy specialization – layered mechanism libraries facilitate introspection, selfmaintenance – extensibility via domain-specific languages (DSLs) allows customization of the runtime system with application-specific policies • Benefits of ISTORE’s software architecture – appliance designer inherits functionality of supplied base mechanisms for monitoring and adaptation – easy for designer to customize mechanisms and define adaptation policies by writing domain-specific code Slide 11

ISTORE and IRAM • ISTORE relies on intelligent devices • IRAM is an easy

ISTORE and IRAM • ISTORE relies on intelligent devices • IRAM is an easy way to add intelligence to a device – embedded, low-power CPU meets size and power constraints – integrated DRAM reduces chip count – fast network interface (serial lines) meets connectivity needs • Initial ISTORE prototype won’t use IRAM – will use collection of commodity components that approximate IRAM functionality, not size/power – but results will be applicable to IRAM-based systems » success with ISTORE will encourage integration of IRAM-like processors into standard devices Slide 12

Agenda • • Overview of ISTORE: Motivation and Architecture Hardware Details and Prototype Plans

Agenda • • Overview of ISTORE: Motivation and Architecture Hardware Details and Prototype Plans Software Runtime Architecture Discussion and Feedback Slide 13

Backup Slides Slide 14

Backup Slides Slide 14

Related Work • ISTORE extends several recent research efforts – Intelligent/Active Disks, NASD (UCB,

Related Work • ISTORE extends several recent research efforts – Intelligent/Active Disks, NASD (UCB, UCSB, CMU) » used as a component in ISTORE; ISTORE adds a global runtime system, introspection, and self-maintenance – network service appliances (Net. App, Snap!, Qube, Swarm, . . . ) » ISTORE adds the idea of a meta-appliance & self-maintenance – – – self-maintaining systems (Sun Project Store. X, Tandem, . . . ) extensible operating systems (SPIN, Exokernel, VINO, Scout, . . . ) domain-specific languages (OGI Microlanguages, Prolac, VHDL, . . . ) adaptive systems (HP Auto. RAID, MS Auto. Admin, MS Millennium) plug-and-play system construction (Sun Jini, PC Plug&Play, . . . ) • ISTORE leverages Berkeley’s experience with reliable, large, distributed storage systems and applications – RAID, Tertiary Disk, XFS, Sprite, NOW, Post. Gres/Gi. ST, . . . Slide 15