The Horus and Ensemble Projects Accomplishments and Limitations

The Horus and Ensemble Projects: Accomplishments and Limitations Ken Birman, Robert Constable, Mark Hayden, Jason Hickey, Christoph Kreitz, Robbert van Renesse, Ohul Rodeh, Werner Vogels Department of Computer Science Cornell University

Reliable Distributed Computing: Increasingly urgent, still unsolved w Distributed computing has swept the world n n n Impact has become revolutionary Vast wave of applications migrating to networks Already as critical a national infrastructure as water, electricity, or telephones w Yet distributed systems remain n Unreliable, prone to inexplicable outages Insecure, easily attacked Difficult (and costly) to program, bug-prone January, 2000 Cornell Presentation at DISCEX

A National Imperative w Potential for catastrophe cited by n n n DARPA ISAT study commissioned by Anita Jones (1985, I briefed the findings, became basis for refocusing of much of ITO under Howard Frank) PCCIP report, PTAC NAS study of trust in cyberspace w Need a quantum improvement in technologies, packaged in easily used, practical forms January, 2000 Cornell Presentation at DISCEX

Quick Timeline w Cornell has developed 3 generations of reliable group communication technology n n n Isis Toolkit: 1987 -1990 Horus System: 1990 -1994 Ensemble System: 1994 -1999 w Today engaged in a new effort reflecting a major shift in emphasis n Spinglass Project: 1999 - January, 2000 Cornell Presentation at DISCEX

Questions to consider w Have these projects been successful? n n n What did we do? How can impact be quantified? What limitations did we encounter? w How is industry responding? w What next? January, 2000 Cornell Presentation at DISCEX

Timeline Isis Horus • • January, 2000 Ensemble Introduced reliability into group computing Virtual synchrony execution model Elaborate, monolithic, but adequate speed Many transition successes • New York, Swiss Stock Exchanges • French Air Traffic Control console system • Southwestern Bell Telephone network mgt. • Hiper-D (next generation AEGIS) Cornell Presentation at DISCEX

Virtual Synchrony Model G 0={p, q} G 1={p, q, r, s} p G 2={q, r, s} G 3={q, r, s, t} crash q r s t r, s request to join r, s added; state xfer p fails t requests to join t added, state xfer

Why a “model”? w Models can be reduced to theory – we can prove the properties of the model, and can decide if a protocol achieves it w Enables rigorous application-level reasoning w Otherwise, the application must guess at possible misbehaviors and somehow overcome them January, 2000 Cornell Presentation at DISCEX

Virtual Synchrony w Became widely accepted – basis of literally dozens of research systems and products worldwide w Seems to be the only way to solve problems based on replication w Very fast in small systems but faces scaling limitations in large ones January, 2000 Cornell Presentation at DISCEX

How Do We Use The Model? w Makes it easy to reason about the state of a distributed computation w Allows us to replicate data or computation for fault-tolerance (or because multiple users share same data) w Can also replicate security keys, do loadbalancing, synchronization… January, 2000 Cornell Presentation at DISCEX

French ATC system (simplified) Onboard Radar X. 500 Directory Controllers January, 2000 Air Traffic Database (flight plans, etc) Cornell Presentation at DISCEX

A center contains. . . w Perhaps 50 “teams” of 3 -5 controllers each w Each team supported by workstation cluster w Cluster-style database server has flight plan information w Radar server distributes real-time updates w Connections to other control centers (40 or so in all of Europe, for example) January, 2000 Cornell Presentation at DISCEX

Process groups arise here: w Cluster of servers running critical database server programs w Cluster of controller workstations support ATC by teams of controllers w Radar must send updates to the relevant group of control consoles w Flight plan updates must be distributed to the “downstream” control centers January, 2000 Cornell Presentation at DISCEX

Role For Virtual Synchrony? w French government knows requirements for safety in ATC application w With our model, we can reduce their need to a formal set of statements w This lets us establish that our solution will really be safe in their setting w Contrast with usual ad-hoc methodologies. . . January, 2000 Cornell Presentation at DISCEX

More Isis Users w New York Stock Exchange w Swiss Stock Exchange w Many VLSI Fabrication Facilities w Many telephony control applications w Hiper-D – an AEGIS rebuild prototype w Various NSA and military applications w Architecture contributed to SC-21/DD-21 January, 2000 Cornell Presentation at DISCEX

Timeline Isis Horus Ensemble • Simpler, faster group communication system • Uses a modular layered architecture. Layers are “compiled, ” headers compressed for speed • Supports dynamic adaptation and real-time apps • Partitionable version of virtual synchrony • Transitioned primarily through Stratus Computer § Phoenix product, for telecommunications January, 2000 Cornell Presentation at DISCEX

Layered Microprotocols in Horus Interface to Horus is extremely flexible Horus manages group abstraction group semantics (membership, actions, events) defined by stack of modules Ensemble stacks plug-and-play modules to give design flexibility to developer vsync filter encrypt ftol sign

Layered Microprotocols in Horus Interface to Horus is extremely flexible Horus manages group abstraction group semantics (membership, actions, events) defined by stack of modules Ensemble stacks plug-and-play modules to give design flexibility to developer vsync filter encrypt ftol sign

Layered Microprotocols in Horus Interface to Horus is extremely flexible Horus manages group abstraction group semantics (membership, actions, events) defined by stack of modules Ensemble stacks plug-and-play modules to give design flexibility to developer filter ftol vsync encrypt sign

Group Members Use Identical Multicast Protocol Stacks ftol vsync encrypt

With Multiple Stacks, Multiple Properties ftol vsync encrypt ftol vsync encrypt

Timeline Isis Horus Ensemble • Horus-like stacking architecture, equally fast • Includes an group-key mechanism for secure group multicast and key management • Uses high level language, can be formally proved, an unexpected and major success • Many early transition successes § DD-21, Quorum via collaboration with BBN § Nortel, STC: commercial users § Discussions with MS (COM+): could be basis of standards. January, 2000 Cornell Presentation at DISCEX

Proving Ensemble Correct w Unlike Isis and Horus, Ensemble is coded in a language with strong semantics (ML) w So we took a spec. of virtual synchrony from MIT’s IOA group (Nancy Lynch) w And are actually able to prove that our code implements the spec. and that the spec captures the virtual synchrony property! January, 2000 Cornell Presentation at DISCEX

Why is this important? w If we use Ensemble to secure keys, our proof is a proof of security of the group keying protocol… w And the proof extends not just to the algorithm but also to the actual code implementing it w These are the largest formal proofs every undertaken! January, 2000 Cornell Presentation at DISCEX

Why is this feasible? w Power of the Nu. PRL system: a fifth generation theorem proving technology w Simplifications gained through modularity: compositional code inspires a style of compositional proof w Ensemble itself is unusually elegant, protocols are spare and clear January, 2000 Cornell Presentation at DISCEX

Other Accomplishments w An automated optimization technology n n n Often, a simple modular protocol becomes complex when optimized for high performance Our approach automates optimization: the basic protocol is only coded once and we work with a single, simple, clear version Optimizer works almost as well as handoptimization and can be invoked at runtime January, 2000 Cornell Presentation at DISCEX

Optimization ftol vsync encrypt Original code is simple but inefficient January, 2000 Optimized is provably the same yet inefficiencies are eliminated Cornell Presentation at DISCEX

Other Accomplishments w Real-Time Fault-Tolerant Clusters n n Problem originated in AEGIS tracking server Need a scalable, fault-tolerant parallel server with rapid real-time guarantees January, 2000 Cornell Presentation at DISCEX

AEGIS Problem Emulate this… With this… Tracking Server 100 ms deadline January, 2000 Tracking Server Tracking Server Tracking Server Tracking Server Cornell Presentation at DISCEX

Other Accomplishments w Real-Time Fault-Tolerant Clusters n n n Problem originated in AEGIS tracking server Need a scalable, fault-tolerant parallel server with rapid real-time guarantees With Horus, we achieved 100 ms response time, even when nodes crash, scalability to 64 nodes or more, load balancing and linear speedups w Our approach emerged as one of the major themes in SC-21, which became DD-21 January, 2000 Cornell Presentation at DISCEX

Other Accomplishments w A flexible, object-oriented toolkit n n n Standardizes the sorts of things programmers do most often Programmers are able to work with high level abstractions rather than being forced to reimplement common tools, like replicated data, each time they are needed Embedding into NT COM architecture January, 2000 Cornell Presentation at DISCEX

Security Architecture w Group key management w Fault-tolerant, partitionable w Currently exploring a very large scale configuration that would permit rapid key refresh and revocation even with millions of users w All provably correct January, 2000 Cornell Presentation at DISCEX

Transition Paths? w Through collaboration with BBN, delivered to DARPA QUOIN effort w Part of DD-21 architecture w Strong interest in industry, good prospects for “major vendor” offerings within a year or two w A good success for Cornell and DARPA January, 2000 Cornell Presentation at DISCEX

What Next? w Continue some work with Ensemble n n Research focuses on proof of replication stack Otherwise, keep it alive, support and extend it Play an active role in transition Assist standards efforts w But shift in focus to a completely new effort n n Emphasize adaptive behavior, extreme scalability, robustness against local disruption Fits “Intrinisically Survivable Systems” initiative January, 2000 Cornell Presentation at DISCEX

Throughput Stability: Achilles Heel of Group Multicast w When scaled to even modest environments, overheads of virtual synchrony become a problem n n One serious challenge involves management of group membership information But multicast throughput also becomes unstable with high data rates, large system size, too. w A problem in every protocol we’ve studied including other “scalable, reliable” protocols January, 2000 Cornell Presentation at DISCEX

Thoughput Scenario Most members are healthy…. … but one is slow January, 2000 Cornell Presentation at DISCEX

Group throughput (healthy members) Virtually synchronous Ensemble multicast protocols 250 200 32 group members 150 64 group members 96 group members 100 50 0 0 0. 1 0. 2 0. 3 0. 4 0. 5 0. 6 Degree of slowdown 0. 7 0. 8 0. 9 Throughput as one member of a multicast group is "perturbed" by forcing it to sleep for varying amounts of time. January, 2000 Cornell Presentation at DISCEX

Bimodal Multicast in Spinglass w A new family of protocols with stable throughput, extremely scalable, fixed and low overhead per process and per message w Gives tunable probabilistic guarantees w Includes a membership protocol and a multicast protocol w Ideal match for small, nomadic devices January, 2000 Cornell Presentation at DISCEX

Throughput with 25% Slow processes 180 160 average throughput 200 140 120 100 80 80 60 60 40 40 20 20 0 0. 1 0. 2 0. 3 0. 4 0. 5 0. 6 0. 7 0. 8 0. 9 0 0. 1 January, 2000 0. 2 0. 3 0. 4 0. 5 0. 6 slowdown Cornell Presentation at DISCEX 0. 7 0. 8 0. 9

Spinglass: Summary of objectives w Radically different approach yields stable, scalable protocols with steady throughput w Small footprint, tunable to match conditions w Completely asynchronous, hence demands new style of application development w But opens the door to a new lightweight reliability technology supporting large autonomous environments that adapt January, 2000 Cornell Presentation at DISCEX

Conclusions w Cornell: leader in reliable distributed computing w High impact on important Do. D problems, such as AEGIS (DD-21), QUOIN, NSA intelligence gathering, many other applications w Demonstrated modular plug-and-play protocols that perform well and can be proved correct w Transition into standard, off the shelf O/S w Spinglass – the next major step forward January, 2000 Cornell Presentation at DISCEX