Scalable Applications and Real Time Response Ashish Motivala

Scalable Applications and Real Time Response Ashish Motivala CS 614 April 17 th 2001

Scalable Applications and Real Time Response n Using Group Communication Technology to Implement a Reliable and Scalable Distributed IN Coprocessor; Roy Friedman and Ken Birman; TINA 1996. n Manageability, availability and performance in Porcupine: a highly scalable, cluster-based mail service; Yasushi Saito, Brian N. Bershad and Henry M. Levy; Proceedings of the 17 th ACM Symposium on Operating Systems Principles , 1999, Pages 1 – 15.

Real-time n Two categories of real-time – When an action needs to be predictably fast. i. e. Critical applications. – When an action must be taken before a time limit passes. n More often than not real-time doesn’t mean “as fast as possible” but means “slow and steady”.

Real problems need real-time n Air Traffic Control, Free Flight – when planes are at various locations. n Medical Monitoring, Remote Tele-surgery – doctors talk about how patients responded after drug was given, or change therapy after some amount of time. n Process control software, Robot actions – a process controller runs factory floors by coordinating machine tools activities.

More real-time problems n Video and multi-media systems – synchronous communication protocols that coordinate video, voice, and other data sources n Telecommunications systems – guarantee real-time response despite failures, for example when switching telephone calls

Predictability n If this is our goal… – Any well-behaved mechanism may be adequate – But we should be careful about uncommon disruptive cases • For example, cost of failure handling is often overlooked • Risk is that an infrequent scenario will be very costly when it occurs

Predictability: Examples n Probabilistic multicast protocol – Very predictable if our desired latencies are larger than the expected convergence – Much less so if we seek latencies that bring us close to the expected latency of the protocol itself

Back to the paper n Telephone networks need a mixture of properties – Real-time response – High performance – Stable behavior even when failures and recoveries occur n Can we use our tools to solve such a problem?

Role of coprocessor n A simple database – Switch does a query • How should I route a call to 1800 -327 -2777 from 607 -266 -8141? • Reply: use output line 6 – Time limit of 100 ms on transaction Call ID, call conferencing, automatic transferring, voice menus, etc n Update database n

IN coprocessor SS 7 switch

IN coprocessor SS 7 switch coprocessor SS 7 switch

Present coprocessor n Right now, people use hardware faulttolerant machines for this – E. g. Stratus “pair and a spare” – Mimics one computer but tolerates hardware failures – Performance an issue?

Goals for coprocessor n Requirements – Scalability: ability to use a cluster of machines for the same task, with better performance when we use more nodes – Fault-tolerance: a crash or recovery shouldn’t disrupt the system – Real-time response: must satisfy the 100 ms limit at all times Downtime: any period when a series of requests might all be rejected n Desired: 7 to 9 nines availability n

SS 7 experiment Horus runs the “ 800 number database” on a cluster of processors next to the switch n Provide replication management tools n Provide failure detection and automatic configuration n

IN coprocessor example Switch itself asks for help when remote number call is sensed SS 7 switch External adaptor (EA) processors run the query protocol EA EA Query Element (QE) processors do the number lookup (inmemory database). Goals: scalable memory without loss of processing performance as number of nodes is increased Primary backup scheme adapted (using small Horus process groups) to provide fault-tolerance with real-time guarantees

Options? n A simple scheme: – Organize nodes as groups of 2 processes – Use virtual synchrony multicast • For query • For response • Also for updates and membership tracking

IN coprocessor example SS 7 switch EA EA Step 1: Switch sees incoming request

IN coprocessor example SS 7 switch EA EA Step 2: Switch waits while EA procs. multicast request to group of query elements (“partitioned” database)

IN coprocessor example SS 7 switch EA EA Think Step 3: The query elements do the query in duplicate

IN coprocessor example SS 7 switch EA EA Step 4: They reply to the group of EA processes

IN coprocessor example SS 7 switch EA EA Step 5: EA processes reply to switch, which routes call

Results!! n Terrible performance! – Solution has 2 Horus multicasts on each critical path – Experience: about 600 queries per second but no more n Also: slow to handle failures – Freezes for as long as 6 seconds n Performance doesn’t improve much with scale either

Next try Consider taking Horus off the critical path n Idea is to continue using Horus n – It manages groups – And we use it for updates to the database and for partitioning the QE set n But no multicasts on critical path – Instead use a hand-coded scheme n Use Sender Ordering (or fifo) instead of Total Ordering

Hand-coded scheme Queue up a set of requests from an EA to a QE n Periodically (15 ms), sweep the set into a message and send as a batch n Process queries also as a batch n Send the batch of replies back to EA n

Clever twists n Split into a primary and secondary EA for each request – Secondary steps in if no reply seen in 50 ms – Batch size calculated so that 50 ms should be “long enough” n Alternate primary and secondary after each request.

Handling Failure and Overload n Failure – QE: backup EA reissues request after half the deadline, without waiting for the failure detector – EA: the other EA takes over and handles all the requests n Overload – Drop requests if there is no chance of servicing them, rather than missing all deadlines – High and low watermarks

Results Able to sustain 22, 000 emulated telephone calls per second n Able to guarantee response within 100 ms and no more than 3% of calls are dropped (randomly) n Performance is not hurt by a single failure or recovery while switch is running n Can put database in memory: memory size increases with number of nodes in cluster n

Other settings with a strong temporal element n Load balancing – Idea is to track load of a set of machines – Can do this at an access point or in the client – Then want to rebalance by issuing requests preferentially to less loaded servers

Load balancing in farms n Akamai widely cited – They download the rarely-changing content from customer web sites – Distribute this to their own web farm – Then use a hacked DNS to redirect web accesses to a close-by, less-loaded machine n Real-time aspects? – The data on which this is based needs to be fresh or we’ll send to the wrong server

Conclusions n Protocols like pbcast are potentially appealing in a subset of applications that are naturally probabilistic to begin with, and where we may have knowledge of expected load levels, etc. n More traditional virtual synchrony protocols with strong consistency properties make more sense in standard networking settings

Future directions in real-time Expect GPS time sources to be common within five years n Real-time tools like periodic process groups will also be readily available (members take actions in a temporally coordinated way) n Increasing focus on predictable high performance rather than provable worst-case performance n Increasing use of probabilistic techniques n

Dimensions of Scalability We often say that we want systems that “scale” n But what does scalability mean? n As with reliability & security, the term “scalability” is very much in the eye of the beholder n

Scalability n As a reliability question: – Suppose a system experiences some rate of disruptions r – How does r change as a function of the size of the system? • If r rises when the system gets larger we would say that the system scales poorly • Need to ask what “disruption” means, and what “size” means…

Scalability n As a management question – Suppose it takes some amount of effort to set up the system – How does this effort rise for a larger configuration? – Can lead to surprising discoveries • E. g. the 2 -machine demo is easy, but setup for 100 machines is extremely hard to define

Scalability n As a question about throughput – Suppose the system can do t operations each second – Now I make the system larger • Does t increase as a function of system size? Decrease? • Is the behavior of the system stable, or unstable?

Scalability n As a question about dependency on configuration – Many technologies need to know something about the network setup or properties – The larger the system, the less we know! – This can make a technology fragile, hard to configure, and hence poorly scalable

Scalability n As a question about costs – Most systems have a basic cost • E. g. 2 pc “costs” 3 N messages – And many have a background overhead • E. g. gossip involves sending one message per round, receiving (on avg) one per round, and doing some retransmission work (rarely) n Can ask how these costs change as we make our system larger, or make the network noisier, etc

Scalability n As a question about environments – Small systems are well-behaved – But large ones are more like the Internet • Packet loss rates and congestion can be problems • Performance gets bursty and erratic • More heterogeneity of connections and of machines on which applications run – The larger the environment, the nastier it may be!

Scalability n As a pro-active question – How can we design for scalability? – We know a lot about technologies – Are certain styles of system more scalable than others?

Approaches n Many ways to evaluate systems: – Experiments on the real system – Emulation environments – Simulation – Theoretical (“analytic”) n But we need to know what we want to evaluate

Dangers n “Lies, damn lies, and statistics” – It is much to easy to pick some random property of a system, graph it as a function of something, and declare success – We need sophistication in designing our evaluation or we’ll miss the point n Example: message overhead of gossip – Technically, O(n) – Does any process or link see this cost? • Perhaps not, if protocol is designed carefully

Technologies n n n n TCP/IP and O/S message-passing architectures like U-Net RPC and client-server architectures Transactions and nested transactions Virtual synchrony and replication Other forms of multicast Object oriented architectures Cluster management facilities

You’ve Got Mail Cluster research has focused on web services n Mail is an example of a write-intensive application n – disk-bound workload – reliability requirements – failure recovery n Mail servers have relied on “brute force” approach to scaling – Big-iron file server, RDBMS

Conventional Mail Servers Static partitioning Performance problems: No dynamic load balancing Manageability problems: Manual data partition decision Availability problems: popd sendmail Limited fault tolerance NFS Server User DB Server

Porcupine’s Goals Use commodity hardware to build a large, scalable mail service n Performance: Linear increase with cluster size n Manageability: React to changes automatically n Availability: Survive failures gracefully 1 billion messages/day (100 x existing systems) 100 million users (10 x existing systems) 1000 nodes (50 x existing systems)

Key Techniques and Relationships Functional Homogeneity “any node can perform any task” Automatic Load Replication Reconfiguration Balancing Availability Manageability Performance Framework Techniques Goals

Porcupine Architecture SMTP server POP server IMAP server Load Balancer User map Membership RPC Manager Replication Manager . . . Node A Node B . . . Mailbox storage Node Z User profile Mail map

Basic Data Structures “bob” Apply hash function B CACABAC User map bob: {A, C} Suzy’s MSGs Bob’s MSGs A suzy: {A, C} ann: {B} Ann’s MSGs Joe’s MSGs B joe: {B} Bob’s MSGs Mail map /user info Suzy’s Mailbox MSGs storage C

Porcupine Operations Protocol handling User lookup Internet A Message store C A DNS-RR 1. “send selection mail to 3. “Verify bob” . . . Load Balancing B C . . . 2. Who manages bob? A 4. “OK, bob has msgs on C and D 6. “Store msg” B 5. Pick the best nodes to store new msg C

Measurement Environment 30 node cluster of not-quite-all-identical PCs 100 Mb/s Ethernet + 1 Gb/s hubs Linux 2. 2. 7 42, 000 lines of C++ code Synthetic load Compare to sendmail+popd

Performance Goals Scale performance linearly with cluster size Strategy: Avoid creating hot spots Partition data uniformly among nodes Fine-grain data partition

How does Performance Scale? 68 m/day 25 m/day

Availability Goals: Maintain function after failures React quickly to changes regardless of cluster size Graceful performance degradation / improvement Strategy: Hard state: email messages, user profile Optimistic fine-grain replication Soft state: user map, mail map Reconstruction after membership change

Soft-state Reconstruction 1. Membership protocol Usermap recomputation A B C A B A C bob: {A, C} B A A B A B bob: {A, C} suzy: 2. Distributed disk scan A C A C bob: {A, C} suzy: {A, B} B C A B A C joe: {C} B A A B A B joe: {C} ann: A C A C joe: {C} ann: {B} B C A B A B A C suzy: {A, B} ann: {B} B C A B A C suzy: {A, B} ann: {B} Timeline

How does Porcupine React to Configuration Changes?

Hard-state Replication Goals: Keep serving hard state after failures Handle unusual failure modes Strategy: Exploit Internet semantics Optimistic, eventually consistent replication Per-message, per-user-profile replication Efficient during normal operation Small window of inconsistency

How Efficient is Replication? 68 m/day 24 m/day

How Efficient is Replication? 68 m/day 33 m/day 24 m/day

Load balancing: Deciding where to store messages Goals: Handle skewed workload well Support hardware heterogeneity Strategy: Spread-based load balancing Spread: soft limit on # of nodes per mailbox Large spread better load balance Small spread better affinity Load balanced within spread Use # of pending I/O requests as the load measure

How Well does Porcupine Support Heterogeneous Clusters? +16. 8 m/day (+25%) +0. 5 m/day (+0. 8%)

Claims Symmetric function distribution n Distribute user database and user mailbox n – Lazy data management n Self-management – Automatic load balancing, membership management n Graceful Degradation – Cluster remains functional despite any number of failures

Retrospect n Questions: – How does the system scale? – How costly is the failure recovery procedure? n Two scenarios tested – Steady state – Node failure n Does Porcupine scale? – Papers says “yes” – But in their work we can see a reconfiguration disruption when nodes fail or recover • With larger scale, frequency of such events will rise • And the cost is linear in system size – Very likely that on large clusters this overhead would become dominant!

Some Other Interesting Papers n The Next Generation Internet: Unsafe at any Speed? Ken Birman n Lessons from Giant-Scale Services Eric Brewer, UCB