Mercury Building Distributed Applications with Publish Subscribe Ashwin

Mercury: Building Distributed Applications with Publish. Subscribe Ashwin Bharambe Carnegie Mellon University Monday Seminar Talk

Quick Terminology Recap n Basics q q q n Mercury: distributed publish-subscribe system q n Publishers: inject data/events/publications Subscribers: register interests/subscriptions Brokers: match subscriptions with publications and deliver to subscribers Performs matching and content routing in distributed fashion Data model Name = ashwin Age = 23 X = 192. 3 Y = 223. 4 Publication Ashwin Bharambe Name = * Age > 35 X > 100 X < 180 Subscription Carnegie Mellon University

Virtual reality example (50, 250) (100, 200) Events x 100 y 200 User x x y y ≥ ≤ 50 150 250 Interests Ashwin Bharambe Arena (150, 150) Virtual World Carnegie Mellon University

Mercury goals n Implement distributed publish-subscribe q n n Support range queries Avoid hot-spots in the system Flooding anything is bad q q Avoid publication flooding completely Avoid subscription flooding as much as is possible n n Consider queries like SELECT * from RECORDS Peer-to-peer scenario q q No dedicated brokers Highly dynamic network Ashwin Bharambe Carnegie Mellon University

Talk Contents n n Mercury Architecture Overlay construction Routing guarantees Overlay properties q n n How randomness is useful Load balancing; histogram maintenance Application Design Ashwin Bharambe Carnegie Mellon University

Attribute Hubs 1000 0 150 250 900 age name x y Hubs in the system n n X - hub 700 450 Structure of a single hub Each attribute range is divided into bins A node responsible for range of attribute values q Assigned when the node joins; can change dynamically Ashwin Bharambe Carnegie Mellon University

Routing Generating point y S age S name Subscription x n Name = * X > 100 X < 180 Send a subscription to one hub q q Which one? Interesting question in itself! Determine query selectivity – send to “highest selective” hub Ashwin Bharambe Carnegie Mellon University

Routing (contd. ) Generating point x age P P P name y n Publication Name = ashwin Age = 23 We must send publications to all hubs q Ensures matching Ashwin Bharambe Carnegie Mellon University

Routing illustrated Subscription [240, 320) [160, 240) Hx Rendezvous [80, 160) point Ashwin Bharambe 50 ≤ x ≤ 150 ≤ y ≤ 250 [0, 80) Publication x 100 y 200 [0, 105) Hy [105, 210) [210, 320) Carnegie Mellon University

Hub structure and routing (~Symphony) n n Naïve routing along the circle scales linearly Utilize the small-world phenomenon [Kleinberg 2000] q q n Know thy neighbors and one random person; and you can contact anybody quickly Routing policy: choose the link which gets you closest to destn Performance q Average hop length = O(log 2 (n)/k) with k “random” links Choose this link with probability: P(x) = 1/(x ln n) x Need to be careful when node ranges are not uniform Ashwin Bharambe Carnegie Mellon University

Caching n O(log 2 (n)) is good, but each hop is still an application level hop q q n n Latency can be quite large if overlay non-optimized For distributed applications like games, this is way off from optimal Exploit locality in the access patterns of an application In addition to k “random” links, have cached links q Store nodes which were the rendezvous points for recent publications Ashwin Bharambe Carnegie Mellon University

Performance (Uniform workload) #long links = 6 #cache links = log(n) Publications were generated from a uniform distribution Ashwin Bharambe Carnegie Mellon University

Performance (Skewed workload) #long links = 6 #cache links = log(n) Publications were generated from a high skew Zipf distribution Ashwin Bharambe Carnegie Mellon University

Performance (Memory reference trace) #long links = 6 #cache links = log(n) Publications were generated from memory references of SPEC 2000 benchmark Ashwin Bharambe Carnegie Mellon University

Two Problems 1. Load Balancing q Pr(X=x) q Concern because publication values need not follow a uniform, or a priori known, distribution Node ranges are assigned when the nodes join x Ashwin Bharambe Carnegie Mellon University

Problems (contd. ) 2. Hub Selectivity q q q Recall: subscription is sent to one “randomly” chosen hub! Ideally, it should be sent to the “highest selective” hub Need to estimate selectivity of a subscription Name = * X > 100 X < 180 Sending to Name hub vs. X hub Ashwin Bharambe Carnegie Mellon University

Hail randomness n Randomized construction of the network gives additional benefits! q q n Uniform sampling non-trivial q n Node ranges are not uniform across nodes Random walks: efficient way of sampling q n Turns out, this network is an Expander with high probability Random walks mix rapidly – i. e. , they approach the stationary distribution rapidly No explicit hierarchy required (as in RANSUB [USITS ’ 03]) In general, several statistics about a very dynamic network can be efficiently maintained Ashwin Bharambe Carnegie Mellon University

Hub Selectivity (ideas) n n Use sampling to build approximate histograms Approach 1: (Push) q q q n Each “Rendezvous point” selects publications with a certain probability and sends them off with specific TTL log 2(n) length random walk ensures good mixing Traffic overhead / #publications Approach 2: (Pull) q q q Perform uniform random sampling periodically Each sample = histogram of sampled node Question: how to combine histograms? Ashwin Bharambe Carnegie Mellon University

Load balancing (ideas) n n n Sample “average” load in the system Utilize the histograms to quickly know high/low load areas Strategy 1: q q n A “light” load gracefully leaves the overlay Re-inserts itself into a “high” load area Strategy 2: q Use load “diffusion” – “heavy” nodes shed load to neighbors n Only if the neighbor is “light” Ashwin Bharambe Carnegie Mellon University

Distributed Game Design n n Current implementation: Distributed version of the Asteroids game! Questions: q q q How is state distributed across the system? How is consistency handled in the system? Cheating? ? ? Ashwin Bharambe Carnegie Mellon University

Conclusion n Distributed publish-subscribe system supporting q q n Randomized network construction q q n n n Range queries Scalable routing and matching Provides routing guarantees Also yields an elegant way of sampling in a distributed system Exports an API for applications Implemented; deployed on Emulab Distributed game using Mercury q q Almost done To be deployed on Planetlab soon Ashwin Bharambe Carnegie Mellon University