Distributed PublishSubscribe Nalini Venkatasubramanian with slides from Roberto

Distributed Publish/Subscribe Nalini Venkatasubramanian (with slides from Roberto Baldoni, Pascal Felber, Hojjat Jafarpour etc. )

Publish/Subscribe (pub/sub) systems Asynchronous communication n What is • Publish/Subscribe (pub/sub)? • Selective dissemination • Push model Stock ( Name=‘IBM’; Price < 100 ; Volume>10000 ) • Decoupling publishers and subscribers Stock ( Name=‘IBM’; Price =95 ; Volume=50000 ) Pub/Sub Service Stock ( Name=‘IBM’; Price =95 ; Volume=50000 ) Stock ( Name=‘HP’; Price < 50 ; Volume >1000 ) Football( Team=‘USC’; Event=‘Touch Down’) Hojjat Jafarpour Stock ( Name=‘IBM’; Price < 110 ; Volume>10000 ) CCD: Efficient Customized Content Dissemination in Distributed Pub/Sub 2

Publish/Subscribe (pub/sub) systems n Applications: q q q q News alerts Online stock quotes Internet games Sensor networks Location-based services Network management Internet auctions … Hojjat Jafarpour CCD: Efficient Customized Content Dissemination in Distributed Pub/Sub 3

Publish/subscribe architectures n Centralized q q n Broker overlay q q q n Single matching engine Limited scalability Multiple P/S brokers Participants connected to some broker Events routed through overlay Peer-to-peer q q q Publishers & subscribers connected in P 2 P network Participants collectively filter/route events, can be both producer & consumer ……. Scalable Publish/Subscribe Architectures & Algorithms — P. Felber 4

Distributed pub/sub systems n Broker – based pub/sub q A set of brokers forming an overlay n n Clients use system through brokers Benefits q Scalability, Fault tolerance, Cost efficiency Dissemination Tree

Challenges in distributed pub/sub systems Broker Responsibility Subscription Management Matching: Determining the recipients for an event Routing: Delivering a notification to all the recipients Broker overlay architecture • How to form the broker network • How to route subscriptions and publications Broker internal operations • Subscription management • How to store subscriptions in brokers • Content matching in brokers • How to match a publication against subscriptions 6

EVENT vs SUBSCRIPTION ROUTING n Extreme solutions q Sol 1 (event flooding) n n n q flooding of events in the notification event box each subscription stored only in one place within the notification event box Matching operations equal to the number of brokers Sol 2 (subscription flooding) n n each subscription stored at any place within the notification event box each event matched directly at the broker where the event enters the notification event box MINEMA Summer School - Klagenfurt (Austria) July 11 -15, 2005 7

Major distributed pub/sub approaches n Tree-based q n DHT-based: q n Brokers form a structured P 2 P overlay [Meghdoot, Baldoni et al. ] Channel-based: q n Brokers form a tree overlay [SIENA, PADRES, GRYPHON] Multiple multicast groups [Phillip Yu et al. ] Probabilistic: q Unstructured overlay [Picco et al. ] 8

Tree-based n n n Brokers form an acyclic graph Subscriptions are broadcast to all brokers Publications are disseminated along the tree with applying subscriptions as filters 9

Tree-based n Subscription dissemination load reduction q q n Subscription Covering Subscription Subsumption Publication matching q Index selection 10

Pub/Sub Sysems: Tib/RV [Oki et al 03] n n Topic Based Two level hierarchical architecture of brokers (deamons) on TCP/IP Event routing is realized through one diffusion tree per subject Each broker knows the entire network topology and current subscription configuration MINEMA Summer School - Klagenfurt (Austria) July 11 -15, 2005 11

Pub/Sub systems: Gryphon [IBM 00] n n Content based Hierarchical tree from publishers to subscribers Filtering-based routing Mapping content-based to network level multicast MINEMA Summer School - Klagenfurt (Austria) July 11 -15, 2005 12

DHT Based Pub/Sub: SCRIBE [Castro et al. 02] n n n Topic Based on DHT (Pastry) Rendez-vous event routing A random identifier is assigned to each topic The pastry node with the identifier closest to the one of the topic becomes responsible for that topic MINEMA Summer School - Klagenfurt (Austria) July 11 -15, 2005 13

DHT-based pub/sub MEGHDOOT n n Content Based on Structured Overlay CAN Mapping the subscription language and the event space to CAN space Subscription and event Routing exploit CAN routing algorithms MINEMA Summer School - Klagenfurt (Austria) July 11 -15, 2005 14

Fault-tolerance Pub/Sub architecture n n n Brokers are clustered Each broker knows all brokers in its own cluster and at least one broker from every other clusters Subscriptions are broadcast just in clusters Every brokers just have the subscriptions from brokers in the same cluster Subscription aggregation is done based on brokers 15

Fault-tolerance Pub/Sub architecture n Broker overlay q q q Join Leave Failure n n Detection Masking Recovery Load Balancing q q q Ring publish load Cluster subscription load 16

Customized content delivery with pub/sub Customize content to the required formats before delivery! Español!!! Hojjat Jafarpour CCD: Efficient Customized Content Dissemination in Distributed Pub/Sub 17

Motivation n Leveraging pub/sub framework for dissemination of rich content formats, e. g. , multimedia content. Same content format may not be consumable by all subscribers!!! Hojjat Jafarpour CCD: Efficient Customized Content Dissemination in Distributed Pub/Sub 18

Content customization n How content customization is done? q Adaptation operators Original content Size: 28 MB Hojjat Jafarpour Transcoder Operator CCD: Efficient Customized Content Dissemination in Distributed Pub/Sub Low resolution and small content suitable for mobile clients Size: 8 MB 19

Challenges n How to do customization in distributed pub/sub? Hojjat Jafarpour CCD: Efficient Customized Content Dissemination in Distributed Pub/Sub 20

Challenges n Option 1: Perform all the required customizations in the sender broker 28 MB 28+12+8 = 48 MB 8 MB 15 MB 12 MB 8 MB Hojjat Jafarpour 12 MB 28 MB 15 MB 28 MB CCD: Efficient Customized Content Dissemination in Distributed Pub/Sub 8 MB 21

Challenges n Option 2: Perform all the required customization in the proxy brokers (leaves) 28 MB Repeated Operator 8 MB 15 MB 28 MB Hojjat Jafarpour 12 MB 28 MB 15 MB 28 MB CCD: Efficient Customized Content Dissemination in Distributed Pub/Sub 8 MB 22

Challenges n Option 3: Perform all the required customization in the broker overlay network 28 MB 8 MB Hojjat Jafarpour 15 MB 12 MB 28 MB 15 MB 28 MB CCD: Efficient Customized Content Dissemination in Distributed Pub/Sub 8 MB 23

Publisher of C [(Shelter Info, Santa Ana, School), (Spanish, Voi ce)] 1130 1230 Super Peer Network RP Peer for C Translation 1030 2130 2330 0130 2230 1330 2330 1130 3130 [(Shelter Information, Irvine, School), (English, Text)] 0330 Speech to text [(Shelter Information, Irvine, School), (English, Text)] 24

Publisher of C Translation [(Shelter Info, Santa Ana, School), (Spanish, Voi ce)] 1130 1230 Super Peer Network RP Peer for C 1030 2130 2330 0130 2230 1330 Speech to text 1130 0330 2330 3130 [(Shelter Information, Irvine, School), (English, Text)] 25

Publisher of C [(Shelter Info, Santa Ana, School), (Spanish, Voi ce)] 1130 1230 Super Peer Network RP Peer for C 1030 2130 Translation 2130 2330 0130 2230 1330 0330 Speech to text 2330 1130 3130 [(Shelter Information, Irvine, School), (English, Text)] 26

DHT-based pub/sub n DHT-based routing schema, q We use Tapestry [ZHS 04] Rendezvous Point Hojjat Jafarpour CCD: Efficient Customized Content Dissemination in Distributed Pub/Sub 27

Example using DHT based pubsub n Tapestry (DHT-based) pub/sub and routing framework q Event space is partitioned among peers n q q Each partition is assigned to a peer (RP) Publications and subscriptions are matched in RP n q Single content matching All receivers and preferences are detected after matching Content dissemination among matched subscribers are done through a dissemination tree rooted at RP where leaves are subscribers. 28

Background n Tapestry DHT-based overlay q q Each node has a unique L-digit ID in base B Each node has a neighbor map table (Lx. B) Routing from one node to another node is done by resolving one digit in each step Sample routing map table for 2120 29

Dissemination tree n For a published content we can estimate the dissemination tree in broker overlay network q q Using DHT-based routing properties The dissemination tree is rooted at the corresponding rendezvous broker Rendezvous Point Hojjat Jafarpour CCD: Efficient Customized Content Dissemination in Distributed Pub/Sub 30

Subscriptions in CCD n n How to specify required formats? Receiving context: q Receiving device capabilities n q Context: PC, DSL, AVI Subscription: • Team: USC • Video: Touch Down Context: Phone, 3 G, FLV Communication capabilities n q Display screen, available software, … Subscription: • Team: USC • Video: Touch Down Available bandwidth User profile n Location, language, … Hojjat Jafarpour Subscription: • Team: USC • Video: Touch Down Context: Laptop, 3 G, AVI, Spanish subtitle CCD: Efficient Customized Content Dissemination in Distributed Pub/Sub 31

Content Adaptation Graph (CAG) n n All possible content formats in the system All available adaptation operators in the system Size: 28 MB Frame size: 1280 x 720 Frame rate: 30 Size: 15 MB Frame size: 704 x 576 Frame rate: 30 Size: 8 MB Frame size: 128 x 96 Frame rate: 30 Size: 10 MB Frame size: 352 x 288 Frame rate: 30 Hojjat Jafarpour CCD: Efficient Customized Content Dissemination in Distributed Pub/Sub 32

Content Adaptation Graph (CAG) n A transmission (communication) cost is associated with each format q n Sending content in format Fi from a broker to another one has the transmission cost of A computation cost is associated with each operator q Performing operator O(i, j) on content of has the computation cost F 1/28 V={F 1, F 2, F 3, F 4} E={O(1, 2), O(1, 3), O(1, 4), O(2, 3), O(2, 4), O(3, 4)} 60 F 2/15 60 25 F 3/12 60 25 F 4/8 25 Hojjat Jafarpour CCD: Efficient Customized Content Dissemination in Distributed Pub/Sub 33

CCD plan A CCD plan for a content is the dissemination tree: n q q Each node (broker) is annotated with the operator(s) that are performed on it Each link is annotated with the format(s) that are transmitted over it {O(1, 2), O(2, 4)} {F 2} F 1/28 60 F 2/15 60 60 25 F 3/12 25 {} {O(2, 3)} {F 2} F 4/8 25 {} Hojjat Jafarpour CCD: Efficient Customized Content Dissemination in Distributed Pub/Sub {F 4} {F 2} {} {F 4} {F 3} {} {} 34

CCD algorithm n n Input: q A dissemination tree q A CAG q The initial format q Requested formats by each broker Output: q The minimum cost CCD plan Hojjat Jafarpour CCD: Efficient Customized Content Dissemination in Distributed Pub/Sub 35

CCD Problem is NP-hard n Directed Steiner tree problem can be reduced to CCD Given a directed weighted graph G(V, E, w) , a specified root r and a subset of its vertices S, find a tree rooted at r of minimal weight which includes all vertices in S. Hojjat Jafarpour CCD: Efficient Customized Content Dissemination in Distributed Pub/Sub 36

CCD algorithm n n n Based on dynamic programming Annotates the dissemination tree in a bottom-up fashion For each broker: q q Assume all the optimal sub plans are available for each child Find the optimal plan for the broker accordingly Ni Nj Hojjat Jafarpour …. CCD: Efficient Customized Content Dissemination in Distributed Pub/Sub Nk 37

CCD algorithm F 1/28 60 F 2/15 25 60 F 3/12 25 Hojjat Jafarpour 25 F 2 F 4 60 F 4/8 F 4 F 3 F 1 CCD: Efficient Customized Content Dissemination in Distributed Pub/Sub F 2 F 1 F 4 38

System model n Set of supported formats and communication cost for transmitting content in each format n Set of operators with cost of performing each operator n Operators are available is all brokers 39

System model n Content Adaptation Graph q q n n Represents available formats and operators and their relation G = (V , E) where V = F and E = O Fx. F For a given CAG and dissemination tree, , find CCD plan with minimum total cost. Optimal content adaptation is NP-Hard q Steiner tree problem 40

System model n Subscription model: q n [SC, SF ] where SC is the content subscription and SF corresponds to the format in which the matching publication is to be delivered. n S=[{SC: Type = ’image’, Location = ’Southern California’, Category = ’Wild Fire’}, {Format = ’PDA-Format’}] Publication model: q A publication P = [PC, PF ] also consists of two parts. PC contains meta data about the content and the content itself. The second part represents the format of the content. n [{Location = ’Los Angeles County’ , Category =’Fire, Wildfire, Burning’, image}, {Format = ’PC-Format’}] 41

Customized dissemination in homogeneous overlay n Optimal operator placement q q q n Results in minimum dissemination cost Needs to know the dissemination tree for the published content Assumes small adaptation graphs (Needs enumeration of different subsets of formats) Observation: q If B is a leaf in dissemination tree q Otherwise 42

Customized dissemination in homogeneous overlay n The minimum cost for customized dissemination tree in node B is computed as follow. q If B is a leaf in the dissemination tree then q Otherwise 43

Operator placement in homogeneous overlay n Optimal operator placement 44

Experimental evaluation n Implemented scenarios q Homogeneous overlay n n n q Optimal Only root TRECC All in root All in leaves Heterogeneous n n n Optimal All in root All in leaves 45

Experimental evaluation 46

Extensions n n Extending the CAG to represent parameterized adaption Heuristics for larger CAGs and parameterized adaptations 47

Fast and scalable notification using Pub/Sub n A general purpose notification system q n On line deals, news, traffic, weather, … Supporting heterogeneous receivers User Profile Client User Subscriptions Notifications Pub/Sub Server Web 48

User profile n Personal information q q q n Name Location Language Receiving modality q PC, PDA n n n q Email Live notification IM (Yahoo Messenger, Google Talk, AIM, MSN) Cell phone n n SMS Call 49

Subscription n Subscription language in the system q n SQL Subscriptions language for clients q Attribute value n E. g. , q q Website = www. dealsea. com Keywords = Laptop, Notebook Price <= $1000 Brand = Dell, HP, Toshiba, SONY 50

Notifications n n Customized for the receiving device Includes q q Title URL Short description May include multimedia content too. 51

Client application n A stand alone java-based client q q JMS client for communications Must support many devices 52

Experimental evaluation n System setup q q 1024 brokers Matching ratio: percentage of brokers with matching subscription for a published content n q Zipf and uniform distributions Communication and computation costs are assigned based on profiling Hojjat Jafarpour CCD: Efficient Customized Content Dissemination in Distributed Pub/Sub 53

Experimental evaluation n Dissemination scenarios q q q Annotated map Customized video dissemination Synthetic scenarios Hojjat Jafarpour CCD: Efficient Customized Content Dissemination in Distributed Pub/Sub 54

Cost reduction in CCD algorithm Cost reduction percentage (%) 50 CCD vs. All In Leaves 45 CCD vs. All In Root 40 35 30 25 20 15 10 5 0 1 5 10 20 50 70 Matching Ratio Hojjat Jafarpour CCD: Efficient Customized Content Dissemination in Distributed Pub/Sub 55

Cost reduction in Heuristic CCD Cost reduction percentage (%) 60 50 40 Heuristic CCD vs. All In Leaves 30 Heuristic CCD vs. All In Root 20 10 0 1 5 10 20 50 70 Matching Ratio Hojjat Jafarpour CCD: Efficient Customized Content Dissemination in Distributed Pub/Sub 56

CCD vs. heuristic CCD 6% Cost reduction percentage (%) Matching ratio = 5% Matching ratio = 50% 5% Matching ratio = 70% 4% 3% 2% 1% 0% 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 Iteration number Hojjat Jafarpour CCD: Efficient Customized Content Dissemination in Distributed Pub/Sub 57

References n n n [AT 06] Ioannis Aekaterinidis, Peter Triantafillou: Pastry. Strings: A Comprehensive Content-Based Publish/Subscribe DHT Network. IEEE ICDCS 2006. [CRW 04] A. Carzaniga, M. J. Rutherford, and A. L. Wolf: A Routing Scheme for Content-Based Networking. IEEE INFOCOM 2004. [DRF 04] Yanlei Diao, Shariq Rizvi, Michael J. Franklin: Towards an Internet-Scale XML Dissemination Service. VLDB 2004. [GSAE 04] Abhishek Gupta, Ozgur D. Sahin, Divyakant Agrawal, Amr El Abbadi: Meghdoot: Content-Based Publish/Subscribe over P 2 P Networks. ACM Middleware 2004 [JHMV 08] Hojjat Jafarpour, Bijit Hore, Sharad Mehrotra and Nalini Venkatasubramanian. Subscription Subsumption Evaluation for Content-based Publish/Subscribe Systems, ACM/IFIP/USENIX Middleware 2008. [JHMV 09] Hojjat Jafarpour, Bijit Hore, Sharad Mehrotra and Nalini Venkatasubramanian. CCD: Efficient Customized Content Dissemination in Distributed Publish/Subscribe. ACM/IFIP/USENIX Middleware 2009. [JMV 08] Hojjat Jafarpour, Sharad Mehrotra and Nalini Venkatasubramanian. A Fast and Robust Content-based Publish/Subscribe Architecture, IEEE NCA 2008. [JMV 09] Hojjat Jafarpour, Sharad Mehrotra and Nalini Venkatasubramanian. Dynamic Load Balancing for Clusterbased Publish/Subscribe System, IEEE SAINT 2009. [JMVM 09] Hojjat Jafarpour, Sharad Mehrotra, Nalini Venkatasubramanian and Mirko Montanari, MICS: An Efficient Content Space Representation Model for Publish/Subscribe Systems, ACM DEBS 2009. [OAABSS 00] Lukasz Opyrchal, Mark Astley, Joshua S. Auerbach, Guruduth Banavar, Robert E. Strom, Daniel C. Sturman: Exploiting IP Multicast in Content-Based Publish-Subscribe Systems. Middleware 2000. [ZHS 04] Ben Y. Zhao, Ling Huang, Jeremy Stribling, Sean C. Rhea, Anthony D. Joseph, John Kubiatowicz: Tapestry: a resilient global-scale overlay for service deployment. IEEE Journal on Selected Areas in Communications 22(1). Hojjat Jafarpour CCD: Efficient Customized Content Dissemination in Distributed Pub/Sub 58