VOLTTRONRabbit MQ Message Bus Overview Shwetha Niddodi Chandrika
VOLTTRON-Rabbit. MQ Message Bus Overview Shwetha Niddodi Chandrika Sivaramakrishnan Kyle Monson Craig Allwardt Jereme Haack 1
Outline Background Message Bus Plugin Framework Rabbit. MQ Overview Rabbit. MQ VOLTTRON Conclusion 2
Background VOLTTRON’s Zero. MQ based message bus has been key for meeting the security and interoperability goals of the platform At the same time, Rabbit. MQ has become more mature as it has seen major investment by commercial companies. Rabbit Technologies, now part of Pivotal Technologies (VMWARE spin-out). $105 million investment by GE in 2013. Used by: Instagram, Indeed. com, Google Cloud Platform, Tesla … Goals of the Refactor: Maintain essential features of current message bus and minimize transition cost Leverage an existing and growing community dedicated to the further development of Rabbit. MQ Move services provided currently by VOLTTRON agents to services natively provided by Rabbit. MQ Decrease VOLTTRON development time spent on supporting message bus which is now a commodity technology. Address concerns from community about Zero. MQ View this effort as essential to the long-term future of the platform Working with heavy users in the community to get feedback Reduce long term costs of platform by moving message bus development out of core Maintain support for ZMQ short term (3 – 5 years) as funding allows 3
Why Rabbit. MQ? Supports different messaging patterns, routing topologies Easy to use, has most of the features already built-in Well developed security feature Large scale deployment Flexibility in deployment Recommended by VOLTTRON community Other competitive message buses that were considered. Kafka – Uses “dumb broker, smart consumer approach” which is opposite of our requirement. MQTT – Limitation in security feature. The payload can be secured but not the header 4
Message Bus Plugin Framework Decouple VOLTTRON from message bus implementation Similar concept to Historian and Driver frameworks Provide hooks to allow new solutions to be leveraged “Proxy Agent” that handles VOLTTRON specific code needed to interact with different types of message buses Must prevent fracturing of ecosystem Ensure VOLTTRONs using different messaging systems can still communicate with each other 5
Upgrading to Rabbit. MQ Based Platform Minimize changes needed to agent code No effect unless agent has ZMQ specific code (Forward. Historian) No changes required to driver configuration Data collection unaffected Backward compatibility Allow ZMQ and Rabbit. MQ instances to communicate Will require new code to connect Rabbit. MQ based systems together 6
Message Bus Plugin Framework Consists of five components New connection class per message bus Extensions to router module functionality Extensions to core agent functionality Adding a proxy agent for each message bus Authentication related changes 7
Platform Level Changes On startup, checks the type of message bus used. Creates appropriate router module ZMQ Router – Handles actual routing since ZMQ is broker less protocol. Functionality is unchanged. RMQ Router – Light weight. Maintains connection to RMQ message bus Actual routing is handled by RMQ broker Handles router specific subsystem messages Handles unrouteable messages 8
Agent Core Changes Application agent code remain unchanged Agent Core changes On startup, checks the type of message bus used. Maintains connection to appropriate message bus. All subsystem messages are encapsulated inside a message bus agnostic VIP message object. 9
Compatibility between VOLTTRON instances running different message buses Proxy Agent – Acts like a bridge between local message bus and remote message bus Different proxy agent for each type of remote message bus connection. Maintains connections to internal and external message bus Route messages from internal to external Route messages from external to internal 10
Proxy Router Agent acting as bridge ` V 2 runs on RMQ message bus V 1 runs on ZMQ message bus RMQ Message Translation layer ZMQ Router ZMQ RMQ Message Translation layer XYZ ZMQ Message Translation layer XYZ V 3 runs on XYZ message bus 11
Rabbit. MQ Overview (contd. ) Rabbit. MQ uses AMQP (Advanced Message Queuing Protocol) Exchanges – Responsible for routing of messages to Queues. They look at the routing key in the message when deciding how to route messages to queues. 12
Rabbit. MQ Overview Queues - Buffer that stores the messages until consumed by consumer. Bindings – Queues bind to the exchange with binding keys or routing pattern. Messages are routed to one or many queues based on a matching between a message routing key and binding key. Queue Consumer 1 Binding key: “green” Producer Exchange Queue Consumer 2 Binding key: “red” Routing Key: “green” Queue Consumer 3 Binding key: “green” Queue Binding key: “yellow” 13 Consumer 4
Rabbit. MQ VOLTTRON 14 Uses Pika library – Python library for Rabbit. MQ Each VOLTTRON instance creates a single topic exchange – “volttron” Each Agent Core Connects to Rabbit. MQ broker Creates VIP queue and binds to exchange with binding key “<instance name>. <agent identity>” Send/Receive VIP messages using Pika methods Routes incoming messages to appropriate subsystem Platform Router Connects to Rabbit. MQ broker Creates VIP queue and binds to exchange with binding key “<instance name>. <router>” Handles messages intended to the router Handles unrouteable messages
RPC Subsystem Agent_B VIP ID: “agent_b” Agent_A VIP ID: “agent_a” agent_a VIP queue Binding Key: “volttron 1. agent_a” Message: Destination Routing Key: “volttron 1. agent_b” Pika properties: Type: ”rpc” “user_id”: “volttron 1. agent_a” Body: message arguments Topic Exchange Alternate Exchange (Fan out) agent_b VIPqueue Binding Key: “volttron 1. agent_b” Router/Bad Message Handler Message: Destination Routing Key: “volttron 1. agent_a” Pika properties: Type: ”rpc” app_id: ”volttron 1. agent_b” “user_id”: “volttron 1. agent_a” Message_id: message id Body: message return result
Rabbit. MQ PUBSUB Agent functionality remains unchanged. Agents continue to use same Pub. Sub interfaces User can continue to use topics delimited by “/”. Rabbit. MQ PUBSUB converts it to “. ” internally. Agent topics are internally prefixed “__pubsub__. <platform_id>” to differentiate from main Agent binding. If agent wants to subscribe to topic from remote instances, it uses agent. vip. subscribe(“pubsub”, “devices. hvac 1”, all_platforms=True”) It is internally set to “__pubsub__. *. <remainder of topic>” 16
Pub. Subsystem Agent_B VIP ID: “agent_b” Agent_A VIP ID: “agent_a” User PUBSUB message handler Publish Message: Routing Key: “__pubsub__. volttron 1. devices. hvac 1” VIP queue Binding Key: “volttron 1. agent _a” Pika properties: Type: ”pubsub” user_id: ”volttron 1. agent_a” Message_id: result. ident Body: message arguments Subscribe: Pubsub queue: “volttron 1. *” Routing Key: “__pubsub__. volttron 1. devices. #” Message_id: message id Callback handler: user pubsub message handler Topic Exchange
Multi-Platform Communication With ZMQ based VOLTTRON Write an agent that would connect to remote instance directly. Use special agents such as forwarder/data puller agents to forward/receive messages to/from remote instances. Shovel Plugin can be used in Rabbit. MQ VOLTTRON Configure VIP address of all remote instances in $VOLTTRON_HOME/external_discovery. json. Let the router module in each instance manage the connection and message routing for us. Federation plugin can be used in Rabbit. MQ VOLTTRON 18
Federation Plugin Used for connecting multiple brokers Loose coupling of nodes WAN friendly – Tolerates network intermittency Specificity – Not everything needs to be federated. Scalability - Does not require O(n 2) connections between n brokers Allows you to make exchanges and queues federated. A federated exchange can route messages published upstream to a local queue. A federated queue lets a local consumer receive messages from an upstream queue 19 Reference: https: //www. rabbitmq. com/federation. html
Shovel Plugin Used to move messages from one broker to another broker Loose coupling WAN friendly - Tolerates network intermittency Dynamic shovels do not need restart of broker Acts as well-written client application that Connects to source and destination broker Consumes messages from the queue Re-publishes messages to the destination maintaining the same message format (and routing key if needed) Messages forwarded based on routing key or pattern match Useful if one of the instance is behind NAT Shovel setup requires foreknowledge of topic subscriptions. Limit unneeded traffic across the wire Does not adapt to subscriptions automatically like a Federation link. Reference: https: //www. rabbitmq. com/shovel. html 20
VOLTTRON - 1 RPC Multiplatform Agent_A VIP ID: “agent_a” RMQ User ID: “volttron 1. agent_a” Agent_B VIP ID: “agent_b” RMQ User ID: “volttron 1. agent_b” Agent_B input queue Binding Key: “volttron 1. agent_a” Agent_B input queue Binding Key: “volttron 1. agent_b” Message: Routing Key: “volttron 2. agent_b” AMPQ: Type: ”rpc” user_id: ”volttron 1. agent_a” 8/21/2018 Topic Exchange Federation Link VOLTTRON - 2 Agent_B VIP ID: “agent_b” RMQ User ID: “volttron 2. agent_b” Agent_B input queue Binding Key: “volttron 2. agent_b” Topic Exchange Message: Routing Key: “volttron 1. agent_a” AMPQ: Type: ”rpc” user_id: “volttron 2. agent_b” 21
VOLTTRON – 1 (behind NAT) RPC over NAT Multiplatform VOLTTRON - 2 Agent_A VIP ID: “agent_a” Agent_B VIP ID: “agent_b” Agent_B input queue Binding Key: “volttron 1. agent_a” Agent_B input queue Binding Key: “volttron 1. agent_b” Agent_B input queue Binding Key: “volttron 2. agent_b” Message: Key: “volttron 2. agent_b” AMPQ: Type: ”rpc” user_id: ” volttron 1. a gent_a” Topic Exchange Shovel Key: “volttron 1. *” or Federation Shovel Key: “volttron 2. *” Topic Exchange Message: Key: “volttron 1. agent_a” AMPQ: Type: ”rpc _reply” 22
VOLTTRON - 1 PUBSUB Multiplatform Agent_A VIP ID: “agent_a” Agent_B VIP ID: “agent_b” Agent_B VIP queue Binding Key: “volttron 1. agent_a” Agent_B VIP queue Binding Key: “volttron 1. agent_b” Message: Key: “__pubsub_ _. volttron 1. devices. pnl. i sb 1. hvac 1” Topic Exchange VOLTTRON - 2 Agent_B pubsub queue Binding Key: “__pubsub__. *. devices. #” Federation Link Agent_B VIP ID: “agent_b” Agent_B VIP queue Binding Key: “volttron 2. agent_b” Topic Exchange 23
Authentication with Rabbit. MQ message bus Rabbit. MQ supports multiple authentication mechanisms For VOLTTRON we use SSL peer verification using with x 509 certificates SSL certificates of interest Root CA Server certificate signed by root Client certificate signed by root 24
Rabbit. MQ Server SSL certificates Rabbit. MQ Server 1. Root CA File - CA 1 public key 2. Server cert signed by Root CA 3. Server key Presents agent 1. crt and agent 1. pem Agent 1 agent 1. crt – public cert signed by Root CA agent 1. pem – private key Presents agent 2. crt and agent 2. pem Agent 2 agent 2. crt – signed by Root CA agent 2. pem – private key Presents agent 3. crt and agent 3. pem Agent 3 agent 3. crt – signed by Root CA agent 3. pem – private key 25
Multi-Platform Connection with SSL certificates VOLTTRON 1 VOLTTRON 2 Rabbitmq Server - 1 1. CA cert file – contains CA 1 public key and CA 2 public key 2. Server cert signed by CA 1 3. Server private key Rabbitmq Server - 2 1. CA cert file – contains public key of CA 2 public key and CA 1 public key 2. Server cert signed by CA 2 3. Server private key Shovel Presents volttron 2 user cert, key Presents agent 1. crt and agent 1. pem Presents agent 2. crt and agent 2. pem Federation Agent 1 agent 1. crt – signed by Root CA 1 agent 1. pem – private key Presents volttron 2 user cert, key Agent 2 agent 2. crt – signed by Root CA 2 agent 2. pem – private key 26
Authorization in Rabbit. MQ message bus • For protected topics, topic permissions are set using Rabbit. MQ’s management interface based on agent’s user id • Allow publish rights – set read + write permission on the topic for the agent • Allow subscribe rights – Restrict read permission on the topic for the agent • Capabilities on RPC methods – Remains unchanged • Specify required capabilities inside agent’s code @RPC. allow(‘SET_TEMP’) @RPC. export def set_temperature(): …. • Authorization entries (specified via volttron-ctl auth commands) 27
VOLTTRON Rabbit. MQ Management Utility • Extend “volttron-ctl” utility to include Rabbit. MQ management commands. • Uses Rabbit. MQ management http plugin • Some of the functions • • Create/Delete vhosts for each platform Create/Delete unique user, password for each agent Set permissions on the user Create/Delete exchanges and queues Create/Delete federation and shovel setup for multi-platform deployment. Set topic permissions for protected topics List the status of • Open Connections • Exchanges • Queues 28
Scalability Tests Single instance can support close to 1000 agents Tested with 100 instances forwarding messages to single root node connected together Latest Code in experiment branch: https: //github. com/VOLTTRON/volttron/tree/rabbitmqvolttron 29
Next Steps Integrate Volttron Central agent to use Rabbit. MQ message bus. Further improvements to installation steps and SSL based authentication. Scalability tests with large scale deployments. Complete testing of Rabbit. MQ VOLTTRON. 30
Conclusion Questions? Suggestions? Let us know if you would like to be in the message bus refactor working group: volttron@pnnl. gov 31
Backup 32
Proxy Router Agent acting as bridge Platform Agent Core ZMQ Router Subsystems RMQ XYZ ZMQ RMQ/XYZ ZMQ/XYZ Message Translation layer ZMQ RMQ ZMQ Message Translation layer XYZ 33 Proxy Agents Platform Subsystems RMQ XYZ ZMQ/RMQ Agent Core RMQ XYZ Instance V 2 runs on ZMQ Router ZMQ RMQ Instance V 1 runs on RMQ XYZ
- Slides: 33