Felix Ehm CERN BECO JMS HIGH LEVEL MESSAGING

Felix Ehm CERN BE-CO JMS – HIGH LEVEL MESSAGING

Content � Introduction � JMS in the Controls System � Deployment � Conclusion and Operation

Introduction

Introduction to JMS – The API � Java Messaging Service �Messaging API defined in 2002 �Decouples Source and Destination �Topic / Queue Concepts for named Destinations

Introduction to JMS – A Broker � Technology originated from the financial sector �“Few” sources and large set of readers � A Broker is the central entity �Accepts messages from 1. . * publisher and Broker re-distributes to 0. . * subscribers �Provides Quality of Service for Messages ○ Persistency, Transactions, Buffering, Expiration Policies, Slow Client Handling �Design to distribute data reliably and scalable �Load balancing / fail-over / embedded deployment scenarios possible

Introduction to JMS - A Broker � Many broker implementations available �Proprietary IBM Websphere. MQ, Sonic. MQ, TIBCO, Red. Hat MRG, … �Open. Source Apache Active. MQ, Apache Apollo, Rabbit. MQ, Hornet. MQ, … � Broker may be implemented in another language than Java (e. g. Qpid & Rabbit. MQ) � Communication protocol is non-standard �Brokers of different vendors are not interoperable �AMQP & STOMP address this issue

Introduction to JMS Request/Reply � “Direct communication” via Broker using Request – Reply mechanism Requestor sends creates Broker Replier Request Queue Temporary Reply Queue � Request MSG Reply MSG sends Due to intermediate Broker two hops are required

Introduction to JMS - Selectors � Filter �Each message can be enriched CYCLE=A TIME=9845 with Attributes �Subscriber receives filtered messages via defined selector Broker BUILDING=A BUILDING=B BUILDING=C Topic/ Queue BODY selector: BUILDING ≠ A Subscriber JMS Message messages from one Destination using Messages Selectors HEADERS

Introduction to JMS Persistency � JMS defines persistent and non-persistent messages 1. send � Broker stores message to Sender disk before acknowledging 2. store 3. ack to sender � Avoids loss of data when broker crashes � Drawbacks �Higher disk load �Slower message processing

JMS in the Controls System

CERN’s Accelerator Complex

JMS in the Controls System � Purpose �Reliable transport of data between (Java) Processes �High level controls applications + middle tier servers ○ Alarm System ○ Data Rendering Services ○ (Control) GUIs ○ Logging Services ○ Beam Security System(s) ○ Access Systems, …. No JMS - No Beam ! Today vital part of Beam Operations � Product choice: �

JMS in the Controls System History �Sonic. MQ since 2001 for CERN’s Alarm and CO Monitoring system �Open. Source Active. MQ since 2005 ○ Middle tier servers update GUIs ○ Middle tier to Middle tier Phased out in 2013

JMS for Accelerator Controls � Early days only Java clients via JAPC �japc-ext-remote contains JMS communication � Today multi-language clients �JAPC clients remain �Java using Active. MQ client libraries directly �C++/Python using STOMP

Deployment and Operation

Deployment and Operation � 29 production brokers � Single or broker cluster � Large spectrum of usage patterns �from few topics to 40 K topics �from few subscribers to >23 K subscribers �from couple msg / minute to 10 K msg / sec �from few bytes/msg to 3 Mbyte/msg

Deployment and Operation Some numbers : � 300 Applications � 4’ 400 Connections � 40’ 000 Subscriptions � 85’ 000 Topics � 8 Million msg/h IN, 3. 5 M msg/h OUT Archived Uptime in 2012: 99. 998% Not all data which is produced is also consumed

Deployment and Operation � Distribute Load �Interconnect two Brokers �Consumers/Producer can choose any Broker �Sharing of work � Smooth Upgrades possible �Client library version != Broker version OK � Drawbacks �Consumer might not get all messages in case of network error �Routing overhead

The STOMProtocol

The STOMProtocol � Simple Text Oriented Message Protocol � Defines messaging API + wire format � Many open source clients libraries available �Python, C/C++, Java, Perl, Ruby, PHP, Erlang, … � Targets the messaging interoperability among language, platforms and brokers � Supported by many brokers � Very easy to use

Example to send STOMP in Python mysocket = socket(socket. AF_INET, socket. SOCK_STREAM) mysocket. connect( ("jms-co-dev", 61680) ) text = '''SENDn destination: /queue/CERN. DEPLOYMENTn HOST: %sn INSTALLPATH: %sn PRODUCT: %sn TIME: %sn persistent: falsennx 00''‘ mysocket. sendall(text) mysocket. sendall("DISCONNECTx 00") mysocket. close()

The AMQProtocol

The AMQProtocol � Advanced Message Queue Protocol � Defines messaging API + wire format � Next evolution after JMS �Consortium of Red. Hat, Credit Suisse, Microsoft, JPMorgen, Barclays, VMWare, Cisco, … � Released as Version 1. 0 �More and more brokers implement it � Direction for the future ?

Conclusion � JMS is vital part of Accelerator Controls System � Performs very well for publish - subscribe scenarios � Active. MQ was a good choice � Running reliably � Low maintenance and flexible deployments � Provides many metrics for diagnostic and monitoring � Multi-language applications can use service � Central (redundant) service

Result Example received/sec over time AVG Messages / sec � Messages time

Result Example of delayed message distribution # Messages � Histogram Delay time [msec]

Result Example � Number of “late” (>300 ms) messages for test Delayed Messages duration time

Message Delay Summary � Measuring delayed Message �~ 99. 32% of messages delivered < 300 ms �Rest < 1 sec � Doubling numbers of messages (~9500 msg/sec) � 100% between 5 and 10 seconds �Further studies for improvement needed

Result Example � CPU load Consumer disconnects % CPU load Consumer connects time 40% less CPU load thanks to optimization

Broker Memory Consumption � Initial Memory is proportional to �Connections �Subscriptions 1000 Subscription / Connection 500 Subscriptions / Connection 400 800 350 700 300 600 250 500 200 400 150 300 100 200 50 100 0 5000 10000 Heap Memory Usage 20000 30000 Live Threads 40000 0 10000 20000 Heap Memory Usage 40000 60000 Live Threads 80000