CMW LHCera controls middleware CMW Controls Middleware from

CMW – LHC-era controls middleware CMW – Controls Middleware from BE-CO DAQ@LHC workshop, 13 th April 2016 Wojciech Sliwinski for the BE-CO Middleware team: Joel Lauener, Felix Medina Ch. Wojciech Zadlo, Vitaliy Rapp (GSI)

Agenda • What is Middleware? • Evolution of middleware technologies • CMW – Controls Middleware from BE-CO • Conclusions CMW - LHC-era controls middleware 2

About the speaker • Full-time Software Engineer • Member of Beams dep. , Controls group (BE-CO) SRC (Software for Real-time and Communication) section Leading Middleware team (5 people) • • 2 Staff, 2 Fellows, 1 External (GSI) Before: • • Fellow in IT-CS (LANDB database & tools) Software Developer in ABB Research Center (Poland) Tech. Student in FAP-AIS (EDH project) Trainee in Rockwell Automation (Switzerland) email: Wojciech. Sliwinski@cern. ch / w. s@cern. ch CMW - LHC-era controls middleware 3

Why Middleware? • How do we build/connect applications over a network? • How do we facilitate Distributed Computing (2 -tier, …, n-tier)? • How to support the heterogeneous environments? CMW - LHC-era controls middleware 4

What is Middleware? • Software, which allows an application to interoperate with other software, without requiring the user to understand to code the low-level operations required to achieve interoperability • • • Software layer between OS and the applications Hides complexity & heterogenity of distributed system Handles issues related to OS, network protocols & hardware platforms Application Standard API Middleware Specific API Operating System 1 Application Middleware Specific API Application Standard API Middleware Specific API Operating System 2 Operating System. N Communication system (e. g. network, bus) CMW - LHC-era controls middleware 5

Aspects of Middleware • Middleware provides support for: • • • Naming, Location, Service discovery, Replication Protocol handling, Communication faults, Qo. S Synchronisation, Concurrency, Failover, Scalability Access control, Authentication Middleware dimensions: • • • Request-Reply Language-specific Proprietary Small-scale Tightly-coupled vs. Asynchronous Messaging Language-independent Standards-based Large-scale Loosely-coupled components CMW - LHC-era controls middleware 6

Middleware – large domain … which to choose? Network Embedded LAN Request/R eply Publish/Sub scribe RPC/ORB Direct Peer-to-Peer WAN/ Internet Message Oriented Broker CMW - LHC-era controls middleware 7

Evolution of middleware technologies 2016 1991 ? ? ? CORBA RPC 1983 T E N BSD sockets PA AR T CP U DP IP v 4 1969 1974 1980 1981 IP v 6 1998 CMW - LHC-era controls middleware TODAY 2016 8

Dynamically evolving domain … Redis Peer-to-Peer RTI DDS Core. DX Brokers Open. AMQ Active. MQ Apollo Kafka Rabbit. MQ Hornet. Q Joram. MQ Thrift Thing. MQ MQtt RSMB Zero. C Ice Hive. MQ Mosquito Mosca Solace Open. Splice. DDS nanomsg Jac. ORB Zero. MQ omni. ORB CMW - LHC-era controls middleware 9

CMW – Controls Middleware from BE-CO CMW - LHC-era controls middleware 10

Our clients: CERN Accelerators CMW - LHC-era controls middleware 11

Controls Software Architecture C++OPERATO R CONSOLES FIXED OPERATO Presentation Layer DISPLAY R S Java CONSOLE S Controls Middleware (CMW), RMI, JMS TCP/IP communication FILE SERVER S APPLICATIO N SERVERS SCADA SERVER S C++ Business Layer RT Lynx/OS VME Front Ends TIMING GENERATIO N WORLDFI P Front Ends TCP/IP communication services T T T CMW Purpose: Reliable and scalable transport of data between controls processes (Java & C++) T TCP/IP communication services BEAM POSITION MONITORS, BEAM LOSS MONITORS, BEAM INTERLOCKS, RF SYSTEMS, ETC… T T T QUENCH PROTECTION AGENTS, POWER CONVERTERS FUNCTIONS GENERATORS, … C++ FIP/IO Front End (Back End) Layer T T CTRL Java PROFIBUS OPTICAL FIBERS C++ services Controls Middleware (CMW) PLC World. FIP SEGMENT CERN GIGABIT ETHERNET TECHNICAL NETWORK Communication ETHERN ET NETWOR K ACTUATORS AND SENSORS CRYOGENICS, VACUUM, ETC… CMW - LHC-era controls middleware 4’ 000 CMW servers 85’ 000 devices 2’ 000 IO-points Uses Zero. MQ & CORBA 12

CMW mandate & scope • Core communication layer -> critical • • • Centrally managed middleware services • • Directory/Naming, RBAC, Proxies, DIP Gateways, … Access syntax -> Device-Property model • • • Reliable communication in distributed system Decentralized (no brokers, etc. ) -> scalable Device: addressable IO point (e. g. channel, slot, module) Device: e. g. LHC. BPM. P 5. B 1, LHC. BPM. P 5. B 2 Property: exposed operation or method Property: e. g. Acquisition, Status, Alarm, Setting Widely deployed for all CERN accelerators • Used in all Eqp. groups (3 deps: BE, EN, TE) CMW - LHC-era controls middleware 13

CMW - A bit of history • • • 1988 – RPC-based middleware 2002 – CORBA-based version of RDA (Remote Device Access) Problems observed with CORBA: • • • 2011 – review & selection of Zero. MQ as replacement for CORBA • • • Technical obsolescence & shrinking community Lack of asynchronous transport & comm. patterns (only RPC) Lack of handling of “slow clients” (had to implement workarounds) Poor scalability for subscriptions & for many clients (>200) See paper: http: //accelconf. web. cern. ch/Accel. Conf/icalepcs 2011/papers/frbhmult 05. p df (or: http: //cern. ch/go/G 9 RC) 2012 -2013 – design & implementation of RDA 3 (based on Zero. MQ) Spring 2014 – operational deployment of RDA 3 CMW - LHC-era controls middleware 14

RDA 3 – CMW core communication library • Uses Zero. MQ for low-level networking • Built upon lock-free, async (event driven) architecture • Operations/Calls: Get, Subscribe • All communication implicitly asynchronous • Public API for developing clients & servers • Provided for several platforms • • Integrated with several frameworks • • FESA, FGCD, Win. CCOA, Lab. VIEW, C 2 MON Provides comprehensive diagnostics • • C++ (C++98) & Java (Java 7) Dependencies: Zero. MQ, Boost (C++) Linux (32/64 -bit: SLC 5, SLC 6), Windows (64 -bit) Admin GUI, command line tools, Kibana reports Exportable -> used in GSI CMW - LHC-era controls middleware 15

Data flow for RDA 3 calls Get & Set calls Subscribe calls RDA 3 client Set d Bin [1] lish Pub [0] Load devices & servers [2 & 2’] Save Device 2/Property. B RDA 3 server Device 11/Property. A Device 22/Property. B … Device. N/Property. X [5’] Device 1/Property. A Cache (Devices & Servers) ind RDA 3 server ’] B [1 Directory Service e crib ubs [4] G et / L S [4’] p ku oo ]L ’] [3 Zero. MQ p ku o o [3 Zero. MQ … Database Zero. MQ Device. NN/Property. X Zero. MQ CMW - LHC-era controls middleware 16

Example – sync Get call #include <iostream> #include <cmw-rda 3/client/service/Client. Service. Builder. h>. . . // First create Client. Service and keep it for all further communication auto_ptr<Client. Service> client = Client. Service. Builder: : new. Instance()->build(); // Get Access. Point by providing device name & property name Access. Point & access. Point = client->get. Access. Point(”LHC. BPM. P 5. B 1", ”Acquisition"); // Perform sync GET call auto_ptr<Acquired. Data> acq. Data = access. Point. get(); // The resulting data is obtained by calling get. Data() method on the Acquired. Data instance const Data & data = acq. Data->get. Data(); cout << data. to. String() << endl; // Data -> supports scalars, arrays of scalars(1 D, 2 D, N-D), structures (nested Data) // Additionally Acquired. Data contains also Acquired. Context, which provides meta-data about the call const Acquired. Context & acq. Context = acq. Data->get. Context(); cout << acq. Context. to. String() << endl; CMW - LHC-era controls middleware 17

Example – async Get call with callback #include <iostream> #include <cmw-rda 3/client/service/Client. Service. Builder. h>. . . class Callback : public Async. Get. Callback { public: void request. Completed(const Request. Handle & request, auto_ptr<Acquired. Data> acq. Data) { cout << acq. Data->get. Data(). to. String() << endl; } void request. Failed(const Request. Handle & request, auto_ptr<Rda. Exception> exception) { cout << exception->what() << endl; } }; auto_ptr<Client. Service> client = Client. Service. Builder: : new. Instance()->build(); Access. Point & access. Point = client->get. Access. Point(”LHC. BPM. P 5. B 1", ”Acquisition"); // Perform async GET call with callback access. Point. get. Async(Async. Get. Callback. Shared. Ptr(new Callback())); // Continue while GET is being performed. . . CMW - LHC-era controls middleware 18

Example – async Get call with future #include <iostream> #include <cmw-rda 3/client/service/Client. Service. Builder. h>. . . auto_ptr<Client. Service> client = Client. Service. Builder: : new. Instance()->build(); Access. Point & access. Point = client->get. Access. Point(”LHC. BPM. P 5. B 1", ”Acquisition"); // Perform async GET call with future Rda. Get. Future. Shared. Ptr future = access. Point. get. Async(); … // Continue processing while GET is being performed. . . cout << “Is GET completed: ” << future->is. Done() << endl; . . . // Perform blocking call until GET data is ready auto_ptr<Acquired. Data> data = future->get(); cout << data->get. Data(). to. String() << endl; CMW - LHC-era controls middleware 19

Example code – sync Set call #include <iostream> #include <cmw-rda 3/client/service/Client. Service. Builder. h>. . . // First create Client. Service and keep it for all further communication auto_ptr<Client. Service> client = Client. Service. Builder: : new. Instance()->build(); // Get Access. Point by providing device name & property name Access. Point & access. Point = client->get. Access. Point(”LHC. BPM. P 5. B 1", ”Setting"); // Prepare data to be sent to server // Data -> supports scalars, arrays of scalars(1 D, 2 D, N-D), structures (nested Data) auto_ptr<Data> data = Data. Factory: : create. Data(); data->append("value", 10); data->append(”gain", 5. 1234); // Perform sync SET call access. Point. set(data); Analogically async Set versions with callback & future CMW - LHC-era controls middleware 20

Example – Subscribe call #include <iostream> #include <cmw-rda 3/client/service/Client. Service. Builder. h>. . . auto_ptr<Client. Service> client = Client. Service. Builder: : new. Instance()->build(); Access. Point & access. Point = client->get. Access. Point(”LHC. BPM. P 5. B 1", ”Status"); // Perform SUBSCRIBE call Subscription. Queue. Shared. Ptr queue = access. Point. subscribe(); . . . // Continue processing while subscription is being established. . . // Blocking wait for a single notification with a timeout of 5000 ms std: : auto_ptr<Notification> notification = queue->poll(5000); // also with no timeout: poll() cout << “Received notifications count: ” << queue->get. Queue. Size() << endl; // Process the received data auto_ptr<Acquired. Data> acq. Data = notification->get(); cout << acq. Data->get. Data(). to. String() << endl; Analogically Subscribe version with callback CMW - LHC-era controls middleware 21

Latency stability measured by BE-CO Timing team client # latency Setup: 500 B message payload, CCC machines, 1 server/publisher, 1 -1100 clients/subscribers latency local time client # CMW - LHC-era controls middleware 22

Observations from operational use of RDA 3 • • Great performance for subscriptions (thanks to async transport & internal batching) Good scalability for many clients (>1000) • No more problem of slow clients (solved properly thanks to async transport) • Much smaller footprint (CPU & memory), mainly thanks to zero-copy • Zero. MQ - no surprises & very reliable! RDA 2 framework RDA 3 framework RDA 2 client RDA 3 client CORBA Zero. MQ TCP/IP CORBA Zero. MQ RDA 2 server RDA 3 server CMW - LHC-era controls middleware 23

CMW ”mixed” infrastructure today Old applications, which are not yet upgraded to RDA 3 New RDA 3 client can connect to both: RDA 3 & RDA 2 Proxy is setup on request and it converts RDA 2 ->RDA 3 RDA 2 client CORBA RDA 3 client Zero. MQ CORBA RDA 2 ->RDA 3 Proxy RD A 3 RDA 3 2 RDA 2 End-of-Life is mid-LS 2 RDA 2 server RDA 3 server CORBA Zero. MQ CMW - LHC-era controls middleware 24

Future plans • Q 4 2016 – provide REST API to access RDA devices • • • Q 1/Q 2 2017 – provide Python binding for RDA • • Via dedicated REST->RDA gateways Requested by web developers & scripting community On top of REST API, no dependency on RDA Q 3 2017 – new major version of Directory Service CMW - LHC-era controls middleware 25

Resources • • • CMW Wikis: https: //wikis. cern. ch/display/MW/ RDA 3 User Guide: https: //wikis. cern. ch/display/MW/RDA 3+User+Guide RDA 3 Java: https: //svnweb. cern. ch/cern/wsvn/acc-co/trunk/cmwrda 3/ RDA 3 C++: https: //svnweb. cern. ch/cern/wsvn/acc-co/trunk/cmwrda 3 -cpp/ Middleware Review: http: //accelconf. web. cern. ch/Accel. Conf/icalepcs 2011/papers/frbhmult 0 5. pdf (or: http: //cern. ch/go/G 9 RC) Emails: • • • cmw-support@cern. ch (questions, requests, etc. ) cmw-news@cern. ch (news from MW team, subscribe via e-groups) Wojciech. Sliwinski@cern. ch (or: w. s@cern. ch) CMW - LHC-era controls middleware 26

Conclusions • Middleware is complex & challenging domain • Many solutions exist -> understand your system and it’s requirements before picking-up a product • Modular design & generic, narrow public API should allow for future evolution • Do you need to stay backward-compatible? • What about security? Yes, it has to be provided • RDA 3 is reliable & generic –> can be used elsewhere • Zero. MQ proved to be stable & very efficient CMW - LHC-era controls middleware 27

CMW - LHC-era controls middleware 28

Backup slides CMW - LHC-era controls middleware 29

CERN Middleware Requirements Lightweight Active community Desirable Friendly API, documentation Request/reply & pub/sub patterns Qo. S Asynchronous Mandatory Performance & Scalability Stability, Maturity & Longevity Open source, redistributable license C++/Java Fundamental Linux/Windows Over TCP/IP LAN 14 th October 2011 30

How did we evaluate –> our criteria Appearance Simple usage • Creators • specification • documentation • Users • forums • bug reports • Internet Testing • Download • licensing • Compile • Lynx. OS & gcc 2. 95 • Run examples • Communication patterns • Performance • Qo. S • Exceptional situations CRITERIA API, look & feel, documentation 14 th October 2011 resources, binary size, memory Community, Communications maturity patterns Qo. S Andrzej Dworak, Middleware Trends and Market Leaders 2011 performance 31

Evaluated middleware products All opinions are based only on our knowledge and evaluation. Each of the products, depending on the requirements, may constitute a good solution. Core. DX Open. AMQ RTI DDS Zero. MQ Open. Splice. DDS Rabbit. MQ Ice YAMI omni. ORB Jac. ORB QPid Thrift CMW - LHC-era controls middleware MQtt RSMB Mosquito 32

Sync, async & msg patterns Qo. S Dependencies & memory f-p Performance Look & feel, API, docs Community & maturity Score Products comparison (according to the criteria) Zero. MQ 6 Ice 5 YAMI 4 4 RTI 3 Qpid 3 CORBA 2 Thrift 2 14 th October 2011 33

RDA 3 Java – Sync Get round-trip time June’ 13 Test setup: 1 k. B message payload, cs-ccr-* machines, 1 server host & 10 client hosts 34

RDA 3 Java – Subscription notification latency June’ 13 Test setup: 1 k. B message payload, cs-ccr-* machines, 1 server host & 10 client hosts CMW overview & architecture 35

RDA 3 Java – Subscription notification latency June’ 13 Test setup: 1 k. B message payload, cs-ccr-* machines, 1 server host & 10 client hosts CMW overview & architecture 36