Karabo The European XFEL software framework Design Concepts

Karabo: The European XFEL software framework Design Concepts Burkhard Heisen for CAS December 2014 The star marks concepts, which are not yet implemented in the current release

Karabo: The European XFEL software framework Functional requirements 2 A typical use case: Control m Sa drive hardware and complex experiments monitor variables & trigger alarms ple r to jec In DAQ data readout online processing quality monitoring (vetoing) allow some control & show hardware status Accelerator Undulator Beam Transport show online data whilst running DM DAQ DM Control SC Tight integration of applications storage of experiment & control data access, authentication authorization etc. setup computation & show scientific results SC processing pipelines distributed and GPU computing specific algorithms (e. g. reconstruction)

Karabo: The European XFEL software framework Functionality: What are we dealing with? 1. Data containers (transport and storage through serialization) 2. Data transport (communication patterns) 3. Devices (distributed end points) 4. States and state machines (when can what be called/assigned on the devices) 5. Log Messages (active, passive, central, local) 6. (Slow) Control-Data Logging 7. (Fast) Data acquisition 8. Time synchronization/tagging (time stamps, cycle ids, etc. ) 9. Real-time needs (where necessary) 10. Notifications and Alarms 11. Security (who’s allowed to do what from where? ) 12. Statistics (control system itself, operation, …) 13. Processing workflows (parallelism, pipeline execution, provenance) 14. Clients / User interfaces (API, languages, macro writing, CLI, GUI) 15. Experiment, Run and Configuration management 16. Software management (coding, building, packaging, deployment, versioning, …) 3

Karabo: The European XFEL software framework Data containers n Some special data containers are provided by Karabo and are exposed in the API § Hash String-key, any-value associative container Keeps insertion order (iteration possible), hash performance for random lookup Provide (string-key, any-value) attributes per hash-key Fully recursive structure (i. e. Hashes of Hashes) Serialization: XML, Binary, HDF 5 Usage: configuration, device-state cache, DB-interface, message protocol, etc. § Schema Describes possible/allowed structures for the Hash. In analogy: Schema would be for Hash, what an XSD document is for an XML file Internally uses Hash § Raw. Image. Data Specialized class for transporting image-like data Easily convertible to numpy in Python and to Cpu. Image<T> in C++ Optimized serialization into HDF 5 Internally uses Hash 4

Karabo: The European XFEL software framework STATUS: Data containers n Recent changes § n n None Future work § Improve Hash serialization implementation with respect to the XML format to allow for slashes “/” in hash-keys § Find a proper data object (eventually plus some description) to exchange data with the DAQ layer Open issues § Understand the conceptual difference between using standardized objects vs. generic container + description throughout the system (see also DAQ section) 5

Karabo: The European XFEL software framework Data transport – Message broker based n 6 Basic communication between objects is established via a central message broker using a publish-subscribe pattern (topic based) § Each communicating object is an instance of the Signal. Slotable class which connects to a configurable broker (host/port/topic) § The Signal. Slotable API allows to register regular functions of any signature (currently up to 4 arguments) to be remotely callable (such a function is called: Slot) § Slots can be uniquely addressed by a pair of strings, the instance. Id (string name of the Signal. Slotable object) and the function. Name (string name of the function) § Slot registration can be done during construction or later at runtime without extra tools § Slot calls can be done cross-network, cross-operating-system and crosslanguage (currently C++ and Python) § The language’s native data types are directly supported as arguments § Additionally supported arguments are Karabo’s data objects (e. g. Hash and Schema) § Data packets are on the fly compressed/decompressed if reaching some size threshold New

Karabo: The European XFEL software framework DETAIL: Data transport Broker based communication API – Four Patterns ① Signals & Slots § § SLOT ( function, [arg. Types] ) SIGNAL ( func. Name, [arg. Types] ) connect ( signal. Instance. Id, signal. Func, slot. Instance. Name, slot. Func ) emit ( signal. Func, [args] ) SLOT(on. Foo, int, std: : string); void on. Foo(const int i, std: : string& s) { } SIGNAL(“foo”, int, std: : string); connect(“Device 1”, “foo”, “Device 2”, “on. Foo”); connect(“”, “foo”, “Device 3”, “on. Goo”); connect(“”, “foo”, “Device 4”, “on. Hoo”); emit(“foo”, 42, “bar”); SLOT(on. Goo, int, std: : string); void on. Goo(const int i) { } Device 2 Notify Device 1 Emit Device 3 Notify Device 4 SLOT(on. Hoo, int, std: : string); void on. Hoo(const int i, std: : string& s) { } 7

Karabo: The European XFEL software framework DETAIL: Data transport Broker based communication API – Four Patterns ② Direct Call 8 SLOT(on. Foo, std: : string); void on. Foo(const std: : string& s) { } call(“Device 2”, “on. Foo”, “bar”); § call ( instance. Id, func. Name, [args] ) Call Device 1 Notify Device 2 ③ Request / Reply § request ( instance. Id, func. Name, [req. Args] ). timeout( msec ). receive( [rep. Args] ) int number; request(“Device 2”, “on. Foo”, 21). timeout(100). receive(number); Request SLOT(on. Foo, int); void on. Foo(const int i) { reply( i + i ); } Notify Device 2 Device 1 Notify Reply

Karabo: The European XFEL software framework DETAIL: Data transport Broker based communication API – Four Patterns 9 ④ Asynchronous Request / Reply § request. No. Wait ( req_instance. Id, req_func. Name, rec_instance. Id, rec_func. Name, [req. Args] ) SLOT(on. Foo, int); void on. Foo(const int i) { reply( i + i ); } request. No. Wait(“Device 2”, “on. Foo”, “on. Bar”, 21); Request Notify Device 2 Device 1 Notify SLOT(on. Bar, int); on. Bar(const int i) { … } Reply New

Karabo: The European XFEL software framework STATUS: Broker communication n Recent changes § Fundamental change of how messages are consumed § n n Before: Each Slot presented an own consumer on the broker, forcing the broker to route (using “Selectors”) messages by selecting on instance. Id and function. Name. Larger installation caused huge number of consumer clients on the broker and needed a thread per Slot on the device Now: Each object is a consumer on the broker, routing is done only utilizing the instance. Id only. Slot selection is done on the client side. Using an own queuing system on the Signal. Slotable the number of threads used per instance is decoupled from the number of slots (can be single threaded context) Heartbeats get first priority (using own topic and by placing on front of queue) Future work § Performance and scalability tests § Check whether trouble with heartbeats is finally solved Open issues 10

Karabo: The European XFEL software framework Data transport – P 2 P n Another fundamental communication pattern between objects is realized by connecting so-called input and output channels to form a direct point-to-point connection (shortcutting the broker) § Unlike slots (which are functions) input and output channels are named objects with a read/write/update API § The Signal. Slotable API allows to create one or more such channels per Signal. Slotable instance § Technically output channels are (multi-client capable) TCP servers, input channels are clients § The connection between them is established by referring to instance. Id and channel. Name instead of host and port. § Host and port are transparently communicated during connection time using the broker based communication § Channels are highly configurable and are intended to serve the need of flexible streaming data pipeline setups 11

Karabo: The European XFEL software framework DETAIL: Data transport P 2 P communication 12 n One channel is specific for one data object (e. g. Hash, Image, Byte-Array) n For input and output channels within the same application data exchange will happen by handing over pointers in memory instead of transmitting via TCP n Users can register two function call-backs indicating availability of data (e. g. on. Data) and (optionally) the end of the data stream (e. g. on. End. Of. Stream) on the input channel n Input channels configure whether they share the sent data with all input channels connected to the same output or whether they receive a copy of each data token n Output channels may specify a special hostname (in case of multiple adapters) to which the clients are routed to New Message Broker l ntro co P 2 P Data […]

Karabo: The European XFEL software framework STATUS: P 2 P communication n Recent changes § n n Added possibility to select the interface on which to communicate Future work § More performance and scalability tests § Whilst reading is already asynchronous to the users-code execution (IO during processing), for writing this is currently not true (was implemented but removed for instability issues) § Asynchronous must again be implemented for performance improvement Open issues § Think carefully whether this communication could also be used (performance issue) for transporting fast DAQ data from Device to DAQ-Layer 13

Karabo: The European XFEL software framework Devices (distributed end points) n The distributed end points follow the “Device Server Model” § Similar to: TANGO or DOOCS § End points are controllable objects managed by a device server § Instance of such an object is a Device, with a hierarchical name § Device classes can be loaded at runtime (plugins) § Devices inherit Signal. Slotable and wrap the communication API into a simpler subset § Actions pertaining to a device given by its properties, commands, and channels i. e. get, set, monitor some property or execute some command write/read some data to/from a channel and update when done § Properties, commands and channels are statically described (expected. Parameters function) and further described via attributes in the device class. This description is saved in form of a Schema. Dynamic (runtime) extension (Schema injection) of expected. Parameters is possible. § Devices can be written in either C++ or Python 14

Karabo: The European XFEL software framework DETAIL: Devices Configuration - API n Any Device uses a standardized API to describe itself. This information is shipped as Schema object and used by interested clients (GUI, CLI other devices) n We distinguish between properties and commands and associated attributes, all of them can be expressed within the expected parameters function need for device developers to validate any parameters. This is internally done taking the expected. Parameters as white-list 15 Class: Motor. Device static expected. Parameters( Schema& s ) { FLOAT_ELEMENT(s). key(“velocity”). description(“Velocity of the motor”). assignment. Optional(). default. Value(0. 3). max. Inc(10). min. Inc(0. 01) Attribute. reconfigurable(). allowed. States(“Idle”). commit(); INT 32_ELEMENT(s). key(“current. Position”). description = “Current position of the motor”. read. Only(). warn. Low(10) […] Command n No n Properties and commands can be nested, such that hierarchical groupings are possible Property SLOT_ELEMENT(s). key(“move”). description = “Will move motor to target position”. allowed. States(“Idle”) […] } // Constructor with initial configuration Motor. Device( const Hash& config ) { […] } // Called at each (re-)configuration request on. Reconfigure( const Hash& config ) { […] }

Karabo: The European XFEL software framework DETAIL: Devices Creating a new device 1. Write a class (say: My. Device) that derives from Device 2. lib. My Device. so Compile it into a shared library (say lib. My. Device. so) 3. 16 Select a running Device-Server or start a plugins signal. New. Device. Class. Available (. xsd) new one 4. Copy the lib. My. Device. so to the plugins folder of the Device-Server 5. The Device-Server will emit a signal to the broker that a new Device class is GUI-Srv available, it ships the expected parameters as read from static context of the My. Device class GUI

Karabo: The European XFEL software framework DETAIL: Devices Creating a new device 6. Given the mask of possible parameters the 17 factory: create(“My. Device”, xml) user may fill a valid configuration and emit an instantiate signal to the broker 7. My. Device 1 The configuration will be validated by the plugins Device factory and if valid, an instance of My. Device will be created 8. The constructor of the device class will be called and provided with the configuration 9. signal. Instantiate(“My. Device”, xml) The run method will be called which starts the state-machine and finally blocks by GUI-Srv activating the event-loop 10. The device will asynchronously listen to allowed events (slots) GUI

Karabo: The European XFEL software framework Device “flavors” Equipment Control e. g. motor, pump, valve, sensor Composite Device DAQ Equipment without Data DAQ Equipment with Data e. g. commercial camera PCLayer Node Service Device e. g. digitizer, beam position monitor, 2 D-detectors e. g. calibration. Manager, project. Manager, broker. Monitor Workflow Node

Karabo: The European XFEL software framework DETAIL: Devices taking part in distributed system Device Instance HV Device-Server Application 19 Digitizer Pump Message Broker (Event Loop) Store Camera Disk Storage Load Calibrate 1 Simulate Terminal(s) Calibrate 2 Logger GUI Server GUI(s)

Karabo: The European XFEL software framework STATUS: Devices n Recent changes n Future work n § Best practices and all concepts for hierarchical device structures must be defined § The composed-in Device. Client API needs more functionality to make composition easier Open issues 20

Karabo: The European XFEL software framework States and state machines n n Any property setting or command execution on a Device can be restricted to a set of allowed states (using the allowed. States attribute) The state of a device can be changed by simply setting the state property (string) to the desired value New The GUI is state and allowed states aware and enables/disables buttons and properties pro-actively Devices may optionally implement a finite state machine (FSM) following the UML standard § In this case an incoming slot call is not directly implemented but triggers and event into the state machine § User defined hooks are executed as consequence of a start-to-finish event processing algorithm. § Possible hooks are: guard, src-state-on-exit, transitionaction, tgt-state-on-entry, on-state-action New 21 Start Stop State Machine Initialization none OK Stopped stop start Started error. Found reset Error

Karabo: The European XFEL software framework DETAIL: States and state machines Finite state machines – There is a UML standard n State Machine: the life cycle of a thing. It is made of states, transitions and processes incoming events. n State: a stage in the life cycle of a state machine. A state (like a submachine) can have an entry and exit behaviors n Event: an incident provoking (or not) a reaction of the state machine n Transition: a specification of how a state machine reacts to an event. It specifies a source state, the event triggering the transition, the target state (which will become the newly active state if the transition is triggered), guard and actions n Action: an operation executed during the triggering of the transition n Guard: a boolean operation being able to prevent the triggering of a transition which would otherwise fire n Transition Table: representation of a state machine. A state machine diagram is a graphical, but incomplete representation of the same model. A transition table, on the other hand, is a complete representation 22

Karabo: The European XFEL software framework DETAIL: States FSM implementation example in C++ (header only) // Events FSM_EVENT 2(Error. Found. Event, FSM_EVENT 0(End. Error. Event, FSM_EVENT 0(Start. Event, FSM_EVENT 0(Stop. Event, on. Exception, string) end. Error. Event) slot. Move. Start. Event ) slot. Stop. Event) // States FSM_STATE_EE(Error. State, error. State. On. Entry, error. State. On. Exit) FSM_STATE_E(Initialization. State, initialization. State. On. Entry) FSM_STATE_EE(Started. State, started. State. On. Entry, started. State. On. Exit) FSM_STATE_EE(Stopped. State, stopped. State. On. Entry , stopped. State. On. Exit) // Transition Actions FSM_ACTION 0(Start. Action, start. Action) FSM_ACTION 0(Stop. Action, stop. Action) Regular callable function (triggers event) Transition table element Regular function hook (will be call-backed) Transition table element // All. Ok. State Machine FSM_TABLE_BEGIN(All. Ok. State. Transition. Table ) // Src. State Event Tgt. State Action Guard Row< Started. State, Stop. Event, Stopped. State, Stop. Action, none >, Row< Stopped. State, Start. Event, Started. State, Start. Action, none > FSM_TABLE_END FSM_STATE_MACHINE (All. Ok. State, All. Ok. State. Transition. Table, Stopped. State, Self) // Start. Stop Machine FSM_TABLE_BEGIN( Start. Stop. Transition. Table) Row< Initialization. State , none, All. Ok. State, none >, Row< All. Ok. State, Error. Found. Event, Error. State, Error. Found. Action, none >, Row< Error. State, End. Error. Event, All. Ok. State, End. Error. Action, none > FSM_TABLE_END KARABO_FSM_STATE_MACHINE(Start. Stop. Machine, Start. Stop. Machine. Transition. Table , Initialization. State, Self) FSM_CREATE_MACHINE (Start. Stop. Machine, m_fsm); FSM_SET_CONTEXT_TOP (this, m_fsm) FSM_SET_CONTEXT_SUB (this, m_fsm, All. Ok. State) FSM_START_MACHINE (m_fsm) 23

Karabo: The European XFEL software framework DETAIL: States FSM implementation example in Python # Events FSM_EVENT 2(self, FSM_EVENT 0(self, ‘Error. Found. Event’, ‘End. Error. Event’, ‘Start. Event’, ‘Stop. Event’, # States FSM_STATE_EE(‘Error. State’, FSM_STATE_E( ‘Initialization. State’, FSM_STATE_EE(‘Started. State’, FSM_STATE_EE(‘Stopped. State’, ‘ on. Exception’) ‘ slot. End. Error’) ‘ slot. Start’) ‘ slot. Stop’) self. error. State. On. Entry, self. error. State. On. Exit ) self. initialization. State. On. Entry ) self. started. State. On. Entry, self. started. State. On. Exit) self. stopped. State. On. Entry , self. stopped. State. On. Exit ) # Transition Actions FSM_ACTION 0(‘Start. Action’, self. start. Action) FSM_ACTION 0(‘Stop. Action’, self. stop. Action) # All. Ok. State Machine all. Ok. Stt = [ # Src. State Event (‘Started. State’, ‘Start. Event’, (‘Stopped. State’, ‘Stop. Event’, ] FSM_STATE_MACHINE (‘All. Ok. State’, Tgt. State Action Guard ‘Stopped. State’, ‘Start. Action’, ‘none’), ‘Started. State’, ‘Stop. Action’, ‘none’) all. Ok. Stt, ‘Initialization. State’) # Top Machine top. Stt = [ (‘Initialization. State’, ‘none’, ‘All. Ok. State’, ‘none’), (‘All. Ok. State’, ‘Error. Found. Event’, ‘Error. State’, ‘none’), (‘Error. State’, ‘End. Error. Event’, ‘All. Ok. State’, ‘none’) ] FSM_STATE_MACHINE (‘Start. Stop. Device. Machine’ , top. Stt, ‘All. Ok. State’) self. fsm = FSM_CREATE_MACHINE(‘Start. Stop. Machine’) self. start. State. Machine() 24

Karabo: The European XFEL software framework STATUS: States n n Recent changes § A hook for performing some (periodic) action whilst being in a state was added to the FSM § A clean way of implementing devices without FSM is available (and is now recommended) Future work § n In case of no FSM: Device-side validation of command executions and property settings against allowed states attribute Open issues 25

Karabo: The European XFEL software framework Data logger n All property changes of all devices are archived centrally and in an eventdriven way § The archive can be used to debug the system at a later point § The data logger allows fast retrieval of two kinds of information: Values of a property in a selected time range (feeding e. g. trend line plots in GUI) The full configuration of a device at a given time point § By default all devices and all their properties are logged. However, entire devices or individual properties of those may be flagged to be excluded from logging § Logging is done in a per-device fashion and for any device currently 3 append able text files are generated: *_configuration. txt: Stores all changes of the device properties *_schema. txt: Stores all changes of the device schema *_index. txt: Index file for speeding up queries Changed

Karabo: The European XFEL software framework Data logger n n Any regular device has a Data. Logger_<device. Name> companion Device. B Device. A A Data. Logger. Manager composite device couples the life-time of the two companions Data. Logger Manager Data. Logger Device. A Data. Logger Device. B

Karabo: The European XFEL software framework STATUS: Data Logger n n n Recent changes § A hook for performing some (periodic) action whilst being in a state was added to the FSM § A clean way of implementing devices without FSM is available (and is now recommended) Future work § Further scaling will be done by running Data. Loggers on several hosts (connected to a shared file system) as configured via the Data. Logger. Manager § A second device will be implemented that reads the generated files and asynchronously populates a RDBMS Open issues § Is the data we log complete? Should not command executions also be part of the logged data? 28

Karabo: The European XFEL software framework Data acquisition 29 direct TCP channels via broker Data aggregation, integration & dissemination Multiple aggregator instances to handle all slow & fast data Borrowed from Djelloul Boukhelef

Karabo: The European XFEL software framework Concept thoughts: DAQ integration DAQ Equipment without Data Equipment Control via broker Data aggregation, integration & dissemination 30 DAQ Equipment direct TCP with Data channels Multiple aggregator instances to handle all slow & fast data Aggregator PCLayer Node Workflow Node Borrowed from Djelloul Boukhelef

Karabo: The European XFEL software framework STATUS: Data Acquisition integration n Future work § n Think about the best way how to transport the data (which is send by Karabo devices) to the DAQ layer Open questions § § Requirements for sending data from Karabo devices to DAQ layer instead of sending data between devices for workflow purposes are different § No “smartness” needed (like load balancing, multi-cast, etc. ) § Writing to file can be done more generic, than further processing (what format is the best) Can we and should we try to use the same API and implementation for scientific workflows and DAQ sinking? Burkhard Heisen (WP 76) 31

Karabo: The European XFEL software framework Real time needs (where necessary) n 32 Karabo itself does not provide real time processes/communications § Motor 1 Motor 2 Pump 1 Real time processes (if needed) must be defined and executed in layers below Karabo devices will only start/stop/monitor real time processes § Gather/Scatter An example for a real-time system are the TCP (own protocoll) Ethercat based solutions from the company Beckhoff which we can interface to § Beck Com Interlock/Supervisory code can be PLCCPU Ethercat Motor 2 Pump 1 implemented at either PLC (realtime) and Karabo Burkhard Heisen (WP 76) Motor 1

Karabo: The European XFEL software framework Time synchronization (time stamps, cycle ids, etc. ) n Concept: Any changed property will carry timing information as attribute(s) § Time information is assigned per property § Karabo’s timestamp consists of the following information: Seconds since unix epoch, uint 64 Fractional seconds (up to atto-second resolution), uint 64 Train ID, uint 64 § Time information is assigned as early as possible (best: already on hardware) but latest in the software device § On event-driven update, the device ships the property key, the property value and associated time information as property attribute(s) § Real-time synchronization is not subject to Karabo § Correlation between control system (monitor) data and instrument data will be done using the archived central DB information (or information previously exported into HDF 5 files) Burkhard Heisen (WP 76) 33

Karabo: The European XFEL software framework DETAIL: Time synchronization Distributed Train ID clock n Concept: A dedicated machine with a time receiver board (h/w) distributes clocks on the Karabo level § Scenario 1: No time information from h/w § Example: commercial cameras Burkhard Heisen (WP 76) creates timestamp and associates to train. Id Device Timestamp is associated to the event-driven data in the Karabo device If clock signal is too late, the next train. Id is calculated (extrapolated) given the previous one and the interval between train. Id's The interval is configurable on the Clock device and must be stable within a run. Error is flagged if clock tick is lost. Scenario 2: Time information is already provided by h/w 34 The timestamp can be taken from the h/w or the device (configurable). The rest is the same as in scenario 1. signals: 1. train. Id 2. epoch. Time 3. interval Clock Time receiver board

Karabo: The European XFEL software framework Central services - Name resolution/access n The only central service technically needed is the broker, others are optional § Start-up issues Any object connecting to the same broker (host/port/topic) must have a unique ID (string) All communication objects will finally derive the Signal. Slotable class which can be instantiated with a given ID (configured) or generates one if no ID is provided If no instance ID is provided the ID is auto-generated locally § Servers: hostname_Server_pid § Devices: hostname-pid_class. Id_counter Any instance ID is validated (by request-response trial) prior startup to be unique in the distributed system Burkhard Heisen (WP 76) 35

Karabo: The European XFEL software framework DETAIL: Access levels 36 We will initially have five access levels (enums) with intrinsic ordering § ADMIN = 4 § EXPERT = 3 § OPERATOR = 2 § USER = 1 § OBSERVER = 0 n Any Device can restrict access globally or on a per-parameter basis § Global restriction is enforced through the “visibility” property (base class) Only if the requestor is of same or higher access level he can see/use the device The “visibility” property is part of the topology info (seen immediately by clients) § Parameter restriction is enforced through the “required. Access. Level” schema-attribute Parameter restriction typically is set programmatically but may be re-configured at initialization time (or even runtime? ) The “visibility” property might be re-configured if the requestors access level is higher than the associated “required. Access. Level” (should typically be ADMIN) The default access level for settable properties and commands is USER The default access level for read-only properties is OBSERVER The default value for the visibility is OBSERVER n Burkhard Heisen (WP 76)

Karabo: The European XFEL software framework DETAIL: Access levels A role is defined in the DB and consists of a default access level and a deviceinstance specific access list (overwriting the default level) which can be empty. § SPB_Operator default. Access. Level => USER access. List § SPB_* => OPERATOR § Undulator_Gap. Mover_0 => OPERATOR § Global_Observer default. Access. Level => OBSERVER § Global_Expert default. Access. Level = EXPERT n After authentication the DB computes the user specific access levels considering current time, current location and associated role. It then ships a default access and an access level list back to the user. § If the authentication service (or DB) is not available, Karabo falls back to a compiled default access level (in-house: OBSERVER, shipped-versions: ADMIN) n For a ADMIN user it might be possible to temporarily (per session) change the access list of another user. n Burkhard Heisen (WP 76) 37

Karabo: The European XFEL software framework DETAIL: Security 38 Broker-Message GUI or CLI GUI-Srv Header […] __uid=42 __access. Level=“admin” Body […] user. Id session. Token default. Access. Level access. List username password provider own. IP* broker. Host* broker. Port* broker. Topic* Locking: if is locked: if is __uid == owner then ok Access control: if __access. Level >= visibility: if __access. Level >= param. access. Level then ok Central DB 1. 2. Burkhard Heisen (WP 76) Authorizes Computes context based access levels Device

Karabo: The European XFEL software framework Statistics (control system itself, operation, …) n Concept: Statistics will be collected by regular devices § Open. MQ implementation provides a wealth of statistics (e. g. messages in system, average flow, number of consumers/producers, broker memory used…) § Have a (broker-)statistic device that does system calls to retrieve information § Similar idea for other statistical data Burkhard Heisen (WP 76) 39

Karabo: The European XFEL software framework Logging (active, passive, central, local) n Concept: Categorized into the following classes § Active Logging Additional code (inserted by the developer) accompanying the production/business code, which is intended to increase the verbosity of what is currently happening. Code Tracing Macro based, no overhead if disabled, for low-level purposes Code Logging Conceptual analog to Log 4 j, network appender, remote and at runtime priority (re-)configuration § Passive Logging Recording of activities in the distributed event-driven system. No extra coding is required from developers, passive logging transparently records system relevant events. Broker-message logging Low-level debugging purpose, start/stop, not active during production Burkhard Heisen (WP 76) Transactional logging Archival of the full distributed state (see Data. Logger) 40

Karabo: The European XFEL software framework Project n 41 The project is an organizational structure for logically related devices § The project does not describe: Which device-server should run on what host Which plugin is loaded to what device-server § The project acts on top of existing (running) device-servers and loaded plugins § It describes initial configurations, runtime configurations, macros, scenes, monitors and resources for a set of logically connected devices § Example projects could be: Detector_FXE Laser_FXE DAQ_FXE § Macros have an API to work with the project § Projects are associated to a user (can be a functional user) § The project itself is a set of files, it does not maintain a state (like “started” or “stopped”)

Karabo: The European XFEL software framework Project n Centralized project storing via a Karabo service device “Project. Manager” § Implement in Python (code already exists in GUI code) § Analog to Data. Logger. Manager or Calibration. Manager within Karabo Framework § Implement an output (loading project) and an input (saving project) channel § Allow multi-user (read) and single-user (write) access 42

Karabo: The European XFEL software framework Detail: Project file organization n The project is saved as a zipped folder named <projectname>. krb § The folder contains a project. xml file with the following structure: <project> § <devices>[…]</devices> § <macros>[…]</macros> § <scenes>[…]</scenes> § <monitors>[…]</monitors> § <resources>[…]</resources> </project> § And sub-folders containing files which are referenced by the above mentioned project. xml Devices § Containing <device>. xml files Macros § Containing <macro>. py files Scenes § Containing <scene>. svg files Resources § Containing any files (images, specific configurations, notes, etc. ) etc. Burkhard Heisen (WP 76) 43

Karabo: The European XFEL software framework STATUS: Project n n Recent changes § Logical grouping of devices of same class is possible § Groups allow multi-edit functionality (very useful for work-flow configurations) Future work § Central project handling must be implemented § The Monitors section must be implemented n Monitors are a user defined collection of properties that will be associated to a experimental run (or a control scan) Open questions § Current idea is to introduce another top hierarchy level -> a project group § Groups should be logical associations and only point/link to the physical projects § A project group could reflect all settings an experiment needs by aggregating all specialists projects (like laser, detector, daq, experiment) with the user project § Experiment configurations could be started by copying a (template) group and then modifying it by the individual experts until the specified setup is reached § Still not completely clear whether this approach will cover all needs for experiment control 44

Karabo: The European XFEL software framework Processing workflows (parallelism, pipeline execution, provenance) n Concept: Devices as modules of a scientific workflow system § § § § § Configurable generic input/output channels on devices One channel is specific for one data structure (e. g. Hash, Image, File, etc. ) New data structures can be “registered” and are immediately usable Input channel configuration: copy of connected output’s data or share the data with other input channels, minimum number of data needed Compute. Fsm as base class, developers just need to code the compute method IO system is decoupled from processing system (process whilst transferring data) Automatic (API transparent) data transfer optimization (pointer if local, TCP if remote) Broker-based communication for workflow coordination and meta-data sharing GUI integration to setup workflows graphically (drag-and-drop featured) Workflows can be stored and shared (following the general rules of data privacy and security) executed, paused and stepped Parallel execution Burkhard Heisen (WP 76) 45

Karabo: The European XFEL software framework DETAIL: Processing workflows Parallelism and load-balancing by design n 46 Devices within the same device-server: Data will be transferred by handing over pointers to corresponding memory locations § Multiple instances connected to one output channel will run in parallel using CPU threads § n Memory CPU-threads Devices in different device-servers: Data will be transferred via TCP § Multiple instances connected to one output channel will perform distributed computing § TCP Distributed processing n Output channel technically is TCP server, inputs are clients n Data transfer model follows an event-driven poll architecture, leads to load-balancing and maximum per module performance even on heterogeneous h/w n Configurable output channel behavior in case no input currently available: throw, queue, wait, drop Burkhard Heisen (WP 76)

Karabo: The European XFEL software framework DETAIL: Processing workflows GPU enabled processing n 47 Concept: GPU parallelization will happen within a compute execution The data structures (e. g. image) are prepared for GPU parallelization § Karabo will detect whether a given hardware is capable for GPU computing at runtime, if not falls back to corresponding CPU algorithm § Differences in runtime are balanced by the workflow system § CPU IO whilst computing Pixel parallel processing (one GPU thread per pixel) Notification about new data possible to obtain GPU Burkhard Heisen (WP 76)

Karabo: The European XFEL software framework Clients / User interfaces (API, languages, macro writing, CLI, GUI) n Concept: Two UIs – graphical (GUI) and scriptable command line (CLI) § § GUI Have one multi-purpose GUI system satisfying all needs See following slides for details Non-GUI We distinguish APIs for programmatically set up of control sequences (others call those Macros) versus and API which allows interactive, commandline-based control (IPython based) The programmatic API exists for C++ and Python and features: § Querying of distributed system topology (hosts, device-servers, devices, their properties/commands, etc. ): get. Servers, get. Devices, get. Classes § instantiate, kill, set, execute (in “wait” or “no. Wait” fashion), get, monitor. Property, monitor. Device Both APIs are state and access-role aware, caching mechanisms provide proper Schema and synchronous (poll-feel API) although always event-driven in the backend The interactive API integrates auto-completion and improved interactive functionality suited to i. Python Burkhard Heisen (WP 76) 48

Karabo: The European XFEL software framework GUI: What do we have to deal with? n Client-Server (network protocol, optimizations) n User management (login/logout, load/save settings, access role support) n Layout (panels, full screen, docking/undocking) n Navigation (devices, configurations, data, …) n Configuration (initialization vs. runtime, loading/saving, …) n Customization (widget galleries, custom GUI builder, composition, …) n Notification (about alarms, finished pipelines, …) n Log Inspection (filtering, configuration of log-levels, …) n Embedded scripting (i. Python, macro recording/playing) n Online documentation (embedded wiki, bug-tracing, …) Kerstin Weger (WP 76) 49

Karabo: The European XFEL software framework Client-Server (network protocol, optimizations) n Message Broker Concept: One server, many clients, TCP § Server knows what each client user sees (on a device level) and optimizes traffic accordingly § Client-Server protocol is TCP, messages are header/body style using Hash serialization (default binary protocol) § Client side socket will be threaded to decouple from main-event loop § § On client start server provides current distributed state utilizing the DB, later clients are updated through the broker Image data is pre-processed on server-side and brought into QImage format before sending Central DB Master GUI-Srv I only see device “A” on. Change information only related to “A” GUI-Client Kerstin Weger (WP 76) 50

Karabo: The European XFEL software framework User management (login/logout, load/save settings, access role support) n 51 Concept: User centralized, login mandatory § Login necessary to connect to system § Access role will be computed (context based) § User specific settings will be loaded from DB § View and control is adapted to access role § User or role specific configuration and wizards are available user. Id access. Role session username password Central DB 1. 2. Kerstin Weger (WP 76) Authorizes Computes context based access role

Karabo: The European XFEL software framework Layout (panels, full screen, docking/undocking) n Six dock-able and slide-able (optionally tabbed) main panels § Panels are organized by functionality Navigation Custom composition area (sub-GUI building) Configuration (non-tabbed, changes view based on selection elsewhere) Documentation (linked and updated with current configuration view) Logging Notifications Project § Panels and their tabs can be undocked (windows then belongs to OS’s window manager) and made full-screen (distribution across several monitors possible) § GUI behaves natively under Mac. OSX, Linux and Windows Kerstin Weger (WP 76) 52

Karabo: The European XFEL software framework Graphical interface - Overview Live Navigation drag & drop Custom Scene 53 Configuration n User centric and access-controlled setup (login at startup) n Dock-able and resizable multi-panel, all-in-one user interface n Live navigation showing all device-servers, plugins, and device instances n Automatically generated configuration panel, allowing to read/write/execute n Power. Point like, drag & droppable, tabbed custom scene n Project panel for persisting configurations, macros, scenes, resources, etc. Centralized logging information, notification handling, documentation, etc. Log Messages Documentation Project Interactive Command Line Bug Reporting n Burkhard Heisen (CAS Group)

Karabo: The European XFEL software framework Navigation (devices, configurations, data, …) n Concept: Navigate device-servers, devices, configurations, data(-files), etc. § Different views (tabs) on data Hierarchical distributed system view Device ownership centric (view compositions) Hierarchical file view (e. g. HDF 5) § Automatic (by access level) filtering of items § Auto select navigation item if context is selected somewhere else in GUI Kerstin Weger (WP 76) 54

Karabo: The European XFEL software framework Configuration (initialization vs. runtime, loading/saving, …) n Concept: Auto-generated default widgets for configuring classes and instances § Widgets are generated from device information (. xsd format) § 2 -column layout for class configuration (label, initialization-value) § 3 -column layout (label, value-on-device, edit-value) for instance configuration § Allows reading/writing properties (all data-types) § Allows executing commands (as buttons) § Is aware about device’s FSM, enables/disables widgets accordingly § Is aware about access level, enables/disables widgets accordingly § Single, selection and all apply capability Kerstin Weger (WP 76) 55

Karabo: The European XFEL software framework Customization (widget galleries, custom GUI builder, composition, …) n Concept: Combination of Power. Point-like editor and online properties/commands with changeable widget types § Tabbed, static panel (does not change on navigation) § Two modes: Pre-configuration (classes) and runtime configuration (instances) § Visual composition of properties/commands of any devices § Visual composition of devices (workflow layouting) § Data-type aware widget factory for properties/commands (edit/display) § Power. Point-like tools for drawing, arranging, grouping, selecting, zooming of text, shapes, pictures, etc. § Capability to save/load custom panels, open several simultaneously Kerstin Weger (WP 76) 56

Karabo: The European XFEL software framework DETAIL: Customization Property/Command composition Display widget (Trend-Line) Editable widget drag & drop Display widget Kerstin Weger (WP 76) 57

Karabo: The European XFEL software framework DETAIL: Customization Property/Command composition Display widget (Image View) Display widget (Histogram) Kerstin Weger (WP 76) drag & drop 58

Karabo: The European XFEL software framework DETAIL: Customization Device (workflow) composition n Whole devices can be dragged (from left side) as pipeline nodes n Dragging individual parameters from right is still possible (e. g. control parameters) n Devices can be grouped and edited as group (connections and configurations) n Distributed computing will happen if different hosts are involved n Display of per node or nodegroup utilization Kerstin Weger (WP 76) drag & drop 59

Karabo: The European XFEL software framework Macro editing and execution n Macro editing and execution from within GUI possible n Macro parameters and functions integrate automatically into configuration panel n Macros are running within the GUI’s event loop (direct widget manipulation possible) n Macro API can be interactively executed in embedded IPython interpreter n Asynchronous operations use Python 3’s coroutines and the yield from keyword (extension written allowing this for IPython) 60 Courtesy of M. Teichmann Burkhard Heisen (WP 76)

Karabo: The European XFEL software framework Notification (about alarms, finished runs, …) n Concept: Single place for all system relevant notifications, will link-out to more detailed information § Can be of arbitrary type, e. g. : Finished experiment run/scan Finished analysis job Occurrences of errors, alarms Update notifications, etc. § Intended to be conceptually similar to now-a-days smartphone notification bars § Visibility and/or acknowledgment of notifications may be user and/or access role specific § May implement some configurable forwarding system (SMS, email, etc. ) Kerstin Weger (WP 76) 61

Karabo: The European XFEL software framework Log Inspection (filtering, configuration of log-levels, …) n 62 Concept: Device’s network appenders provide active logging information which can be inspected/filtered/exported § Tabular view § Filtering by: full-text, date/time, message type, description § Export logging data to file § Logging events are decoupled from main event loop (threading) § Uses Qt’s model/view with SQLite DB as model (MVC design) Kerstin Weger (WP 76)

Karabo: The European XFEL software framework Online documentation (embedded wiki, bug-tracing, …) n 63 Concept: Make the GUI a rich-client having embedded internet access. Use it for web based device documentation, bug tracking, feature requests, etc. § Any device class will have an individual (standardized) wiki page. Pages are automatically loaded (within the documentation panel) as soon as any property/command/device is selected elsewhere in GUI (identical to configuration panel behavior). Depending on access role, pages are immediately readable/editable. § Device wiki pages are also readable/editable via European XFEL’s document management system (Alfresco) using standard browsers § For each property/command the coded attributes (e. g. description, units, min/max values, etc. ) is shown. § European XFEL’s bug tracking system will be integrated Kerstin Weger (WP 76)

Karabo: The European XFEL software framework Software management (coding, building, packaging, deployment, versioning, …) n Concept: Spiced up Net. Beans-based build system, software-bundle approach § Clear splitting of Karabo-Framework (distributed system) from Karabo-Packages (plugins, extensions) § Karabo-Framework (SVN: karabo/karabo. Framework/trunk) § Coding done using Net. Beans (for c++ and python), Makefile based Contains: karabo-library (libkarabo. so), karabo-deviceserver, karabobrokermessagelogger, karabo-gui, and karabo-cli Karabo-library already contains python bindings (i. e. can be imported into python) Makefile target “package” creates self-extracting shell-script which can be installed on a blank (supported) operating system and is immediately functional Embedded unit-testing, graphically integrated into Net. Beans (c++ and python) Karabo-Packages (SVN: karabo/karabo. Packages/category/package. Name/trunk) After installation of Karabo-Framework packages can be build SVN checkout of a package to any location and immediate make possible Everything needed to start a full distributed Karabo instance available in package A tool for package development is provided (templates, auto svn integration, etc. ) Burkhard Heisen (WP 76) 64

Karabo: The European XFEL software framework Software management - Tools n Continuous integration system using Jenkins (nightly builds on different platforms) n Jenkins automatically runs all unit-tests for each build and tests execution of binaries n Redmine for project management (features, bugs, releases, versioning integration) n Installation through software bundle approach (all dependencies are shipped), user does not need to compile nor install any system packages n Deployment system for distributed device-servers and their plugins Burkhard Heisen (CAS Group) 65

Karabo: The European XFEL software framework DETAIL: Software management The four audiences and their requirements n Framework Developer § § § n Package Developer § § n Flexible access to the Karabo framework ($HOME/. karabo encodes default location) Allow "one package - one software" project mode (each device project has its own versioning cycle, individual Netbeans project) Standards for in-house development or XFEL developers need to be fullfilled: use parametrized templates provided, development under Netbeans, use SVN, final code review Possibility to add further extern dependencies to the Karabo framework (see above) System Integrator/Tester § § § n SVN interaction, versioning, releases Code development using Netbeans/Visual Studio Addition of tests, easy addition of external dependencies Tools for packaging the software into either binary + header or source bundles Allow for being framework developer and package developer (see below) in one person at the same time Simple installation of Karabo framework and selected Karabo packages as binaries Start broker, master, i. e. a full distributed system Flexible setup of device-servers + plugins, allow hot-fixes, sanity checks XFEL-User/Operator § § Easy installation of pre-configured (binary framework + assortment of packages) karabo systems Run system (GUI, CLI) Burkhard Heisen (WP 76) 66

Karabo: The European XFEL software framework DETAIL: Software management Unit-testing C++ Burkhard Heisen (WP 76) 67 Python

Karabo: The European XFEL software framework DETAIL: Software management Continuous integration n Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. [Wikipedia] n Required Features: § § § n Support for different build systems and different OS Automated builds – nightly builds Continuous builds – on demand, triggered by SVN commit Build matrix – different OS, compiler options Web interface – configuration, results Email notification Build output logging – easy access to output of build errors Reporting all changes from SVN since last successful build – easy trace of guilty developer Plugin for any virtualization product (Virtual. Box, VMWare, etc. ) Netbeans plugin for build triggering Easy uploading of build results (installation packages) to web repository CI systems on the market: Hudson, Cruise. Control, buildbot, Team. City, Jenkins … Burkhard Heisen (WP 76) 68

Karabo: The European XFEL software framework DETAIL: Software management Continuous integration Burkhard Heisen (WP 76) 69

Karabo: The European XFEL software framework Conclusions n The distributed system is device-centric (not attribute-centric), devices inherently express functionality for communication, configuration and flow control n The provided services focus on solving general problems like data-flow, configuration, project-tracking, logging, parallelization, visualization, provenance n XFEL. EU software will be designed to allow simple integration of existing algorithm/packages n The ultimate goal is to provide a homogenous software landscape to allow fast and simple crosstalk between all computing enabled categories (Control, DAQ, Data Management and Scientific Computing) Burkhard Heisen (WP 76) 70

Karabo: The European XFEL software framework 71 Thank you for your kind attention. Burkhard Heisen (WP 76)