LHCbs Experiment Control System Step by Step Clara

LHCb’s Experiment Control System Step by Step Clara Gaspar, March 2006

Overview ❚ LHCb’s Experiment Control System ❙ What do we (JCOP/LHCb) provide ❙ What sub-detectors/sub-systems need to implement ❚ PVSS & Framework reminder ❚ Interfacing Electronics Boards ❙ SPECS & CC-PC Tools ❙ The Configuration DB ❚ Hierarchical Control ❙ The FSM Toolkit ❚ Note: This tutorial is meant as an overview ❙ The PVSS & Framework and the FSM courses are still required in order to use the tools! Clara Gaspar, March 2006 2

ECS Scope Experiment Control System DCS Devices (HV, LV, GAS, Temperatures, etc. ) Detector Channels L 0 TFC Front End Electronics Readout Network High Level Trigger Storage DAQ External Systems (LHC, Technical Services, Safety, etc) Clara Gaspar, March 2006 3

T. S. GAS LHC . . . Det. Dcs 1 Sub. Sys 1 DAQ DCS Det. Dcs. N Sub. Sys 2 Dev 1 Dev 2 Det. Daq 1 Sub. Sys. N Dev 3 DSS . . . Commands ECS Status & Alarms Abstract levels ECS Generic Architecture Dev N To Devices (HW or SW) Clara Gaspar, March 2006 4

What do we provide? ❚JCOP + LHCb Online provide: ❙Not complete applications, but: ❙A Framework, i. e. a set of tools to help subsystems create their control systems: ❘Complete, configurable components (ex. CAEN HV) ❘Tools for defining User Components: � Electronics boards (SPECS/ CC-PC) � Specific equipment/software tasks (DIM protocol) ❘Other Tools, for example: � FSM for Building Hierarchies � Configuration DB Clarahandling, Gaspar, March � Archiving, Alarm etc. 2006 5

We also provide: ❚Integration of Infrastructure Services: ❘Power Distribution and Rack/Crate Control ❘Cooling and Ventilation Control ❘Magnet Control (Monitoring) ❘Gas Control ❘Detector Safety System ❚And interface to: ❘LHC machine ❘Access Control System ❘CERN Safety System ❚Sub-detectors can use these components: ❙For defining logic rules (using their states) ❙For high-level operation (when applicable) ❘Switch ON, Switch Off, Set parameters Clara Gaspar, March 2006 6

And also Database Tools Experimental Equipment ❚Interfaces to the three Logical Databases in the Online System Conf. DB . . . PVSS Cond. . . DB Clara Gaspar, March 2006 PVSS Arch. To Offline 7

Online Database Contents ❙Configuration DB contains: ❘All data needed to configure the HW (or SW) for the various running modes � Ex. : HV V 0 Settings, Pedestal settings, trigger settings, etc. ❙PVSS Archive contains: ❘All monitoring data read from HW for monitoring and debugging of the Online System � Ex. : HV Vmon Readings, temperatures, pedestal readings, etc. ❙Conditions DB contains: ❘A subset of the monitoring data read from HW if it is needed for Event processing (prob. packaged differently) � Ex. : HV Vmon Readings if changed by more than n Volts ❘Some configuration data once it has been used � Ex. : Trigger settings used by a particular run Clara Gaspar, March 2006 8

The Configuration DB ❚The Configuration DB will contain: ❙All "static" information about the devices ❘Connectivity, addresses, etc. (also inventory and history) ➨ Developed within LHCb (supports queries) ❙All "dynamic" data needed by the devices (for different running modes and different versions): ❘Settings (voltages, alarm limits, etc. ), Calibration constants, Pedestals, FPGA code (probably a pointer to it), etc. ➨The settings for a particular running mode are called a “Recipe” (partial recipes available) ➨The JCOP FW component implements a cache: � Can be used without Oracle for tests � Can pre-load several recipes before “Start of Run” Clara Gaspar, March 2006 9

What needs to be done: ❚Start bottom up ❙Integrate each device into PVSS ❙Define configuration recipes ❘for the various running modes ❙Build a hierarchy for each sub-system ❘According to the guidelines ❙Integrate the devices in the hierarchy Clara Gaspar, March 2006 10

Device Integration ❚Device Types ❙HV & LV channels ❘CAEN, ISEG, WIENNER -> JCOP Framework ❙Analog inputs ❘ELMB -> JCOP Framework ❙Electronics boards ❘SPECS & CC-PC -> Tools to describe boards ❘TELL 1 -> FW component (for common part) ❙Other Components ❘HW or SW -> Fw. DIM component ➨ Needs: PVSS, Framework, DIM, … Clara Gaspar, March 2006 11

PVSS Clara Gaspar, March 2006 12

PVSS Distribution Clara Gaspar, March 2006 13

Datapoint Concept ❚DP type -> DP Configs Clara Gaspar, March 2006 14

Graphical Objects ❚Reference Panels ❙Can be “inherited” dynamically ❙“$parameters” get replaced by instance value Clara Gaspar, March 2006 15

Building User Interfaces ❚Static Part -> Drag & Drop ❚Dynamic part -> Control Scripts ("C" like) ❙A few usefull calls for accessing DPs: ❘dp. Get (string dp. Name, <data_type> value) ❘dp. Set (string dp. Name, <data_type> value) ❘dp. Connect (string callback, string dp. Name) ❙A few usefull calls for accessing Widgets: ❘get. Value (string widget. Name, string widget. Property, <widget dependent data>) ❘set. Value (string widget. Name, string widget. Property, <widget dependent data>) Clara Gaspar, March 2006 16

PVSS Features ❚Open Architecture ❙We can write our own managers ➨It can be interfaced to anything (FSM, DIM) ❚Highly Distributed ❙ 130 Systems (PCs) tested ➨No major problem found ❚Standard Interface ❙All data of all sub-systems defined as Data. Points! Clara Gaspar, March 2006 17

Demo-1 ❚Start PVSS console ❚Create a project (add installation tool) ❚PVSS basic functionality ❙PVSS Managers ❙Parameterization Module ❘Datapoint structures ❙Graphic editor Clara Gaspar, March 2006 18

Demo-2 ❚Install Framework ❘fw. Core ❘fw. Analog. Digital ❘fw. Caen ❘fw. Configuration. DB ❘fw. DIM ❘fw. Specs ❘fw. Hw ❚CAEN component: ❙Create Crates/Boards/Channels ❘“Crate 0” will be used by FSM later ❙Show Operation panels Clara Gaspar, March 2006 19

DIM Distributed Information Management System ❙Publish/Subscribe mechanism ❘Servers publish Services. ❘Clients subscribe to Services: � On change or at regular intervals ❘Clients can send commands to Servers ❙Services ❘A set of data � any type or size � Identified by a name ❙A Name Server ❘Keeps a list of available Services Clara Gaspar, March 2006 20

DIM Some Characteristics ❙Transparency ❘DIM clients do not know where their interlocutors are. ❘DIM components can move from one machine to another, all connections are transparently re-established. ❙Available on mixed environments: ❘UNIX (HP-UX, Sun-OS, Sun-Solaris, IBM-AIX, DEC-OSF, Linux), Windows, VMS, Real-time OSs (OS 9, Lynx. OS, Vx. Works) � API available in “C”, C++ and Java ❙Easy to Use ❘One “call” and a process can become a server or a client. ❘Monitoring and Visualization Tools Available. ❘Documentation and examples at: http: //www. cern. ch/dim Clara Gaspar, March 2006 21

PVSS<->DIM ❚Fw. DIM component: ❙Server is a DIM Server ❙Client is a PVSS Manager (PVSS 00 dim) ❙Correspondence: PVSS DPs <-> DIM Services ❘Can be setup graphically via fw. DIM panel ❘Or via a script library ❙When setup ❘When Server updates Service data goes into DP ❘Writing to DP will send a DIM Command ❙Documentation at: ❘http: //www. cern. ch/lhcb-online/ecs/fw/Fw. Dim. html Clara Gaspar, March 2006 22

Non-standard components ❚Integrating user components: ❙Create a DIM server (C or C++) ❘Publishes device status & data ❘Receives Commands ❙ Create a PVSS Datapoint ❘That matches the structure of DIM services ❙Connect the DP to the DIM services ❘Using the Fw. DIM tools ❙Make a PVSS panel to control the device ❚ Used for: farm monitoring, trigger algorithms, etc. Clara Gaspar, March 2006 23

Demo-3 ❚Fw. DIM ❙Configure DIM_DNS_NODE ❙Start a DIM server (ex. : pvss_dim_server) ❙Start DIM visualization tool ❘DIMTree on Windows ❘DID on Linux ❙Start fw. DIM. pnl ❘Connect services to DPs ❘Visualize from PVSS Clara Gaspar, March 2006 24

Electronics Interface ❚CC-PC & SPECS tools: ❙Low-level Software ❘A “C” library for accessing board components � Via I 2 C, JTAG or parallel bus (and FPGA programming) ❘A Generic DIM server for PVSS Communication ❙PVSS Tools (FW components: fw. Ccpc/fw. Specs) ❘A library (PVSS scripting) for accessing board components on any board with a CC-PC/Specs (equivalent to the low-level library) ❘A graphical user interface providing the functionality available in the library Clara Gaspar, March 2006 25

Electronics Integration ❚Electronics Boards: ❙Can use the CCPC/SPECS FW Tools for tests, but accessing the “chips” is not enough ❙Boards have to be modeled in PVSS according to guidelines (ex. registers have to correspond to datapoints) in order to: ❘Provide access to the Conf. DB � Select a device/group of devices and say: Save as “Physics” recipe. ❘Be able to archive the data ❘Be able to send the data to the Cond. DB ❘Integrate into the FSM, Generate alarms, etc. Clara Gaspar, March 2006 26

Electronics Integration ❚We provide a tool for modeling boards and their components (FWcomponent: Fw. Hw) ❘Declaring boards (access via SPECS or CC-PC) Containing: � Groups of Chips (recursive) Containing: ❘Chips (TTCrx, Beetle, etc. ) Containing: ❘Registers (access via I 2 C/JTAG/Parallel Bus) ❙Contacts: ❘Ricardo Fernandes: SPECS ❘Stefan Koestner: CC-PC Clara Gaspar, March 2006 27

Electronics boards ❚Demo Setup Client. PC: Portable PVSS Server. PC: pclbcecs 03 PVSS 00 dim Ethernet Specs. Srv SPECS Master PVSS DNS SPECS Mezzanine Croquette I 2 C PVSS 00 dim Support. PC: pclhcb 155 I 2 C widget Note: The DNS should run on a stable machine (same as PVSS), not on a portable… Clara Gaspar, March 2006 28

Demo-4 ❚Fw. Specs: ❙Server PC: pclbcecs 03 ❘Configure DIM_DNS_NODE ❘Start Specs. Server remotely ❙Client PC: portable ❘Configure DIM_DNS_NODE ❘Start Specs. Client “direct access” panel � Exercise I 2 C, JTAG, DCU ❘Explain the “Monitoring” feature ❘Show Advanced Panel (User Scripts) ❙Documentation at (not this version yet): ❘http: //www. cern. ch/lhcb-online/ecs/PVSS_SPECS ❚ Fw. Ccpc: very similar ❙ Tools will be presented at Online meeting Clara Gaspar, March 2006 29

Custom Electronics ❚Demo Example Server PC Specs. Srv SPECS Master SPECS Mezzanine I 2 C TTCrx Beetle 1 Beetle 2 Velo Board Clara Gaspar, March 2006 30

Demo-5 ❚Fw. Hw ❙Create HW types: ❘ TTCrx, Beetle and Velo. Board ❙Configure Default Settings ❙Create velo. Boards ❘“Operate” the board ❙Interface to Configuration Database (cache) ❘Save recipes (“PHYSICS”, “TEST”, etc. ) ❘Download recipes Clara Gaspar, March 2006 31

Electronics guidelines ❚Fw. Hw: Some Guidelines ❙If a “chip” has many registers ❘If they can be written in one single operation � Declare them as 1 register of size N � This will optimize configuration time ❘Some (a few) can also be declared separately � If they are often accessed individually ❚After using Fw. Hw to define the boards: ❙Design a user interface to operate each board type ❘The library fw. Specs or fw. Ccpc will give you access to the data to be visualized or sent to the board ex. : fw. Specs_read(“board 1. ttcrx 1. reg 2”, …) Clara Gaspar, March 2006 32

Control Hierarchy ❚Building a Control Hierarchy ❙And integrating Devices ❚Needs: Fw. FSM, LHCb guidelines ECS T. S. GAS LHC . . . Det. Dcs 1 Sub. Sys 1 DAQ DCS Det. Dcs. N Sub. Sys 2 Dev 1 Dev 2 Det. Daq 1 DSS . . . Sub. Sys. N Dev 3 Dev N Clara Gaspar, March 2006 33

Control Units ❚Each node is able to: ❙Summarize information (for the above levels) ❙“Expand” actions (to the lower levels) ❙Implement specific behaviour & Take local decisions DCS Tracke r ❘Sequence & Automate operations ❘Recover errors H V Tem p Muon H V GA S ❙Include/Exclude children (i. e. partitioning) ❘Excluded nodes can run is stand-alone ❙User Interfacing ❘Present information and receive commands Clara Gaspar, March 2006 34

Device Units ❚Device Units ❙Provide the interface to real devices: (Electronics Boards, HV channels, trigger algorithms, etc. ) Dev ❘Can be enabled/disabled N ❘In order to integrate a device within FSM � Deduce a STATE from device readings (in DPs) � Implement COMMANDS as device settings ❘Commands can apply the recipes previously defined Clara Gaspar, March 2006 35

The Control Framework ❚The Fw. FSM Component is based on: ❙PVSS for: Device Units Control Units ❘Device Description (Run-time Database) ❘Device Access (OPC, Profibus, drivers) ❘Alarm Handling (Generation, Filtering, Masking, etc) ❘Archiving, Logging, Scripting, Trending ❘User Interface Builder ❘Alarm Display, Access Control, etc. ❙SMI++ providing: ❘Abstract behavior modeling (Finite State Machines) ❘Automation & Error Recovery (Rule based system) Clara Gaspar, March 2006 36

SMI++ ❚Method ❙Classes and Objects ❘Allow the decomposition of a complex system into smaller manageable entities ❙Finite State Machines ❘Allow the modeling of the behavior of each entity and of the interaction between entities in terms of STATES and ACTIONS ❙Rule-based reasoning ❘Allow Automation and Error Recovery Clara Gaspar, March 2006 37

SMI++ ❚Method (Cont. ) ❙SMI++ Objects can be: ❘Abstract (e. g. a Run or the DCS) ❘Concrete (e. g. a power supply or a temp. sensor) ❙Concrete objects are implemented externally either in "C", in C++, or in PVSS (ctrl scripts) ❙Logically related objects can be grouped inside "SMI domains" representing a given sub-system (Framework: Control Unit) Clara Gaspar, March 2006 38

SMI++ Run-time Environment ❙Device Level: Proxies Obj SMI Domain Obj Obj Obj ❘drive the hardware: � deduce. State � handle. Commands ❘C, C++, PVSS ctrl scripts ❙Abstract Levels: Domains ❘Implement the logical model ❘Dedicated language - SML ❘A C++ engine: smi. SM ❙User Interfaces ❘For User Interaction Proxy Hardware Devices ❙All Tools available on: ❘Windows, Unix (Linux) ❘All communications are transparent and dynamically (re)established Clara Gaspar, March 2006 39

SMI++ ❚SMI++ - The Language ❙SML –State Management Language ❘Finite State Logic � Objects are described as FSMs their main attribute is a STATE ❘Parallelism � Actions can be sent in parallel to several objects. Tests on the state of objects can block if the objects are still “transiting” ❘Asynchronous Rules � Actions can be triggered by logical conditions on the state of other objects Clara Gaspar, March 2006 40

SML – The language ❚ Devices: ❚ Sub System: ❚ Objects can be dynamically included/excluded in a Set Clara Gaspar, March 2006 41

SML example (automation) ❚ External Device: ❚ Sub System: Clara Gaspar, March 2006 42

PVSS/SMI++ Integration ❚ Graphical Configuration of SMI++ Using PVSS Clara Gaspar, March 2006 43

Building Hierarchies ❚Hierarchy of CUs ❙Distributed over several machines ❘"&" means reference to a CU in another system ❙Editor Mode: ❘Add / Remove / Change Settings ❙Navigator Mode ❘Start / Stop / View Clara Gaspar, March 2006 44

Control Unit Run-Time ❚Dynamically generated operation panels (Uniform look and feel) Clara Gaspar, March 2006 ❚ Configurable User Panels 45

Features of PVSS/SMI++ ❚Task Separation: ❙SMI Proxies/PVSS Scripts execute only basic actions – No intelligence ❙SMI Objects implement the logic behaviour ❙Advantages: ❘Change the HW -> change only PVSS ❘Change logic behaviour sequencing and dependency of actions, etc -> change only SMI rules Clara Gaspar, March 2006 46

Features of PVSS/SMI++ ❚Error Recovery Mechanism ❙Bottom Up ❘SMI Objects react to changes of their children � In an event-driven, asynchronous, fashion ❙Distributed ❘Each Sub-System recovers its errors � Each team knows how to recover local errors ❙Hierarchical/Parallel recovery ❙Can provide complete automation even for very large systems Clara Gaspar, March 2006 47

Demo-6 ❚Show a simple Hierarchy ❙Install fw. LHCb_Fsm. Domains ❙In installs standard LHCb FSM Domain Types ❙And it creates: VELO DCS VELO Motors VELO Temp ❙Show Include/Exclude and Enable/Disable ❙Show “Temp” FSM and Alarm Handling Clara Gaspar, March 2006 48

Sub-detector FSM Guidelines ❚Started defining naming conventions. ❚Defined standard “domains” per sub-detector: ❙ DCS ❘ DCS Infrastructure (Cooling, Gas, Temperatures, pressures, etc) that is normally stable throughout a running period ❙ HV ❘ High Voltages or in general components that depend on the status of the LHC machine (fill related) ❙ DAQ ❘ All Electronics and components necessary to take data (run related) ❙ DAQI ❘ Infrastructure necessary for the DAQ to work (computers, networks, electrical power, etc. ) in general also stable throughout a running period. ❚And standard states & transitions per domain. ❚ Doc available in EDMS: ❘ https: //edms. cern. ch/document/655828/1 Clara Gaspar, March 2006 49

FSM Guidelines ❚State Diagram for Trigger and DAQ Domains: ❙Possible intermediate “CONFIGURING” and “STARTING” states if operations slow… Clara Gaspar, March 2006 50

Hierarchy ECS Infrast. DCS MUON VELO DCS_1 VELO DCS_2 HV MUON VELO HV HV DAQI MUON VELO DAQI DAQ L 0 MUON VELO DAQ_1 TFC HLT LHC Sub. Farm 1 N VELO DAQ_2 VELO Dev 1 Dev. N Clara Gaspar, March 2006 51

Hierarchy & Conf. DB ECS 1 Infrast. DCS HV DAQI DAQ L 0 TFC HLT LHC 1 MUON VELO DCS MUON VELO HV HV MUON VELO DAQI MUON VELO DAQ 2 Conf. DB 1 VELO DCS_2 VELO DAQ_1 VELO 3 VELO Dev 1 Dev. N VELO DAQ_2 1 2 3 Configure/mode=“PHYSICS” (Get “PHYSICS” Settings) Apply Settings Clara Gaspar, March 2006 52

Demo-7 ❚Using type: DAQ_Domain ❚Create: VELO DAQ VELO FEE VELO TELL 1 VELO Dev 1 Board 1 ❚Create type Velo. Board ❚Integrate velo. Board 1, velo. Board 2, … ❙Apply recipes on “Configure” command ❙Note: There will be a “configurator” object per CU which gets recipes from DB to cache Clara Gaspar, March 2006 53

Demo-8 ❚Using type: ECS_Domain ❚Create: VELO DAQ VELO FEE VELO DCS VELO TELL 1 VELO Motors VELO HV VELO Temp VELO Dev 1 Board 1 Clara Gaspar, March 2006 54

The End ❚Questions? Clara Gaspar, March 2006 55

Hierarchy & Partitioning ECS Infrast. DCS MUON VELO DCS_1 VELO DCS_2 HV MUON VELO HV HV DAQI MUON VELO DAQI DAQ L 0 TFC HLT LHC MUON VELO DAQ_1 VELO DAQ_2 Clara Gaspar, March 2006 VELO 56