ALICE DCS part 1 DCS workshop 3 4

ALICE DCS part 1 DCS workshop 3. 4. 06 L. Jirden for the ALICE team u Organization & responsibilities u Coordination role u JCOP participation u Installation & commissioning u Critical issues part 1 1

Organization SERVICES Gas ACR Consoles ECS Trigger DAQ HLT JCOP boards, w-groups, project meetings 3 staff 3. 5 PJAS 1 Fellow 1 Student TC, TB ALICE Controls Coordination (ACC) DCS Workshop “Controls Board” ~2 d. each ALICE week Individual meetings Offline al u vid ings i d In eet m Individual meetings Facilities Power Cooling Magnets Safety Networks Access INFRASTR B-field Space frame Radiation Environment Detector groups SPD SDD SSD TPC TRD TOF HMP PHO CPV FMD T 0 V 0 PMD MTG MTK ZDC ACO EMC 2

Liaison with Detectors ALICE Controls Coordination (ACC) User Req. Doc (URD) Overview Drawings DCS Workshop Documented ALICE Standards: • • • Individual meetings ALICE DCS State Diagrams Guidelines & Conventions ALICE DCS Computing Rules Fed. Server API for ALICE DCS Interlocks etc Common Solutions Detector groups SPD SDD SSD TPC TRD TOF HMP PHO CPV FMD T 0 V 0 PMD MTG MTK ZDC ACO EMC 3

For each detector: Template version 2 (19 December 2001) DCS User Requirements Document for My-Detector Version x (dd month yyyy) 1. Introduction 2. Description of My-Detector 3. The Control System I. Description and requirements of the Sub-systems 1) a) b) c) d) Sub-system 1 Functionality Device or Equipment Interlocks and Safety aspects Operational and Supervisory aspects etc. 4

[FSM? ]drawing Overview Control room (ACR) TPC(1) 26/11/04 Database(s) PVSS II OPC client User interface PVSS II DIM client Ethernet PVSS II OPC client ISEG OPCserver Wiener OPCserver ELMB OPCserver C PCI-CAN C 1. . 3 DIMclient PCI-CAN C 1. . 22 OPC client PVSS II ELMB OPCserver RS 232 PCI-CAN E 1. . 18 [FED] PCI-CAN PVSS II 3 ISEG S DIMsrv C 1 1? 1 DIMcl Heinzinger ? Eth. Switch E ? 22 Wiener Eth. Switch ? E 18? HV 288 LV 2 HV ELMB 108 1 ELMB 108? Chambers High Voltage Chambers Low Voltage Chambers, detector Temp. Monitor ? DDL LV monitor ~400? 216 DIMsrv RCU FEE HVrod Very High Voltage VHV Curr. 5

Sub-systems: Common and specific solutions HMPI PHOS D SPD SDD SSD TPC TRD TOF HV CAEN iseg CAEN Easy iseg +distr CAEN Iseg +V reg. CAEN +distr CAEN LV CAEN Easy AREM pro CAEN Easy Wiener ALICE box CAEN Easy Wiener VME Wiener VME Wiener VME Wiener VME PLC ELMB MON FED JTAG DCS ELMB JTAG RCU DCS DRM CPV ELMB µ-trk µ-trg FMD RCU T 0 V 0 PMD ZDC ACO EMC CAEN CAEN Iseg CAEN Easy CAEN Easy Wiener VME Wiener VME ELMB RCU DRM DCS Wiener VME ? DCS Trig. DIM ELMB RCU Cool GAS VHV Liquid LED Pulser GMS Thresh Crystal Position LED? Rack. Ctrl Laser Align Laser BField Drift Vel. Rad. Mon Common solution FED concept Specific solution 67 7 11 TOTAL Common/specific 7 Unknown 2 105 6

Responsibilities JCOP + ACC + service groups DAQ + ACC ACR Safety Consoles CSAM CR facilities External Systems ACC ECS DSS JCOP + ACC Central DCS LHC ACC JCOP + ACC Tools & Components ACC ACCDetectors SPD SDD SSD TPC TRD TOF HMP PHO FMD T 0 Electricity Access H/W infrastructure Offline Gas Magnets DAQ HLT Services Cooling Back-End Trigger JCOP + ACC + service groups ACC Infrastr B-field Space Frame Env. Param. Radiation V 0 PMD MTI MTK ZDC ACO ~100 sub-systems 7

Central DCS CERN Consoles Remote Access Backend Servers Domain Controller Firewall System Management DCS Operator IP Reach for KVM Terminal Server DCS Status Terminal Server Safety Status ACR Consoles Domain Controller DB Server (Config. ) DB Server (Archive) File Server Information Server DCS network Worker Nodes Network and Security Alarms FSM General DCS Electricity & Racks Infrastructure, network Racks, network, cabling KVM switches controlled power plugs Gas & Cooling LHC & Access General services Environment Alignment Detector DCS Spares Hot swap spare servers Disks & memories 8

TPC(1) 26/11/04 Control room (ACR) [FSM? ] Database(s) PVSS II ACC provides: Backend + Hardware and Software infrastructure User interface DETECTORS provide: Devices + Detector applications OPC client DIM client Ethernet PVSS II OPC client ISEG OPCserver Wiener OPCserver ELMB OPCserver C PCI-CAN C 1. . 3 DIMclient PCI-CAN C 1. . 22 OPC client PVSS II ELMB OPCserver RS 232 PCI-CAN E 1. . 18 [FED] PCI-CAN PVSS II 3 ISEG S DIMsrv C 1 1? 1 DIMcl Heinzinger ? Eth. Switch E ? 22 Wiener Eth. Switch ? E 18? HV 288 LV 2 HV ELMB 108 1 ELMB 108? Chambers High Voltage Chambers Low Voltage Chambers, detector Temp. Monitor ? DDL LV monitor ~400? 216 DIMsrv RCU FEE HVrod Very High Voltage VHV Curr. 9

Coordination in purchase of equipment u Purchased u u u by ACC Computers u Servers for back-end u PC’s for back-end and front-end (~90) u all details on DCS web page: http: //alicedcs. web. cern. ch/Alice. DCS/Meetings/ u PCI’s: Kvaser, Peak, Caen PCI-VME bridge, MXI Network u Global network (~400 ports), sub-contracted to IT/CS u On-detector network (~1000 ports) by ACC Devices u ELMB’s (~150) + ELMB power u Serial device (RS 232) 10

Serial devices (RS 232) u Aim for standard way of interfacing this class u u u Physical interface (cables and alike) u Needs a COM port on a PC; cable length is limited u A standard (and transparent !) way to use serial over Ethernet overcomes this limit u CERN standard; expertise exists u Examples exist, several detectors start using it u Exist in single and multi-port version Software interface (transporting the data) u PVSSII has all one needs to ‘drive’ a serial connection u Examples exist Functional interface (exchange sensible data) u Application dependent u to be developed by the detector (using the above!) 11

Coordination in purchase of equipment u Coordinated u u u by ACC LVPS u Wiener PL 512, water-cooled (6 detectors ~115 units) u Caen Easy (7 detectors ~110 units) u Caen SY 1527 (2 detectors, 1 units) u AREM Pro (1 detector, 6 units) HVPS u ISEG, 4 module types (4 detectors, ~15 crates) u Caen SY 1527 (13 detectors, ~20 main-frames) VME u Wiener 6 U & 9 U, local PS, water & air cooled (15 detectors, ~60 crates) 12

Detector computer distribution u Strict separation per detector, to allow for: u u u Staged installation Independent operation Distribution of sub systems based on: u u Number of channels Estimated resources needed by a sub system Performance tests done by several groups Some specific requests 13

Detector computers Operator Node - ON Worker Node - WN Front-End Dev - FED SPD SDD SSD TPC TRD TOF Operator Node Operator Node HV + LV HV HV HV FED + Crate LV + FED + Crate + ELMB LV LV FED FED VHV FED + Crate FED FED [18] Pulser FED Laser Drift velocity FED HMPID PHOS CPV Muon Trk Muon Trg FMD Operator Node Operator Node HV + LV HV + FED + LED HV + LV + ELMB HV HV + LV + FED Crate + PLC LV + ELMB + Crate LV Crate + ELMB FED Crate + ELMB +GMS T 0 V 0 PMD ZDC ACORDE EMC Operator Node Operator Node HV + LV + Crate HV + LV + FED + Crate + Laser FED Crate + ELMB FED 14

PVSS & FSM distribution u PVSS u u Baseline approach u Each detector DCS will be a distributed PVSS system u Each WN will run a single PVSS project u when several sub systems are controlled from a single WN these shall be grouped in a single project u All detector DCS’s will form one big distributed system (of distributed systems) FSM u integrated in the respective PVSS projects u One PVSS project can have several FSMs u Detector DCS FSM’ to run on least loaded node 15

Training & Tutorials u JCOP courses u u u DCS workshops u u Are appreciated At least one person from each detector has now taken the basic course Problem: manpower changes, students graduating Make handouts available ALICE “tutorials” DCS week planned u JCOP advanced course + ALICE tutorials 16

ALICE DCS week u Program (preliminary) u u u Monday u Framework tools & components; recap IT/CO Tuesday u Alarm handling IT/CO u Configuration Dbase IT/CO u Access control IT/CO Wednesday u FSM IT/CO + ALICE Thursday u DIM IT/CO u Framework devices IT/CO u Conditions ALICE u Distributed systems ALICE u ALICE Front-End Device (FED) ALICE u ALICE FEE configuration ALICE Friday u ALICE DCS guidelines/standards/rules ALICE u Detector specific applications & AOB ALICE hours 8 2 5 1 8 1 1. 5 1 2 6 17

Detectors H T S T T M M P A P S S F Z C E T V M P P R O T T M C H D S M D P M 0 0 P C D D F K G D O O D D D C V C User requirements Use of tools & components Basic device control FSM for at least one subsystem Test with ECS/DAQ All subsystems ready done/used not done/not used 18

JCOP u u u u ALICE Most Fw components + DSS system + GAS control Advice, assistance: PLC applications, OPC tests, etc Participation in ALICE DCS workshops u ALICE JCOP ISEG component BCAM component Cooling control Terminal Server evaluation Performance testing u Caen HV u PVSS scaling u Oracle archiving 19

Main scheduled “challenges” before LHC start-up u Pre-commissioning u u on surface at P 2 Functional verification of DCS including u 2 detectors with final devices u all main sub-systems and services Final validation of ALICE approaches for u Back-end, FSM, configuration, archival, alarms u Software & hardware interlocks u Interface with ECS, DAQ, TRG, HLT 20

Pre-commissioning at P 2 DCS network ON TPC WN WN WN sensors actuators DSS PC room PLC GWN Console ACR Console HMPID DSS BE ON WN DCS Back-end App GW DB-SRV Disk SRV starting mid April SG CERN Backbone network - DCS pre-commissioning 06 - All TPC and. ON HMPID & B-end DCS DC parts ready ACORDE MUON CR 3 WN DSS ON WN CR 4 SXL 2 sensors actuators 21

DCS final installation - procedure u H/W and S/W infrastructure u u ACC install and verify network, PC’s (with PCI), and base s/w Detector hand over to ACC u u S/W projects u the detector PVSS project u 3 rd party software with documentation and installation kits u Front-end configuration and control tools Documentation (stored on DCS web pages) u Description of the PVSS hierarchy u Assignment of the managers to the detector computers u Configuration of the OPC servers (where they should run, what devices should be accessed) u FED servers (where they should run) u archival – what parameters are archived u parameters to be stored in the conditions database (including the datapoint names) u FERO configuration description 22

DCS final installation - procedure u Verification u u Installation u u ACC checks for obvious errors u incorrect path names, incorrect external dependencies, violations of the DCS conventions such as naming of the systems or DIM services etc. For viruses For completeness of documentation ACC uploads software to the installation server on DCS network ACC + Detector expert install on target machines u setting-up of the PVSS system u integration into global DCS, u configuration of database access and FW access control back-up Commissioning u u Detector expert validates functionality with devices Detector expert demonstrates functionality of alerts, operational panels, FSM, etc. 23

Detector final installation planning DETECTORS From (D time) 10 July 1 Sept TPC 4 Sept 2 Oct SDD, SSD (ITS Barrel) 3 Oct 8 Nov FMD, T 0, V 0 on C-side 9 Nov 28 Nov 29 Nov 10 Jan 07 PHOS, TOF, TRD, HMPID, ACORDE SPD, SDD, SSD TOF, TRD FMD, T 0, V 0, PMD on A-side To 31 Jan 07 6 Mar 07 4 Apr 07 23 Apr 07 24

DCS final installation planning For each sub-system (in total ~ 100): Pre-commissioning on surface (institute/CERN) who u PVSS - OPC – h/w device connection AA u FSM, configuration, archival, alarms BB u s/w interlocks, operations panels CC Commissioning in the experimental area u s/w hand-over to ACC u Installation & commissioning of devices u Commissioning devices with detector who AA BB CC D ready date ready <D-4 weeks <D-2 weeks D = detector installation date 25

Final installation Apr May TODAY NETWORK PC + BASE SYSTEM CONTROL CABLES AND DEVICES DCS PROJECT INST D June Hand-over Mar D- 2 D- 4 weeks COMM INST DETECTOR COMM 26

Final installation: PHOS, TOF, TRD, HMPID, ACORDE Apr CONTROL CABLES AND DEVICES DCS PROJECT June Latest TODAY NETWORK PC + BASE SYSTEM May INST COMM 12 Jun 26 Jun 10 Jul Hand-over Mar INST COMM INST DETECTOR COMM 27

DCS commissioning Contr Room Consoles responsibilities Safety CSAM CR facilities HLT LHC DCS Back-End Chochula DCS H/W Infrastructure Augustinus Offline Interface DAQ De Cataldo Trigger Augustinus ECS Gas Augustinus Chochula EXTERNAL SYSTEMS Services DSS Interface Augustinus + Chochula Detectors SPD SDD SSD TPC TRD TOF HMP PHO FMD T 0 Power Cooling Magnets Access Infrastr B-field Space Frame Env. Param. Radiation V 0 PMD MTI MTK ZDC ACO ~100 sub-systems 28

Testing and validating DIP u First u use/commissioning of DIP CSAM data MCS data DCS data u Assistance DCS LHC pre-commissioning Magnet tests final commissioning May 06 end 06? from JCOP? 29

Integration with other on -line systems u DCS u u ECS DAQ commissioning Done for 2 detectors (HMPID and SPD) Planned for TPC during pre-commissioning June 06 30

Global status today (“dashboard” on DCS website) SPD SDD SSD TPC TRD TOF HMP PHO CPV ID S µtrk µFMD trg T 0 V 0 PMD ZDC ACO EMC Trig. DCS DET HV CAEN iseg CAEN Easy iseg +distr CAEN Iseg +V reg. CAEN +distr CAEN CAEN LV CAEN Easy AREM pro CAEN Easy Wiener ALICE box CAEN Easy Wiener CAEN Easy +distr VME Wiener VME Wiener VME Wiener VME PLC ELMB MON FED JTAG DCS ELMB JTAG RCU DCS DRM ELMB RCU CAEN DRM Iseg CAEN Easy Wiener VME ELMB RCU CAEN Wiener VME ? DCS DIM ELMB RCU Cool GAS VHV Pulser Liquid LED GMS Crystal Thresh Laser Position LED? Rack. Ctrl Align Laser BField Drift Vel. Rad. Mon Control with FSM Basic control with PVSS Activity started, without PVSS Under definition No (SW) activity started 31 No (SW) activity started, but common effort

5 “critical” detectors SPD SDD SSD TPC TRD TOF HMP PHO CPV ID S µtrk µFMD trg T 0 V 0 PMD ZDC ACO EMC Trig. DCS DET HV CAEN iseg CAEN Easy iseg +distr CAEN Iseg +V reg. CAEN +distr CAEN CAEN LV CAEN Easy AREM pro CAEN Easy Wiener ALICE box CAEN Easy Wiener CAEN Easy +distr VME Wiener VME Wiener VME Wiener VME PLC ELMB MON FED JTAG DCS ELMB JTAG RCU DCS DRM ELMB RCU CAEN DRM Iseg CAEN Easy Wiener VME ELMB RCU CAEN Wiener VME ? DCS DIM ELMB RCU Cool GAS VHV Pulser Liquid LED GMS Crystal Thresh Laser Position LED? Rack. Ctrl Align Laser BField Drift Vel. Rad. Mon Control with FSM Basic control with PVSS Activity started, without PVSS Under definition No (SW) activity started 32 No (SW) activity started, but common effort

15 “critical” subsystems SPD SDD SSD TPC TRD TOF HMP PHO CPV ID S µtrk µFMD trg T 0 V 0 PMD ZDC ACO EMC Trig. DCS DET HV CAEN iseg CAEN Easy iseg +distr CAEN Iseg +V reg. CAEN +distr CAEN CAEN LV CAEN Easy AREM pro CAEN Easy Wiener ALICE box CAEN Easy Wiener CAEN Easy +distr VME Wiener VME Wiener VME Wiener VME PLC ELMB MON FED JTAG DCS ELMB JTAG RCU DCS DRM ELMB RCU CAEN DRM Iseg CAEN Easy Wiener VME ELMB RCU CAEN Wiener VME ? DCS DIM ELMB RCU Cool GAS VHV Pulser Liquid LED GMS Crystal Thresh Laser Position LED? Rack. Ctrl Align Laser BField Drift Vel. Rad. Mon Control with FSM Basic control with PVSS Activity started, without PVSS Under definition No (SW) activity started 33 No (SW) activity started, but common effort

Remarks u u u u u If not started on HV and LV, then mainly because of no hardware (waiting for PS) LV wiener, when application exist, it is with CANbus version of PL 500 For several detectors the environment monitoring is still under definition (number/type of sensors etc. ) For HV/LV/ELMB detectors can profit from framework etc. For FEE more effort is needed from detectors. Thus red or yellow in FEE is more worrying than red in HV/LV For Crate control, cooling and gas an effort is expected from the ACC (thus less work for detectors) The FEE for TOF, T 0 is green/yellow: work on SW has started, some PVSS panels exist, but it need to be clarified if this adheres completely to the FED concept The FEE for FMD is yellow/red: They started to use the RCU, but not really using the ‘DCS part’ of it The Laser for TPC is green/yellow: all parts of the system have been worked on, not all under PVSS The Vo is indicated mainly red because of lack of information The environment monitoring for EMC is not yet defined. There will be a LED calibration system, the controls of it need to be clarified. EMC has experience with PVSS (older version of their FEE), but not with the sub-systems listed here 34

Progress monitoring u For u Status dashboard on web u For u Public Experts Status panel on DCS top node accessed via Terminal Server 35