ICS update Henrik Carling Head of ICS division

ICS update Henrik Carling Head of ICS division www. europeanspallationsource. se 2018 -04 -06

The ESS integrated control system • The ESS facility is a large and complex machine with very much and diverse equipment that needs to work in synchronization and with well-known configurations • The Integrated Control System Division (ICS division) is responsible for the control systems within the ESS facility including controls for – – • Accelerator Target Neutron Scattering Systems Conventional Facilities In addition, the control system makes it suitable for implementing – Machine Protection System – Personnel Safety System • To build a distributed control system of this size is a major undertaking 2

ESS/ICS challenges • The ESS control system complexity is very high – About 1 600 000 “process values” – Ambitious approach to automation of control system configuration • Facility availability goals are very high – ICS plays a key role for the availability of the facility - 95% availability – High performance and availability requirements on equipment used • Some new approaches will be implemented at ESS/ICS – Full scale deployment of EPICS 7 • ESS is committed to contributing to the EPICS community – Full scale deployment of Micro. TCA. 4 • ESS is involved in a public procurement for innovation initiative

ICS Project scope Work package 01 Work package 03 Work package 04 Work package 05 Work package 06 Work package 07 Work package 08 Work package 09 Work package 10 Work package 11 Work package 12 Work package 13 Work package 14 Work package 20 Management and administration Software core Hardware core Machine protection Equipment Control system infrastructure Physics Personnel safety systems Integration - Accelerator Integration - Target Integration - Instruments Integration - Conventional facilities Test Stands Installation 4

ICS Organization 2018 -04 Henrik Carling Division head Remy Mudingay Controls infrastructure Alessio Curri System administrator Benjamin Bertrand Anna Gillberg Team assistant Alberto Toral Technician Denis Paulic Software engineer DGL, PLC engineer Johan Christensson Fernando Carrasco Infrastructure technology engineer Annika Nordt Safety and protection Karl Vestin Hardware and integraion David Sanchez Angel Monera Automation engineer FPGA Engineer Enric Bargalló Faye Chicken Lead analyst engineer Technician Manuel Zaera-Sanz Javier Cerejo Technician Lead engineer slow interlocks Ph. D Student Morteza Mansouri Safety systems engineer Mattias Eriksson John Sparger Network administrator Technician Integrator Anders Harrisson Stephane Gabourin Riccard Andersson Peter van Velze Susann Skarin Software configuration manager Fredrik Luthander Software engineer Peter Holgersson Electriocal engineer Lead engineer fast interlocks Szandra Kövecses Lead integrator Yong Kian Sin IEC 61508 engineer Position under consideration Technical coordinator Technician Stuart Birch Saeed Haghtalab Hector Novella Susanne Regnell Timo Korhonen Deputy project manager Benedetto Gallese Integrator François Bellorini Juan Esteban Müller Simone Farina Software engineer Jan-Åke Persson Senior software engineer Software scientist Karin Rathsman Accelerator scientist Ricardo Fernandes Control system software architect Integrator Nour Akel Integrator ICS Installation coordinator Oliver Talevski William Ledda Johannes Kazantzidis Open positions (4) Integrator Georg Weiss Software engineer Accelerator physicist Julen Etxeberria Junior controls engineer Mehdi Mohammednezhad Embedded engineer Software engineer Fredrik Söderberg Philippe Rabis Integrator Dirk Nordt Claudio Rosati Joao Martins Tomasz Brys Miklos Boros Software engineer Emanuele Laface Integrator Chief engineer Ph. D student Jeong Han Lee Integrator Systems engineer Banafsheh Hajinasab Ben Folsom Software engineer Thomas Fay Integrator Controls Software Integrator Viktor Fred Lead engineer electrical installations Michael Beck Consultant off-site Thilo Friedrich Embedded systems egineer Marino Vojneski Temporary employee Maria Romedahl Integrator Work package manager Employee Technical coordinator Senior engineer Work package manager Consultant Wojtek Fabianowski in-kind manager Integrator ESS-0081625 Consultants

Layer Architecture of the ESS Control Systems Human Users Engineers, Operators, Scientists, Technicians, . . . Applications GUIs for data analysis, processing and archive services, system configuration, . . . send/receive ESS facility data Facility Integration runtime & historical data, complex control functions, configuration databases Local System Control Physical Equipment System GUIs Other Human Operations Facilities Experiment workstations Equipmet GUIs Physics Apps post mortem analysis Config. tools PSS GUIs EPICS (Ethernet) Network equipment EVG, Master Oscillator system specific data exchange high dependability, executes local control functions Signal Conditioning (a/d) Equipment Electronics ESS Main Control Room display ESS facility data, task execution EPICS IOC EPICS IOC EPICS Archive Timing System Machine Protection: Beam Interlock System FBIS Personnel Safety System Ether CAT Micro. TCA system remote I/O controller device electronics controller remote I/O flowmeter motor sensor chopper PS switch sensor PLC local control, run-time data PSS PLC Ethernet fieldbus Device signals device specific signals pump machinery, sensors, actuators sensor measure and adjust Physical Processes H 2 O Flow He Flow Mech. Movement R F Proton Neutron Temp Electricity etc.

ICS overview - available soon • A description of ICS concepts, architecture, technologies, organization and more will be available by 2018 -04 • An overview document with many references to more detailed documentation 7

Working together 8

Major Milestones for Control System Development Requirements & Analysis System Design Requirements Detailed Design (detailed) Controls design completed based on completed overall system design • Integration and Test Integration completed Commissioning operational readiness Production Operation Control system in operation good enough to work with in Production Final optimisations, user friendliness etc. Development frozen Terminology as defined by the ESS Handbook for Engineering Management [ESS-0092276] the local control system design is complete and the integration into the global control system is planned At the integration completed milestone – – • Development Commissioning site acceptance At the controls design completed milestone – • Installation Control system development stages (green) are shown in relation to the overall system’s development stages (blue) – • Manufacturing & Preparing for Installation the local control system parts (such as a PLC system) are integrated into the global control system The overall system commissioning can start from a remote control room At the control system in operation milestone – – the overall system can be operated with all specified functionality from a remote control room No further construction activities are needed 9

Prioritizing resources Integrated Control Systems • ICS division is responsible for the construction of the integrated control system • Some parts of the ICS division can work relatively independent of stakeholders, while most of ICS will engage heavily with stakeholders to design and integrate systems • Most significantly, this applies to the hardware and integration group, who will almost exclusively work in close collaboration with stakeholders • The hardware and integration group can in total distribute about 25 engineers and 5 technicians to work together with stakeholders • The main stakeholders are Accelerator, Target, NSS and CF - each has an ICS integration work package in the ICS construction project • In order for ICS resources to be able to cover stakeholder needs, a systematic allocation of resources needs to be implemented • ICS will implement a variant of a classical line/project matrix model in order resolve prioritization and support project planning 10

Line/project matrix - motivation (2) • The line/project matrix model solves the problem of running multiple project activities with limited resources, keeping a correct prioritization that supports stakeholders project planning • It also supports the continued development of experience and competence in the line organization Step 1: Collecting requests • work package manager interacts with stakeholders and together defines what needs to be done and when • work package manager and stakeholder defines a resource estimate with a time envelope • work package manager brings the plan back to the ICS allocation board Target WP 5 Acc. WP 8 ESS Bilbao Acc. WP 7 Control system architect 1 1 0. 5 Industrial automation 2 EPICS 0. 5 1 1 0. 5 Data networks 0. 5 Analog electronics Control System Studio 0. 5 Target WP 7 0. 5 FPGA 2 Micro. TCA 2 0. 5 0. 5 1 1 0. 5 Process control 2 0. 5 1 Vacuum controls 0. 5 Control system integration 0. 5 Controls technicians 0. 5 1 0. 5 0. 5 Information/Controls security . . . 1 Timing system Motion controls . . . 0. 5 1 Embedded systems NSS WP 5 1 1 11 . . .

Line/project matrix - motivation (2) • The line/project matrix model solves the problem of running multiple project activities with limited resources, keeping a correct prioritization that supports stakeholders project planning • It also supports the continued development of experience and competence in the line organization Step 2: Prioritizing requests • ICS allocation board prioritizes requests in discussion with stakeholders • Where conflicts occur, an escalation is made to ESS project management • The prioritization is roughly adjusted to fit the joint ICS resource + stakeholder capacity Target WP 5 Acc. WP 8 Control system architect 1 1 Industrial automation 2 EPICS 0. 5 ESS Bilbao 1 . . . 0. 5 Control System Studio 0. 5 FPGA 2 Micro. TCA 2 Timing system 1 0. 5 1 1 0. 5 2 1 0. 5 Vacuum controls Control system integration 0. 5 Controls technicians 0. 5 0. 5 Information/Controls security . . . 1 Analog electronics Motion controls Target WP 7 1 0. 5 Data networks Process control NSS WP 5 0. 5 1 Embedded systems Acc. WP 7 1 12 . . .

Line/project matrix - motivation (2) • The line/project matrix model solves the problem of running multiple project activities with limited resources, keeping a correct prioritization that supports stakeholders project planning • It also supports the continued development of experience and competence in the line organization Step 3: Allocation • ICS line management allocates resources to activities • ICS line management function considers factors such as competence and experience of the resources • ICS line management considers vacation, and other circumstances to guarantee the allocation within the period • ICS line management is responsible for regulating the line capacity if needed Target WP 5 Acc. WP 8 Control system architect 0. 5 Industrial automation 2 EPICS 0. 5 ESS Bilbao 1 . . . 0. 5 Control System Studio 1 FPGA 1 Micro. TCA 1 Timing system 0. 5 1 0. 5 1 1 0. 5 Vacuum controls Control system integration 1 Controls technicians 0. 5 1 0. 5 Information/Controls security . . . 1 Analog electronics Motion controls Target WP 7 1 0. 5 Data networks Process control NSS WP 5 1 1 Embedded systems Acc. WP 7 1 13 . . .

ICS work packages • The table lists ICS work packages • Mainly, work packages 10, 11, 12, 13 will follow the principles outlined here – – Work package 10: Work package 11: Work package 12: Work package 13: Philippe Rabis Michael Beck John Sparger (acting) Michael Beck 14

Current ICS priorities • High intensity areas – – – – ICS Contributions to SSM warm linac trial operation permit application Completing re-baselining towards the total ESS plan Commissioning of Ion source and LEBT (including PSS 0) RF Test stand 2 support Installation and deployment of PSS and control system infrastructure Beam instrumentation system support Alignment with target division on project processes and responsibilities Alignment with NSS division on project priorities and staffing 15

Some recent achievements and activities 16

Installation of control system infrastructure • Installation and deployment of control system infrastructure is progressing according to plan • Fibre, cabling, switches, routers, patch panels etc. are installed and commissioned 17

Installation of Personnel safety systems • ESS personnel safety systems are among the earliest systems that need to be in place before operating equipment • A lot of installation is ongoing, in particular for Oxygen deficiency hazard monitors 18

Controlling systems from the local control room • Recent achievements include controlling the pure Helium storage and parts of the ion source and LEBT from the local control room 19

Building control systems • Many components are being assembled into ICS subsystems and readied for installation • Meanwhile, work is continuing with maintaining and developing the ICS and EPICS software solutions 20

ICS In-kind partners Agreement signed Agreement waiting for signature Collaboration/other agreement

Examples of ongoing in-kind activities IFE control room contribution PSI IFC card family contribution Testing Tallinn University of Technology contribution An engineering team from Swiss inkind partner PSI visited ESS 22

Special focus ESS standard high-performance controls platform • • • The first production-level results from an intensive in-kind collaboration between Paul Scherrer Institute (PSI), Switzerland ESS are now materializing in the labs at ICS and its stakeholders This Swiss in-kind project is delivering the bulk of the ESS standard highperformance controls platform, an in-kind contribution with a 4+ M€ budget The ESS standard high-performance controls platform consists of a Micro. TCA based AMC [link] capable of carrying versatile FMC [link] modules for high flexibility in control system applications Seen on these pictures are IFC 1410 [link] Intelligent FMC Carrier AMC NXP Qor. IQ T 20811. 8 GHz with Alti. Vec Xilinx Kintex Ultra. Scale FPGA Dual HPC VITA 57. 1 FMC slots DESY D 1. 4 -compliant RTM interface ACQ 420 [link] 4 Channels ADC 16 bit at 1 MS/s DC coupled differential inputs Clock input ADC 3110 and ADC 3111 [link] 8 channels ADC 16 bit at 250 MS/s AC or DC coupling inputs Clock & user-defined inputs ADC 3112 [link] • • The AMC - IFC 1410 - has been developed in a collaboration of PSI, ESS and an industrial partner. The board comes with a full-featured FPGA design kit called TOSCA. The IFC 1410 is equipped with an NXP Qor. IQ T 2081 processor providing quad-core capability at up to 1. 8 GHz with moderate power consumption at ~14 W/1. 4 GHz. The IFC 1410 also has an on-board Xilinx Kintex Ultra. Scale FPGA and comes with a FPGA design kit that enables straightforward integration of FMC modules and rapid development of firmware applications 4 channels ADC 12 bit at 1 GS/s or 2 channels ADC 12 bit at 2 GS/s DC coupling inputs Clock & user-defined inputs ADC 3117 [link] 20 channels ADC 16 bit at 5 MS/s 2 channels DAC 16 bit at 1 MS/s Single ended or differential inputs Clock & user-defined inputs DAC 3113 [link] Dual channel ADC 16 bit at 250 MS/s Dual channel DAC 16 bit at 250 MS/s DC coupling Clock & user-defined inputs TOSCA FPGA Design Kit • A Linux BSP [link] adapted for EPICS applications on the AMC has been delivered by PSI that will be used at ESS for the distributed, state of the art control systems developed by ICS • More information about PSI [link], Micro. TCA [link] or the IFC 1410 [link] Firmware Design Kit for Kintex Ultra. Scale Up to three PCI Express GEN 3 blocks VHDL source code fully available Faster development time through: * User application focus * Access to industrial IP core library * User Area dedicated simulation * Including reference designs * High integration with EPICS systems

Re-baselining status 24

Milestones - Construction -> Initial Operations • The transition for ICS from Construction to Initial operations phase is defined by four major milestones – – ICS Components ready for initial operations ICS ready for Accelerator RBOT ICS ready for Target RBOT ICS ready for NSS Test Beamline 2018 -12 -21 2019 -06 -23 2020 -03 -31 2020 -10 -15 25

Explanation of Cost Changes • Current EAC for ICS is 9. 1 M€ 4. 0 M€ additional installation costs 3. 2 M€ staff delay costs (only accelerator and 9 months delay) 1. 9 M€ accelerator integration cost increase 0. 0 M€ target integration cost increase • New forecast for ICS is 17. 2 M€ 4. 8 M€ for additional installation costs Based on better (evidence based) estimates - only slightly affected by delays Installation costs reduced by using in-house staff in “delayed” segments 8. 3 M€ staff delay costs 2. 8 M€ accelerator integration cost increase 1. 3 M€ target integration cost increase 26

ICS vision and mission statements • ICS vision statement ESS operated efficiently, reliably and safely, with a control system that everyone loves • ICS mission The ICS division shall provide and maintain world-class and costefficient control, protection and safety systems and services for the ESS facility. The division shall develop competence and innovative solutions that can be shared in the community through open processes