ESS EPICS Controls Architecture Remy Mudingay www europeanspallationsource

ESS EPICS Controls Architecture Remy Mudingay www. europeanspallationsource. se 2017 -10 -24

Outline • • Introduction to ESS/ICS Controls Architecture Technical Network Compute and Storage 2

3

Timeline for the ESS facility 2025 ESS construction complete 2067 Decommissioning 2014 Construction work starts on the site 2023 ESS starts user program 2009 Decision: ESS will be built in Lund 2019 2012 ESS Design Update phase complete 2003 First European design effort of ESS completed First beam on target 4

ESS is financed and built through membership Host Countries of Sweden and Denmark 47, 5% Construction 15% Operations In-kind Deliverables ~3% Cash Investment ~97% Non Host Member Countries 52, 5% Construction 85% Operations In-kind Deliverables ~ 70% Cash Investment ~ 30%

Construction investment CURRENT Sweden (member) Denmark (member) Germany (member) United Kingdom (founding observer) France (member) Italy (member) Spain (founding observer) Switzerland (member) Norway (member) Poland (member) Czech Republic (member) Hungary (member) Lithuania (future member) Estonia (member) Total (minus preconstruction) 35. 0 % 12. 5 % 11. 0 % 10. 0 % 8. 0 % 6. 0 % 5. 0 % 3. 5 % 2. 0 % 0. 95 % 0. 45 % 0. 25 % ~97 % Lithuania Czech Republic Poland Hungary Estonia To be determined Norway Switzerland Spain Sweden Italy France United Kingdom FUTURE Belgium (founding observer) Netherlands (founding observer) Greece (future observer) Latvia, Portugal, Finland tbd tbd Denmark Germany 6

Civil Construction – Time of the Groundbreaking September 2014

Spring 2016

Science village View from ESS looking towards MAX IV and Lund University 9

ESS ERIC BOARD OF DIRECTORS DIRECTOR GENERAL MACHINES DIRECTORATE SCIENCE DIRECTORATE PROJECT SUPPORT & ADMINISTRATION DIRECTORATE CONVENTIONAL FACILITIES ACCELERATOR NEUTRON INSTRUMENTS GENERAL SERVICES Operations, ES &H and QA TARGET NEUTRON TECHNOLOGIES HUMAN RESOURCES Communication SYSTEMS ENGINEERING SCIENTIFIC ACTIVITIES FINANCE INTEGRATED CONTROL SYSTEM SCIENTIFIC PROJECTS COMMUNICATIONS & EXTERNAL RELATIONS ENGINEERING & INTEGRATION SUPPORT DATA MANAGEMENT SOFTWARE CENTRE INFORMATION TECHNOLOGY LEGAL SUPPLY, PROCUREMENT & LOGISTICS

ICS Organization 2017 -10 Control System Infrastructure Remy Mudingay Controls infrastructure Anna Gillberg Team assistant Alessio Curri Alberto Toral System administrator Technician Benjamin Bertrand Denis Paulic Annika Nordt Safety and protection David Sanchez Henrik Carling Division head Daniel Piso Hardware and integration Angel Monera Automation engineer FPGA Engineer Enric Bargalló David Brodrick Integrator Wojtek Fabianowski in-kind manager Hector Novella Deputy project manager Benedetto Gallese Integrator Faye Chicken Maria Romedahl Thilo Friedrich Susanne Regnell Timo Korhonen Technical coordinator Controls Software Ben Folsom Ph. D student Emanuele Laface Software engineer PLC engineer Accelerator reliability Johan Christensson Manuel Zaera-Sanz François Bellorini Network Engineer PLC engineer Mattias Eriksson Technician Integrator Ph. D Student Senior software engineer Riccard Andersson John Sparger Joao Martins Karin Rathsman Susann Skarin Morteza Mansouri Network administrator Safety engineer Anders Harrisson Software enigneer Stephane Gabourin Interlocks engineer Fredrik Luthander Szandra Kövecses Software engineer Peter Holgersson Electrical engineer Start 2017 -11 Technical coordinator Integrator Stuart Birch Jeong Han Lee Senior safety engineer Integrator Technician Javier Cerejo Integrator Julen Etxeberria Junior controls engineer Viktor Fred Peter van Velze Technical coordinator Lead electrical engineer Technician Embedded systems egineer Yong Kian Sin Manorma Kumar Tomasz Brys Krisztián Löki IEC 61508 engineer PSS analyst Open positions (1) Integrator Michael Davidsaver Senior Engineer Miklos Boros Employee Consultant Temporary employee Consultant off-site Integrator Mehdi Mohammednezhad Integrator Nour Akel Engineer ICS Installation coordinator Oliver Talevski William Ledda Embedded engineer Position under consideration Simone Farina Integrator Open positions (5) Systems engineer Chief engineer Claudio Rosati Software engineer Fredrik Söderberg Accelerator physicist Senior software engineer Jan-Åke Persson Juan Esteban Müller Software scientist Ricardo Fernandes Senior scientist Senior software engineer Consultants Open positions (2)

Group Structure Line and Project ESS Staff Control System Infrastructure Group • Group Leader - Remy Mudingay Contractor In-Kind partner Work Unit 14. 07. 01 Control Room • Remy Mudingay (Coordinator) Work Unit 14. 07. 02 Data Centre • Alessio Curri (System Administrator) Work Unit 14. 07. 03 Control System Network • Susann Skarin (System Administrator) • Johan Christensson Work Unit 14. 07. 05 Infrastructure Applications • Benjamin Bertrand (Software Engineer) (Infrastructure Engineer) • System Administrator (Vacant) Institute for Energy Technology: • S. Collier • M. Rosenqvist • P. Le Darz Control Room Institute for Energy Technology • Terje Bordal • Thor-Egil Ekeli Equipment • Peter Holgersson (Electrical Engineer) • (Software configuration manager) • Tallinn Institute of Technology • Dr Andro Kull • Dr Rain Ottis – • Dr Hayretdin Bahşi • Dr Olaf Maennel • Dr Risto Vaarandi Cybersecurity Fredrik Luthander Anders Harrisson (Software configuration manager) 12

13

Configuration Workflow Web applications /opt/epics/modules/ ├── afg 3000 │ └── 1. 0. 0 ├── asyn │ └── 4. 21. 0 ├── calc │ ├── 3. 3. 0 │ └── mudingay ├── environment │ └── 1. 0. 0 ├── loki │ └── 1. 0. 0 ├── motor │ └── 6. 8. 1 ├── nds │ └── 2. 3. 1 ├── pci 9116 │ └── 1. 1. 1 ├── pcidask │ └── 1. 0. 0 ├── s 7 plc │ └── 1. 1. 0 NFS +nfsroot CS Studio Archive Appliance BEAST 14

Control System Software Architecture Scripting Open. XAL CS-Studio CR Applications pv. Access as “data bus” • Protocol Buffers Archiver IOC RF 2017 -01 -11 HDF 5 Data. Store (HDF 5) IOC BI Mongo. DB CCDB Channel Finder Configuration Data Broker IOC Magnets IOC Vacuum • • All data accessible through a single API Applications and IOCs isolated from internal changes in services. Offline access via different methods, e. g. WEB/RESTful services, HDF 5 access APIs, etc. 15

Device Registration 1 CCDB 16

Device Registration 2 Cable DB IOC Factory 17

ROOTNFS File structure • /opt/epics/bases – base-3. 15. 2 – base-3. 15. 4 • • • LICENSE Makefile README /bin /configure /dbd /documentation /extensions /html /include /lib /src /startup /templates LICENSE Makefile README /bin /configure /dbd /documentation /html /include /lib /src /startup /templates /opt/epics/modules Runtime • /etc/systemd/system/multi-user. target. wants/ess-boot. service • /usr/bin/proc. Serv --allow -f -L /var/log/proc. Serv/out-SI-H 05_Ctrl-IOC-01 -i -c /var/run/proc. Serv/SI-H 05_Ctrl-IOC-01 2000 /opt/epics/modules/environment/2. 0. 0/3. 15. 4/bin/centos 7 -x 86_64/iocsh /opt/startup/ioc/ics-h 05/SI-H 05_Ctrl-IOC 01/production/st. cmd -n SI-H 05_Ctrl-IOC-01 • /bin/bash /opt/epics/modules/environment/2. 0. 0/3. 15. 4/bin/centos 7 -x 86_64/iocsh /opt/startup/ioc/ics-h 05/SI-H 05_Ctrl-IOC 01/production/st. cmd -n SI-H 05_Ctrl-IOC-01 • /opt/epics/base-3. 15. 4/bin/centos 7 -x 86_64/soft. Ioc -D /opt/epics/base-3. 15. 4/dbd/soft. Ioc. dbd /tmp/iocsh. startup. 3529 18

19

LINAC TARGET Neutron Instrument Inter-gateway High-level Network Overview VACUUM System Timing System Interface Cameras EPICS Network Other Network TECHNICAL NETWORK VLANs 20

Network Access simplified DTLS/TLS or IPSec IKEv 2 + HTTP/S Multi-factor authentication VPN Read OPI Reverse proxy (WAF) External Networks GPN DMZ SSL/TLS RDP/SSH/NX EPICS PV gateway Reverse proxy (WAF) Internal DMZ PV Access Logbook/Archiver/CCDB/Naming/Cable. DB/Alarms PV Access TN Control System Operation PV Access EPICS PV gateway TN Main Control Room Read/Write OPI PV Access EPICS IOC TN PV access 21

22

High-level Compute and Storage Phase 1 23

High-level Compute and Storage Phase 2 Open. Stack Ceph 24

Creating a virtual machine 25

Establishing a connection to a virtual machine 26

Archive Appliance User - read Archiver Appliance Short-term storage RAMDISK Short-term storage (NVMe) Archiver Appliance Short-term storage RAMDISK Writes Reads Writes Archiver Appliance Load balancer Short-term storage RAMDISK Short-term storage (NVMe) Long-term storage (Spinning media or Tape) Short-term storage (NVMe) 27