Lessons learnt during Front End preparation and installation

Lessons learnt during Front End preparation and installation William Ledda Control System Integrator (ICS) on behalf of Thomas Shea (Beam Diagnostic Section Leader) www. europeanspallationsource. se 2018 -04 -06

Outline • • • Overview Organization & coordination Technical challenges Lessons learnt Conclusions 3

Overview • The "Injection system" – Ion source produce the beam – LEBT "transports" the beam till the RFQ for the first stage of acceleration • The "Control system" • – Infrastructure (HW and SW) that make it possible provide required data, information and services to engineers, operators, physicists and the facility itself. The "Integrated Control System Division (ICS)" – responsible of defining and providing control systems services for the entire ESS facility (accelerator, target, neutron scattering instruments and conventional facilities) 4

Outline • • • Overview Organization & coordination Technical challenges Lessons learnt Conclusions 5

Organization & Coordination Distributed team ESS INFN-LNS In-kind contributor for the ISRC & LEBT CEA In-kind contributor for the control system ESS ICS Division ACC Division INFN-LNS 6

Organization & Coordination Distributed team • Relations between the teams are very good, there are bonds between the different team members. • Communications, either through phone, videoconference or in person. • Frequent travels in Catania during for the installation, set-up and testing and acceptance of the system. 7

Organization & Coordination Distributed Development 8

Outline • • • Overview Organization & coordination Technical challenges Lessons learnt Conclusions 9

Technical challenges Device integration • Many different devices to be integrated (PLCs, Field devices, Power Supplies etc. ) • Different interfaces and protocols (Serial vs Ethernet, Modbus, SCPI, GPIB, Step 7, Ether. CAT, etc. ) • The “glue” that makes all these devices to communicate each other is the EPICS framework (Experimental Physics and Industrial Control System) adopted as standard by ICS – ~ 10 IOCs running – ~ 9500 EPICS PVs 10

Ion Source Controls Architecture Archiving PC Vacuum System 11 Server Scalance X 204 -2 (0) Source IPC Source PLC 4 Scalance X 212 -2 (1) Scalance X 204 -2 (2) CAT 5 Ethernet (10. 0. xxx, Channel Acces, boot, nfs, …) Serial, etc. CAT 5 Ethernet (10. 1. xxx) Fibre Electrical, etc. (physical I/O, power, etc. ) 5 Source Ether. CAT I/O 8 1 3 CU 1561 Scalance X 204 -2 (3) SCPI Repeller Electrodes PS (x 2) 2 Scalance X 212 -2 (4) 10 Scalance X 204 -2 (5) Modbus TCP 9 Modbus TCP Scalance X 212 -2 (6) Source I/O Controller ACQ Source Ether. CAT I/O Serial Server SCPI Profinet SCPI HV PS CU 1561 Magnetron ATU ASCII Mass Flow & Gauge Controller Source Coils PS (x 3) Source PLC Remote I/O HV Platform Controls Equipment (PLC, IPC, I/O, etc. ) LINAC Equipment (PS, etc. )

Serial, etc. LEBT Controls Architecture CAT 5 Ethernet Fibre ASCII 5 9 NPM MC 8 Electrical, etc. (physical I/O, power, etc. ) To Source PLC DSM IPC To Source PLC IRIS NPM To Source CB EVR DSM Spectrometer FC Chopper Camera FC Bias PS DSM BCM EMU To Isrc VME Control Box SCPI 2 SCPI 10 4 EMU Plate PS (x 2) ASCII EMU Ether. CAT I/O BCM I/O Controller EMU Bias PS Solenoid PS (x 2) Steerer PS (x 4) EMU MC 3 EMU I/O Controller EMU PLC s 7 plc ACQ To Isrc VME Control Box ASCII IRIS MC BCT RTM Interface Unit ACQ NPM IPC 1 EMU IPC Future Controls Equipment (PLC, IPC, I/O, etc. ) LINAC Equipment (PS, etc. )

Technical challenges Control GUI • Simplified GUI for the operator to have everything under control

Technical challenges Compromises • ICS Integration strategy tailored and adapted to the situation. – Processes and procedures not fully in place when ISRC&LEBT started – Calendar pressure advised to keep moving forward – Expertise of teams involved allowed to mitigate potential risks • EPICS v 3 instead of v 4 used for stability at the time. • Timing system deployed not the final one but required to run the machine. • Machine modes not defined, controls implementation left open to deploy them in the future. • VME architecture was used instead of µTCA. • Geobrick instead of Ether. CAT for motion control • Local Protection System includes also Safety functions required to operate the machine safely.

Technical challenges Achievements • ISRC & LEBT is a reality!!!! – ISRC Control system accepted in July 2017 – LEBT Control system accepted in September 2017 • All EPICS modules under control version, tested, documented. • All equipment are integrated in the control system.

Outline • • • Overview Organization & coordination Technical challenges Lessons learnt Conclusions 16

Lessons General • Structure, methodologies, processes and standards non fully defined since the beginning. • ISRC & LEBT seen as "test stand" more than a final system – Some systems need to be updated. • From VME to µTCA • Timing system • Motion control (Geobrick -> Ether. CAT) • Compression in time schedule led to take fast decisions on the fly – This require a structure inspired to the "agile" thinking. • Sometimes lack of communication between all parties – "Who is doing what? ", misunderstanding, delays, late deliveries. • Documentation is "spread" and sometimes not fully updated. • Some systems integrated but not fully tested in operational conditions.

Lessons Organization and coordination • The "human" factor cannot be underestimated: – A team spread between different countries is a challenge in terms of organization, and it can be very time consuming. – Different teams have different views and expectations. – Side by side work is very useful to keep good relations, to solve issues and to take decisions. • We have always to keep in mind how efficient the communication is - Are the meetings efficient and solve problems? - Do we need to have in regular meetings are they useful to improve the system understanding and speed up the process? - What things do we discuss when we have a crisis? Do we prioritize properly? • Even the strongest team player have a tendency to communicate less or try to solve problems on their own. 18

Lessons Installation • The installation should be seen as for the whole system, not my part and then your part. • The installation involves a huge number of decisions that should not be underestimated in terms of time. • First time a real system installation is happening in ESS and there are many things still to learn from. • Good communication between all parties involved is essential to drive the installation in the right way. 19

Lessons Continuous Improvement Example of continuous improvement through lessons learned Experience: • Beam-induced damage of the Emittance Measurement Unit Analysis: • Delays led to aggressive commissioning schedule and ad-hoc activities • System operated without complete verification Improvement: • Schedule - Include explicit float Molten tungsten - Emittance Measurement Unit • Conduct of operations – • Avoid ad-hoc interlock management • Measure ”safe” beam prior to increasing duty factor 20

Outline • • • Overview Organization & coordination Technical challenges Lessons learnt Conclusions 21

Conclusions • ISRC & LEBT are the first "big" achievements for ESS, result of a strong collaboration of different teams. • We have had issues, of course! We have to extract many good lessons from them in order to avoid the same mistakes. • ISRC & LEBT are the starting point from which ESS can improve methodology, structure, process and knowledge. • The synergy between all parties is essential to avoid decision has to be reconsidered after. • Documentation is important but also the communication between all stakeholders is essential in all phases of the project. 22

Questions? Thank you!