GSI Helmholtzzentrum fr Schwerionenforschung Gmb H Accelerator Availability

GSI Helmholtzzentrum für Schwerionenforschung Gmb. H Accelerator Availability: status quo and challenges ahead O Geithner ACC Operation Department GSI Helmholtzzentrum für Schwerionenforschung Gmb. H

Content - Calculation of the ACC availability in OLOG - PHYS RUN 2018: Operation Performance and Failure Statistic - ENG RUN 2018: Operation Performance and Failure Statistic - Challenges ahead GSI Helmholtzzentrum für Schwerionenforschung Gmb. H O. Geithner, ACC Operation 2

Calculation of the Machine Availability in OLOG GSI Helmholtzzentrum für Schwerionenforschung Gmb. H O. Geithner, ACC Operation 3

Beamtime Schedule 2018 GSI Helmholtzzentrum für Schwerionenforschung Gmb. H 4

PHYS RUN 2018 GSI Helmholtzzentrum für Schwerionenforschung Gmb. H 5

PHYS RUN 2018: Machines Availability Events Chart für jede Maschine fehlt noch GSI Helmholtzzentrum für Schwerionenforschung Gmb. H O. Geithner, ACC Operation 6

GSI Helmholtzzentrum für Schwerionenforschung Gmb. H 7

PHYS RUN 2018: Events Overview Failure 27% Beam setup 17% Retuning 1% IS Service 2% Beam on Target 52% GSI Helmholtzzentrum für Schwerionenforschung Gmb. H O. Geithner, ACC Operation 8

PHYS RUN 2018: General Failures Statistic [ms] GSI Helmholtzzentrum für Schwerionenforschung Gmb. H 9

G U G 3 B XI B 1 G Q U U H XH D 51 2 B R Q 2, G D 4 S G G U 11 2 U H Y 2 3 B MU Q I 1 2 D , 11 G GU G. . . U N 5 Y 2 B Q I 1 G D 12 U R . . . 3 G IP 1 U S N G 4 BB U A 3 G G 1 BA U U Y 2 A 3 1 Q D G BA 11 U G A 3 3 U BB Y 2 6 G U G QD H U 3 B H 1 B 1 G 2 B 2 H UH C 1 F un 2 Q L d Q 1 G E 2 G N a U N. . G 4 B. U X R 1 G FQ U S 4 D 21 G U M A 3 S G 4 H BB U 6 N hà GU 2 I ¤l H 2 G t P B eg R 2 e G l n U A ic. . G 2 BA. U X 2 A G FVV U H 1 S 2 Q G Q 1 U N 3 5 G G KX U U XI E 1 Q G 1 B D UR E 51 3 G G 3 IQ U U XI X 1 H G QD GU IQD U Y 5 51 , Y 5 52 Q G QD D 11 UX 11 G IQ U D 5 Y 5 2 G QD U XI 12 Q D 42 H U G [ms] 0 GSI Helmholtzzentrum für Schwerionenforschung Gmb. H 30 25 20 15 10 5 0 10 GUH 4 B 12, GUA 2 B. . . GUA 3 BA 3 Tunnelöffnung fÃ. . . GUH 2 BR 2 GTE 2 QT 12 Treiberröhrenwec. . . Piepseranlage GUL 5 DS 7 H GUS 4 BB 3, GUH 2 B. . . GUS 4 BB 4 GUH 2 BR 2, GUH 3 B. . . Windows Experime. . . GUL 5 DC 6_P GUA 1 BA 1 GUA 2 B Phasenbeziehunge. . . S 12 QS 2 D GTH 4 VV 1 S FRS-Vakuumfenster GTS 2 QT 11, GTS 2. . . GS 12 QS 1 F Timingsystem GUH 3 BB 1 GUH 2 QQ 11 GUH 2. . . Prallblech Dipol UZ. . . Diverse Geräte an. . . GUXIQD 51 Mikrowelle GS 12 MU 2 A GUS 4 BB 3 GUZEMU 6 / Prallbl. . . GUH 2 QQ 12 GUN 2 IG 220 V und 20 k. V GTK 3 MV 3, GTK 3 M. . . GS 11 MU 2 GTK 4 DG 2_P GTK 4 DG 1, GTK 6 D. . . FRS Targetbereich PHYS RUN 2018: Failures Statistic 250000000 20000 150000000 10000 50000000

ENG RUN 2018 GSI Helmholtzzentrum für Schwerionenforschung Gmb. H 11

ENG RUN 2018: Machines Availability GSI Helmholtzzentrum für Schwerionenforschung Gmb. H 12

ENG RUN 2018: Events Statistic Failure 15% Failure 27% Beam setup 53% Beam setup 17% Retuning 1% IS Service 0. 2% IS Service 2% Beam on Target 31% Beam on Target 52% ENG RUN GSI Helmholtzzentrum für Schwerionenforschung Gmb. H PHYS RUN 13

Calculation of the availability in the future GSI Helmholtzzentrum für Schwerionenforschung Gmb. H O. Geithner, ACC Operation - Possibilities for the accounting: - 1) Manually (OLOG) - 2) „Automatic“ (Arch. System) 14

Issues for the failure analysis Operation 2007 - 2016 Number of Failures 3500 3000 2500 lit qua r o o sis: p 2000 1500 1000 500 Ausfall - Hochfrequen Netzgeräte zsen. . . Ergebnis 3200 2399 aly An e r u il Fa 0 Ausfall - Quellen 730 ata d g n xisti e f o y Ausfall - Ausfall - Vakuum/Stru Sonstiges/Un Interlock/Sich Strahldiagno Steuerung Infrastruktur. . . klar erh. . . se 670 439 388 258 186 142 Ausfall - Operating 130 - OLOG: Update of the Failure root cause in OLOG Event after its clarification - Arch. System: intelligent Data Mining GSI Helmholtzzentrum für Schwerionenforschung Gmb. H O. Geithner, ACC Operation 15

Automated Availability Tracking: CERN <2012: generally poor data 2012: Availability Working Group (AWG) launched Goal to produce data for “avail-sim” – SLAC 2013: AWG proposed Accelerator Fault Tracker (AFT) to solve data issues “downtime tracker” – Fermilab – ARW 2011 “stick the numbers on a popular notice board” – P. Sampson – ARW 2013 • board” 2014: AFT launched 2015: AFT extensively used for data analysis Finding your “best bang for your buck” – D. Mc. Gilvery – ARW 2015 2016: AWG began LHC periodic reporting for LHC 2017: AWG began data capture for injectors Find your “Happy User Index” – A. Luedeke – ARW 2013 2017: Founding of Machine Availability and Reliability Panel – organization level GSI Helmholtzzentrum für Schwerionenforschung Gmb. H O. Geithner, ACC Operation 16

Automated Availability Tracking: DESY A new functionality in CS generates the statistics automatically according to predefined rules. The key components are: 1. Beam Operation Calendar for definition of the planned machine status. Defines the nominal machine state with 1 hour resolution, specifies the nominal beam parameters 2. The State Server to generate the actual machine state (10 Hz upgrade rate). Based on many predefined machine states: User. Run, User. Run out of specs, User. Run low beam current, Test Run, Maintenance, Studies etc 3. Central Alarm System collects alarms for each subsystem. Three types of alarm severity is pre-defined for every subsystem. If there is at least one FATAL alarm in one subsystem then the current ACC state is set to FAILURE. 4. Operation. History. Viewer to view the history of machine states and alarms for any time interval, zooming down to seconds. Accelerator availability and MTBF calculation for the selected time interval Offline state corrections are still frequently done if the machine state is incorrectly defined or incorrect alarm definition/severity. The goal is to minimize the offline corrections. It is an ongoing iterative process to identify incorrect states and alarms where machine coordinators and operation team are highly involved. GSI Helmholtzzentrum für Schwerionenforschung Gmb. H O. Geithner, ACC Operation 17

Definitions 2 Mean Time Between Failure (MTBF) – mean time between the start of 2 failures Mean Time To Failure (MTTF), Mean Time To Repair (MTTR) MTBF = MTTF + MTTR Operation Failure (diagnose, repair/ replace, heat up. . . ) MTBF Operation MTTR MTTF GSI Helmholtzzentrum für Schwerionenforschung Gmb. H O. Geithner, ACC Operation 18

Details on Accounting Total operation time consists of: - Targettime: ‚beam on target‘ time, when all required machines (in production chain) are running and the planed experiments are served with beam - Retuning: planed targettime interruption in order to improve the beam performance (mostly requirement from experiments) - Interruption (Failure): unplaned targettime interruption. Includes failure diagnose, repair/replace, heat up, conditioning, etc. . . - Ion Source Service: planed targettime interruption in order to exchange the cathodes - Beam Setup: planed targettime interruption in order to setup the maschine for the new projectile or new production chain - Stand by: planed targettime interruption when either a) experiments cannot take the beam or b) another machine in the production chain gets failure or c)ion source service takes place or d) retuning of the another machine in the production chain takes place GSI Helmholtzzentrum für Schwerionenforschung Gmb. H O. Geithner, ACC Operation 19

Details on Accounting As the machine availability represents the ratio of failure-free time (targettime) to the sheduled time, one has to subtract retuning, Ion Source Service, Beam Setup and Standby times from the Total time. Though the Beam Setup, Retuning and Ions Source Service time will not be reflected in the availability statistics, they shall be kept reasonably low. GSI Helmholtzzentrum für Schwerionenforschung Gmb. H O. Geithner, ACC Operation 20

Our goal MTTF vs Number of Failures 7000 300 6000 250 5000 200 4000 150 3000 100 2000 50 1000 b' 13 tri e b' 12 Be tri e b' 11 Be tri e b' 10 Be tri e b' 09 Be tri e b' 08 Be tri e b' 07 Be tri e b' 06 Be tri e b' 05 Be tri e b' 04 Be tri e b' 03 Be tri e b' 02 Be tri e b' 01 Be tri e b' 00 Be tri e b' 99 Be tri e b' 98 Be tri e b' 97 Be tri e b' 96 Be tri e eb Be tri e Be '9 5 0 b' 94 0 Statistics example from ESRF shows that investing man power and money into improvement of the availabilty pays back! Blue balks: Beam on target time (MTTF) left axis [0, 7000] increases Red line: Number of Failures right axis [0, 300] decreases GSI Helmholtzzentrum für Schwerionenforschung Gmb. H O. Geithner, ACC Operation 21