Failure Tracking at GSI Status Quo GSI Helmholtzzentrum
- Slides: 14
Failure Tracking at GSI Status Quo GSI Helmholtzzentrum für Schwerionenforschung Gmb. H
OLOG § All failure tracking is presently done manually by the operators during the shift work § The Tool is our Operations Log Book (OLOG) § databased web service § interfaces to shift planning tool, to beam time schedule and partially to LSA settings data of the control system § Data to enter: § all events (event type, start- and end-time) § main accelerator-chain properties (running experiments, ion species, energy, some settings, intensity reductions, . . . ) § transmission data are entered nearly once per shift GSI Helmholtzzentrum für Schwerionenforschung Gmb. H Automated Failure Tracking - Stephan Reimann 19. 09. 2021
OLOG. acc. gsi. de GSI Helmholtzzentrum für Schwerionenforschung Gmb. H Automated Failure Tracking - Stephan Reimann 19. 09. 2021
one page per shift (8 h) one status per experiment GSI Helmholtzzentrum für Schwerionenforschung Gmb. H Automated Failure Tracking - Stephan Reimann 19. 09. 2021
each event has to entered manually GSI Helmholtzzentrum für Schwerionenforschung Gmb. H Automated Failure Tracking - Stephan Reimann 19. 09. 2021
Event categories and sub categories are predefined GSI Helmholtzzentrum für Schwerionenforschung Gmb. H Automated Failure Tracking - Stephan Reimann 19. 09. 2021
weaknesses with present system § Additional workload on the operators § Accuracy depends strongly on the discipline and knowledge of the operators § in challenging shifts with high workload some events might be forgotten to enter or entered afterwards from memory § root cause is not always clear from the beginning, failures are then assigned to the wrong technical system § frequently appearing short events are sometimes ignored, sometimes summed up and entered as one event § no beam quality (no user happiness index) GSI Helmholtzzentrum für Schwerionenforschung Gmb. H Automated Failure Tracking - Stephan Reimann 19. 09. 2021
What do we get out of the system § Overall Availability for Experiments § Specific Availability for one Experiment § Machine-Availability § Root Cause statistics (by device, by system, by department, by impact) § Transmission data § output is a CSV-file, manual postprocessing is necessary GSI Helmholtzzentrum für Schwerionenforschung Gmb. H Automated Failure Tracking - Stephan Reimann 19. 09. 2021
Examples 11% 9% beam on target interruption 2019 physics run setup + adjust + source changes machine availability UNILAC Users UNILAC SISTransfer SIS 18 HEST 87% 99% 90% 80% GSI Helmholtzzentrum für Schwerionenforschung Gmb. H Automated Failure Tracking - Stephan Reimann 19. 09. 2021
experiment specific availability 2019* *preliminary evaluation (clean-up of raw data outstanding) GSI Helmholtzzentrum für Schwerionenforschung Gmb. H Automated Failure Tracking - Stephan Reimann 19. 09. 2021
PHYS RUN 2018: General Failures Statistic Ausfall - Vakuum / Strukturen Ausfall - Strahldiagnose Ausfall - HF / Sender Ausfall - Netzgeraete / Hardware Ausfall-Steuerung / Frontendsoftware Ausfall - Quellen / EZR Ausfall - Netzgeraete / Datenübertragung Ausfall - Sonstiges / Unklar Ausfall - Netzgeraete / Interlock [h] Ausfall - Strahldiagnose / Detektor-Infrastruktur 0 20 40 60 80 100 120 140 160 Ausfall - HF / Sender Ausfall - Netzgeraete / Hardware Ausfall - Magnete Ausfall - Vakuum / Strukturen Ausfall - Netzgeraete / Interlock Ausfall - Quellen / SÃœD Ausfall-Steuerung / Frontendsoftware Ausfall - Quellen / EZR Ausfall - Sonstiges / Unklar Ausfall - Netzgeraete 0 10 GSI Helmholtzzentrum für Schwerionenforschung Gmb. H 20 30 40 50 O. Geithner, ACC Operation 60 70 80 90 11
G U G 3 B XI B 1 G Q U U H XH D 51 2 B R Q 2, G D 4 S G G U 11 2 U H Y 2 3 B MU Q I 1 2 D , 11 G G G UN. . . U Y 2 5 B Q I 1 G D 12 U. R 3. . G IP 1 U S N G 4 BB U A 3 G G 1 BA U U Y 2 A 3 1 Q D G BA 11 U 3 G A 3 B U Y 2 B 6 G U G QD H U 3 B H 1 B 1 G 2 B 2 H UH C 1 F un 2 Q L d Q 1 G E 2 G Na U N. . G 4 B. U X R 1 G FQ U S 4 D 21 G U M A 3 S G 4 H BB U 6 N hà GU 2 I ¤l H 2 G t. P B eg R 2 e G ln U A ic. . G 2 BA. U X 2 A G FVV U H 1 S 2 Q G Q 1 U N 3 5 G G KX U U XI E 1 Q G 1 B D UR E 51 3 3 G G IQ U U XI X 1 H G QD GU IQD U Y 5 51 , Y 5 52 Q Q D GU D 1 11 X 1 G IQD U Y 5 52 G QD U XI 12 Q D 42 H U G [ms] 0 GSI Helmholtzzentrum für Schwerionenforschung Gmb. H 30 25 20 15 10 5 0 12 GUH 4 B 12, GUA 2 B. . . GUA 3 BA 3 Tunnelöffnung fÃ. . . GUH 2 BR 2 GTE 2 QT 12 Treiberröhrenwec. . . Piepseranlage GUL 5 DS 7 H GUS 4 BB 3, GUH 2 B. . . GUS 4 BB 4 GUH 2 BR 2, GUH 3 B. . . Windows Experime. . . GUL 5 DC 6_P GUA 1 BA 1 GUA 2 B Phasenbeziehunge. . . S 12 QS 2 D GTH 4 VV 1 S FRS-Vakuumfenster GTS 2 QT 11, GTS 2. . . GS 12 QS 1 F Timingsystem GUH 3 BB 1 GUH 2 QQ 11 GUH 2. . . Prallblech Dipol UZ. . . Diverse Geräte an. . . GUXIQD 51 Mikrowelle GS 12 MU 2 A GUS 4 BB 3 GUZEMU 6 / Prallbl. . . GUH 2 QQ 12 GUN 2 IG 220 V und 20 k. V GTK 3 MV 3, GTK 3 M. . . GS 11 MU 2 GTK 4 DG 2_P GTK 4 DG 1, GTK 6 D. . . FRS Targetbereich PHYS RUN 2018: Device Failures Statistic 250000000 20000 150000000 10000 50000000
What is missing § § Automatism Accuracy Completeness Live Data § present machine status § continuous transmission data § trend analysis § Information on Beam/Settings quality (Transmission, Focus, Intensity, Spill-Structure, . . . ) GSI Helmholtzzentrum für Schwerionenforschung Gmb. H Automated Failure Tracking - Stephan Reimann 19. 09. 2021
Ende GSI Helmholtzzentrum für Schwerionenforschung Gmb. H Automated Failure Tracking - Stephan Reimann 19. 09. 2021
