News from the LHC Jrg Wenninger Beams Department
News from the LHC Jörg Wenninger Beams Department Operations group CMS Meeting LHC beam commissioning S 34 incident and consequences December 2008 1
Part I Beam ! 2
Beam Commissioning Phases 1 to 3 Pre-10. 09. 2008 beam tests : code name ‘synchronization tests’ q. Commission the injection elements of the LHC. q. Commission controls and instrumentation of the LHC ring. q. Check q. Make magnetic field (predictions), alignment and apertures. sure we do not screw up too much on the 10 th … Ø The tests were interleaved with commissioning of the remaining magnets & circuits. Ø Note the first time we could run all magnets together : The evening of September 9 th !! 3
Synch. test no. 1 8 -11 August 08 Beam 2 stopped on collimator 4
Synch. Test no. 2 Beam 1 stopped on collimator 22 -25 August 08 Beam 2 stopped on collimator 5
5 -8 Sept. 08 9 Sept. 08 Synch. Test no. 3 Beam 1 stopped on collimator Beam 2 to dump line 6
Polarity Errors In single pass (trajectory) mode, focusing errors can be located by: q launching controlled trajectory oscillations q comparing the measurement with the predictions >> Identified a number of sign errors – some rather severe (see below) ! Example of a polarity inversion of a main quadrupole (IR 7). This error would have spoiled the 10 th September show – very difficult to get past such an error. Histogram : measured trajectory change Points : model Polarity error Beam 2 7
Aperture A great relief : the aperture was very good – no buckled bellows & Co. Example for sector 78 : the aperture is probed by increasing trajectory oscillation amplitudes until losses appear. On the figures one sees the excursions of sample trajectories until losses appear. 8
And we quenched with beam… In the very early morning of August 9 th during the first test, we provoked the first beam induced quench: q Bunch intensity ~4 109 p, which is within the expected range. BLM signal (beam 2 side) BLM signal (beam 1 side) M. Sapinski >> reduced the commissioning intensity to ~2 -3 109 p. In preparation of the 10 th, a test revealed that even with ~2 109 p one can quench – but very unlikely in normal operation due to the large impact angle. Preliminary results ! 9
A look at the first quench : magnet perspective 4 1 3 1. Voltage = 0, no resistance, magnet is superconducting. 2. Beam impact, Voltage != 0 : resistive area in the magnet ! 3. Voltage back to 0 – magnet has recovered spontaneously – very little energy deposition ! 4. Voltage != 0 : protection systems enter the game and force-quench the magnet etc… 2 Voltage (V) across magnet 400 ms 10
A crazy, but so successful D-Day !!! 11
September 10 th Despite the presence of an unbelievable crowd of people : q 10: 30 : Beam 1 around the ring (in ~ 1 hour). Beam makes ~ 3 turns. q 15: 00 : Beam 2 around the ring, beam makes 3 -4 turns. q 22: 00 : Beam 2 circulates for hundreds of turns… 12
Beam 2 around the ring 1. Beam to TDI 5. Beam to IR 3 2. Beam to IR 7 3. Beam to IR 6 4. Beam to CMS 13
Beam 2 around the ring 1. Beam to TDI 5. Beam to IR 3 2. Beam to IR 7 6. Beam to ALICE 3. Beam to IR 6 4. Beam to CMS 14
Beam 2 around the ring 1. Beam to TDI 5. Beam to IR 3 2. Beam to IR 7 6. Beam to ALICE 3. Beam to IR 6 7. Beam to ATLAS 4. Beam to CMS 8. Beam to LHCb – First Turn ! 15
First Beam Around Sept 10 th 10: 30 : two beam spots on a screen near ALICE indicate that the beam has made 1 turn. 16
Beam 2 circulating – no RF Evening of September 10 th , after the crowds left : Beam 2 makes hundreds of turns after some empirical correction (no RF) q Turn by turn beam peak intensity signal at the RF position monitor. q The decay is due to the de-bunching of the beam (bunch becomes longer and longer – no RF). 17
Beam 2 captured by RF system Evening/Night of September 11 th RF RFON RF OFF ON & tuned – wrong phase Beam wrong 2 frequency ‘captured’ 18
Beam 2 on stable closed orbit Sept 12 th : Beam 2 captured, lifetime > 1 and closed orbit corrected. Looking very good… RF frequency is not (yet) correct: average hor. orbit is negative… 19
Electrical transformer problem… Late evening of Friday Sept. 12 th an old LEP HV transformer in point 8 failed, leading to a stop of the cryogenics – LHC ‘off’. q TS/EL had no spare (!), but a spare CMS transformer was recuperated and installed during the weekend. q Due to various other problems, the cryogenic system was only completely back for the Friday 19 th. q In the meantime, access and commissioning of remaining circuits wherever cryogenics conditions were OK. q Late in the morning of Sept. 19 th the last dipole circuit of sector 3 -4 is commissioned to 5. 5 Te. V… 20
Status on the morning of Sept. 19 th Beam 2 well advanced: q Beam captured in RF system, good orbit and lifetimes of hours. q Optics in ‘reasonable’ shape, preparing for refinements. Beam 1 in same state as 10 th: q First turn established, beam in for 3 -4 turns. Objectives for the weekend q Bring beam 1 to same level as beam 2. q Improve measurement and correction on beam 2. q Try to circulate both beams together … But…. 21
Part II S 34 Incident or …what you can do with 200 MJ 22
And then came September 19 th 11: 18… During the last commissioning step of main dipole circuit in sector 34, to 9. 3 k. A : q At 8. 7 k. A, development of resistive zone in the dipole bus bar between Q 24. R 3 and the neighboring dipole. Most likely an electrical arc developed which punctured the helium enclosure. q Large amounts of Helium were released into the insulating vacuum. q q Rapid pressure rise inside the LHC magnets – Large pressure wave travelled along the accelerator both ways. – Self actuating relief valves opened but could not handle all. – Large forces exerted on the vacuum barriers located every 2 cells. – These forces displaced several quadrupoles by up to ~50 cm. – Connections to the cryogenic line damaged in some places. – Beam ‘vacuum’ to atmospheric pressure 23
Dipole 7 Te. V • 8. 33 T • 11850 A • 7 M J 24
Inter-connection Vac. chamber Dipole busbar 25
Damage zone Insulating vacuum barrier every 2 cells in the arc n n Some moved Considerable collateral damage over few hundred metres Contamination by soot and debris (magnetic !) of vacuum chambers – extends beyond mechanical damage zone. Damage to super-insulation blankets Large release of helium into the tunnel (6 of 15 tons) 26
Displacements 27
Collateral damage S. Myers AB depart. meeting 28
Intermezzo : Quench Protection q A ‚quench‘ is the phase transition from the super-conducting to a normal conducting state. q Quenches are initiated by an energy in the order of few m. J – – q movement of the superconductor (friction and heat dissipation), beam losses, cooling failures, any other heat sources. . . When part of a magnet quenches, the conductor becomes resistive, which can lead to excessive local energy deposition due to the appearance of Ohmic losses. To protect the magnet: – the quench must be detected. – the energy in the magnet /electrical circuit must be extracted. – the magnet current has to be switched off within << 1 second. 29
Quench Detection and Energy Discharge Power converter Dump resistor 1. 2. 3. 4. 5. Dump resistor The quench is detected based on voltage measurements over the coils (U_mag_A, U_mag_B). The energy is distributed over the entire magnet by force-quenching with quench heaters. The power converter is switched off. The current within the quenched magnet decays in < 200 ms, circuit current now flows through the ‚bypass‘ diode that can stand the current for 100 -200 s. The circuit current/energy is discharged into the dump resistors. 30
Dump Resistors Those large air-cooled resistors can absorb the 1 GJ stored in a dipole magnet circuit (they heat up to few hundred degrees Celsius). 31
Busbar Protection The individual interconnects are not protected. q The entire busbar (i. e. all magnet interconnections of a given electrical circuit) is protected by a global protection system. In case of a busbar quench: q – the power converter is switched off, – the dump resistor switch is opened to discharge the energy. – the busbar is designed to cope with the discharge time of ~ 200 s. q A look at the quench protection system data for the incident revealed that: 1) a busbar quench was developing, 2) suddenly the voltage over the busbar increased dramatically, leading to a fault of the power converter (over-voltage), 3) followed by the dramatic energy release (~ 200 MJ) in the cold mass. >> Most likely cause : an electric arc due to rupture of the interconnection. Unfortunately this is difficult to prove, since the whole dipole interconnect was ‚vaporised‘ during the event ! 32
Busbar interconnection Interconnection resistance ~ 0. 35 n. W 33
Welded interconnection 34
Main Dipole / Quadrupole Interconnection Note that the connection is NOT clamped ! Splice insulation Length He II @ 1. 9 K, 1 Bar current Bus Bar’s Insulation Heat exchange with He II Favored hypothesis for the S 34 incident cause : • Temperature increase due to an excessive resistance (estimate ~ 200 n. W). • Superconductor quenches and becomes resistive at high current (temperature increase due to the resistance). • Up to a certain current, the Copper can take it (cooled by the He II). • Beyond a certain current, ‘run-away’ of the temperature, splice opens, electrical arc … 35
Early Warning Signs q Following the incident, a closer look at the logged cryogenic data (temperatures and valve states) clearly indicated a heat source in the cell that was at the origin of the S 34 incident: Ø The data revealed the presence of a ~ 200 n. W resistance in that cell (before the incident): most likely the interconnect quality. >> Logged data recorded during commissioning of the 7 other sectors was checked to locate other potential problems : a hint was found in a cell of S 12. q Controlled calorimetric measurements (at different magnet currents) were started in the sectors that are still available to: Localize cells with current dependent heat sources. ü Confirm the source and localize precisely with electrical measurements. ü 36
Relative temperature, increase from 1. 9 K Calorimetric analysis of sectors before Sept 19 DT (mk) 7 k. A test on Sector 3 -4, Sept 15 th 7 k. A test on Sector 2 -3, April 14 th +40 m. K Temperatures stabilize Temp of Q 23 keeps rising! 1 h Current in k. A For comparison, another sector, identical scales 9. 3 k. A test on Sector 1 -2, Sept 1 st Suspicious cell in S 12 Adriaan Rijllart
Calorimetry tests : after September 19 th L. Tavian 100 n. W 75 n. W 50 n. W 25 n. W S 1 -2 S 6 -7 S 7 -8 38
Calorimetry tests : candidates Not confirmed by electric measurements, poor(er) data quality 15 R 1 L. Tavian 100 n. W 75 n. W 50 n. W 19 R 1 31 R 1 S 1 -2 31 R 6 S 6 -7 25 n. W S 7 -8 39
Electrical Measurements q S 12 ‘ 100 n. W’ calorimetric anomaly: Ø All interconnects of the suspected cell were checked with high precision (n. W) resistance measurement devices. Result: § Ø q All interconnects OK, all resistances consistent with 0. 35 n. W. A systematic campaign of internal magnet resistance measurements was launched. Result: § A dipole with a 100 n. W resistance was found in the suspected cell. § Test bench data indicates that this resistance was already present when the magnet was tested to 9 T/7. 6 Te. V. S 67 ‘ 50 n. W’ calorimetric anomaly: Ø The same magnet measurements localized a dipole with a 50 n. W internal resistance. » The resistance of those magnets arises from the internal interconnections! » The dipole in S 12 will be exchanged, for the dipole in S 67 the decision is pending… 40
Electrical measurements : the data The Quench Threshold is 100 m. V during 10 ms U_1 U_2 Sampling Rate 5 ms Resolution 0. 125 m. V 13 sec, 2450 points ~4 m. V Average = -0. 88± 0. 02 m. V 41
Magnet resistance measurements Magnets are powered to different current levels. Ø At each step the voltage is measured to determine a possible residual resistance of splices INSIDE each magnet. !! Does not measure the splice resistance of the tunnel interconnects !! Ø 42
Voltage (V) Dipole B 32. R 1 with 47 nΩ splice resistance Current (A) 43
Measurement campaign status S 1 -2 Dipole circuit S 2 -3 S 3 -4 S 4 -5 T R M Quadrupole circuit T T R M IPQ (6 k. A) T T R M T: Transport R: Repair S 5 -6 S 6 -7 S 7 -8 S 8 -1 M: Maintenance Done q A calorimetric and electrical measurement campaign was run into December to cover the largest possible number of sectors a maximum before the start of the shutdown. q A little over ½ the LHC has been checked. 44
Repair – – 55 magnets will be taken out by Xmas (39 dipoles, 14 SSS [quadrupole assemblies]). Magnets will be either replaced by spares or repaired (super-insulation). All magnets will be re-tested in SM 18 before installation in sector 34. The vacuum chambers may have to treated outside the ‘zone’ : buckled bellows and dust in beam vacuum chamber… – Estimate (preliminary) : March 09. 45
Consolidation : limiting pressure with flanges as relief valves Courtesy V. Parma Vacuum instrumentation DN 100 Cryogenic inststrumentation DN 63 BPM DN 100 Each SSS/quadrupole assembly: 4 DN 100 ports (2 for vac. devices, 2 for BPM cable feedthrough) 1 DN 63 port (for cryo instrumentation) Beam position monitors DN 100 46
Limiting Pressure Present: 2 DN 90 Consolidation option B (A), for cold sectors: x 9. 3 SV SV SV Consolidation option C, for warm sectors: x 40 SV SV SV SV SV 47
Consolidation work The following modifications and consolidations will be implemented: A. Upgrade of the quench protection system for precision measurements and protection of all interconnects : early detection of similar problems + protection against symmetry quenches (already in the pipeline before S 34) B. Modifications of commission procedure to include cryogenic/calorimetric information and systematic electrical measurements. C. Improved anchoring of SSS located at vacuum barriers. D. BPM flanges modified as ‘pressure relief valves’. E. Addition of pressure release valves on EVERY dipole cryostat for the warm sector. >> E cannot be implemented for 2009 for the sectors that will be kept cold this shutdown. 48
Summary I Start-up with beam: • Despite totally , crazy’ conditions the beam start-up was excellent. • The speed of progress with beam 2 exceeded even our optimistic hopes. • A lot was learned, but not enough to be sure that the rest of the early commissioning will proceed as well as the first 3 days… • Things were looking were good ! • In the operations group we are analysing what could have been done better and we are working hard to make the next start-up even more efficient ! 49
Summary II Sector 34 incident: • Revealed a weakness in the magnet protection system which did not cover dramatic bus-bar/interconnect incidents. • Repair or replacement of ~ 55 magnets will take most of the shutdown. • Ongoing work to analyse consequences/fix the vacuum chamber ‘pollution’ in the sector. Search for local resistances: • Powerful calorimetric and electric measurements were validated to localize suspicious electrical resistances. • A little over half the ring could be checked. No other bad interconnect was localized, but two dipoles with internal resistances were identified: • Sector 12 / 15 R 1: resistance of ~ 90 n. W >> dipole will be replaced. • Sector 67 / 31 R 6: resistance of ~ 50 n. W. 50
Summary III Repair & consolidation: • Improvements in the quench protection system, ready summer 2009, and a careful monitoring of the temperatures should provide early warning/protection against similar events. • Improved anchoring of SSS at vacuum barriers. • The pressure relief system will be improved : full implementation on warm, partial on cold sectors. ! The pressure relief systems will limit mechanical damage, but cannot help against vacuum chamber pollution (if the beam vacuum is affected). >> also an issue for beam induced damage !! 51
The plan for 2009 Running scenarios will be worked out in February in Chamonix: • Energy(ies) • Intensity LHC cold Last magnet goes into sector 34 2009 Dec Jan Feb Mar Apr May Jun Jul Aug Removal of damaged magnets Cleaning and repair Cold testing Reinstallation Interconnection Pressure testing Cool down 52
53
The CERN Accelerator Complex 5 LHC 4 Beam 1 Beam 2 7 3 TI 8 SPS 2 TI 2 protons LINACS 6 Booster 8 1 CPS Ions LEIR Linac PSB CPS SPS LHC Top energy/Ge. V Circumference/m 0. 12 30 1. 4 157 26 628 = 4 PSB 450 6’ 911 = 11 x PS 7000 26’ 657 = 27/7 x SPS Note the energy gain/machine of 10 to 20 – and not more ! The gain is typical for the useful range of magnets !!! 54
Injection Regions First difficulty : get past the tight and crowed injection regions near ALICE (Beam 1) and LHCb (Beam 2) q Injected beam is brought parallel to LHC beam by magnetic septa (MSI). q Injected beam is kicked vertically down onto the LHC plane by fast kicker magnets (MKI). q Protection block (TDI, 4 m long) to protect downstream elements from mis-injections. Right of IP 8 (H plane) TI 8 D Q 2 4 TDI (MKI +90˚) Q 5 5 x. MKI 5 x. MSI 55
Beam Dump Both dumps were nearly fully commissioned for 450 Ge. V. 56
Another… 57
LHC Arc ‘Cell’ • • • 23 regular cells in each arc 106. 9 m long, made from two 53. 45 m long half-cells Half cell – 3 15 m cryodipole magnets, each with spool-piece correctors – 1 Short Straight Section (~6 m long) • Quadrupole and lattice corrector magnets 58
- Slides: 58