Operational aspects of the VELO cooling system of

Operational aspects of the VELO cooling system of LHCb Eddy Jans (Nikhef) on behalf of the LHCb VELO group • • • Introduction Main components and operation principle of the system Issues: how to prevent and to tackle them Keep the detectors cold 24/7 Summary & outlook 19 September 2013 Vertex 2013 Eddy Jans 0

Introduction • VELO-module is double-sided (300 mm, oxygenated, n+-on-n) and operated in vacuum, • Strip closest to the beams is at 8. 2 mm, • Per double-sided module: 2 x 16 frontend chips that together dissipate ~20 W, • 4 NTCs give temperature readings • Two movable detector halves with 21 VELO + 2 Pile. Up modules each (400 W/side) NTCs 19 September 2013 Vertex 2013 Eddy Jans 1

One detector half CO 2 connections module base is kept at +20 o. C 0. 3 mm thick RF-box Vacuum tank cooling block frontend chips Detectors are operated in a secondary vacuum absolutely no leaks allowed. Orbital welding or vacuum brazing of the pipes. All tested at 170 bars. 19 September 2013 Vertex 2013 Eddy Jans 2

Some cooling considerations • in 1999 it was proposed to use CO 2 as refrigerant for the vertex detector of LHCb [LHCb note 99 -046], • CO 2 is radiation hard, • CO 2 has a high latent heat value can use small diameter capillaries small amount of dead material in the acceptance, • stainless steel capillaries: Finner=1 mm, wall thickness 0. 25 mm • system uses bi-phase CO 2 via the accumulator controlled method. cooling blocks 19 September 2013 Vertex 2013 Eddy Jans 3

x pressure [MPa] critical point liquid phase bi-phase area Enthalpy diagram of CO 2 At -30 o. C: 300 J/g gas phase area -30 o. C pressure [bar] Pressure mass flow: 10 g/s per module: 0. 43 g/s full evaporation 130 W liquid CO 2 -speed: 28 cm/s all-gas speed: 240 cm/s isothermal cooling 0 vapour quality 1 enthalpy [k. J/kg] 19 September 2013 Vertex 2013 Eddy Jans 4

2 insulation 55 m transfer line sub-cooled liquid in bi-phase return 1 2: pump increases the pressure of the sub-cooled liquid 2 3: heat exchange in the transfer line brings evaporator pre-expansion per definition right above saturation point, since (E 2 -E 3) = -(E 5 -E 6) 3 4: pressure drop in restriction and expansion in capillary brings CO 2 in cooling blocks in bi-phase state, 4 5: isothermal cooling via evaporation 5 6: warming up of incoming sub-cooled liquid 6 1: condensation and cooling of the CO 2 19 September 2013 Vertex 2013 P 7 Heat in 6 1 pump f=80 mm only passive components 4 in the 3 evaporator radiation zone restriction 5 Pressure Condenser R 507 a chillers Heat in P 7 Heat out accumulator Heat out The accumulator controlled cooling cycle vapor liquid 2 1 3 bi-phase 4 6 5 Enthalpy Eddy Jans 5

Main components of the system • evaporative CO 2 cooling system • • • • “independent” system for either side PLC-controlled 2. 5 k. W water-cooled chiller at – 40 o. C 1 k. W air-cooled backup chiller at -25 o. C 55 m CO 2 transfer lines 10 heat exchangers 8 actuators 9 heaters 31 pressure sensors 192 temperature sensors 350 parameters monitored in PVSS only passive components at VELO 2*400 W heat load of detectors 2*12 kg CO 2 Performance @ detector • main chiller: -28 o. C operational CO 2 temp. LV on: sensors @ -7 o. C • backup chiller: LV off: @ -8 o. C • stability < 0. 1 o. C 19 September 2013 Vertex 2013 Eddy Jans 6

Design considerations and operational experience • • redundancy of crucial components insulation clogging filters superheated CO 2 dependence on electrical power dependence on chilled water safety measures to prevent overheated detectors keeping the system 24/7 cold 19 September 2013 Vertex 2013 Eddy Jans 7

Redundancy in the design To minimise down time the system has a few redundant crucial components: • 3 CO 2 pumps, where 2 are needed, • 2 chillers, water-cooled an air-cooled as backup, • for controls crucial temperature and pressure sensors are two-fold implemented, • possibility to interconnect the two sides by hand, • PLC is on a 1500 VA/1000 W UPS, • PLC, backup chiller and CO 2 pumps are connected to a diesel generator. 19 September 2013 Vertex 2013 Eddy Jans 8

Insulation Liquid pumped system cold transfer lines good thermal insulation required. This seems trivial, but turned out not to be so in practice. Originally CERN safety regulations forced us to use Armaflex NH. Glued surfaces started to delaminate after 2 years. Renewed insulation of the transfer lines and most of the cooling plant during Winter shutdown ‘ 10 -’ 11. Now foamglass covered by an Aluminium protection shield and Armaflex AF, respectively. 19 September 2013 Vertex 2013 Eddy Jans 9

Filters Throughout the system eleven 15 mm filters are installed. (5 (CO 2 -plant), 2(@VELO), 2(manifold), 1(main chiller), 1(backup chiller)). In one detector half we have experienced a few times clogging filters. pressure [bar] 17 16 15 2 months Replacement procedure is tricky and risk of additional dirt in the system due to difficult accessible filter houses. 19 September 2013 Vertex 2013 piece of Armaflex thermal insulation once completely blocked a restriction valve Eddy Jans 10

Post mortem analysis of the filters 75 mm Scanning Electron Microscope image Energy Dispersive Spectrometer analysis of an orange particle found inside the filter Many >15 µm orange objects have been observed inside the filter. They mainly contain Fe and O. Before, particles containing Cl had been observed. Possibly due to connections soldered with flux for a testbeam experiment risk in terms of corrosion. Work extremely clean from construction to installation. 19 September 2013 Vertex 2013 Eddy Jans 11

Superheated CO 2 after startup After startup we occasionally observe in a varying number (a few all) of cooling blocks the phenomenon of superheated CO 2. Issue: cooling performance is very bad because liquid cooling has much less cooling power than evaporative cooling. 19 September 2013 Vertex 2013 Eddy Jans 12

Superheated CO 2 after warmup and cooldown Tsilicon [o. C] LV off -14 -22 cooldown warmup of the cooling plant DT=3 o. C -30 time 30 minutes 19 September 2013 Vertex 2013 Eddy Jans 13

Superheated CO 2 after warmup and cooldown Pressure Tsilicon [o. C] LV off -14 2 1 start adding heat -22 vapor liquid 3 bi-phase 4 6 5 Enthalpy DT=3 o. C -30 time 30 minutes Remedy: add heat by means of a dedicated heater to bring the incoming CO 2 in the liquid+gas state. 19 September 2013 Vertex 2013 Eddy Jans 14

Some more superheated CO 2 Not all cooling blocks behave the same way: - not all show superheating - when adding heat they don’t start boiling at the same moment silicon temperatures of 4 modules DT 4 o. C start adding heat 19 September 2013 Vertex 2013 Eddy Jans 15

Power cuts PLC and backup chiller are connected to the power of a diesel generator of LHCb and the PLC also to its own UPS (1500 VA/1000 W) sensor temperature When the power gets cut the -10 switch-over from main to backup chiller is handled automatically by the PLC. After switching back to the main chiller the system is stable after ~20 minutes. After switching on the LV the sensors are at their operational temperatures after 10 minutes. half an hour recovery time. -20 -30 10 minutes LV on 19 September 2013 Vertex 2013 Eddy Jans 16

Failure of chilled water supply Chilled water supply, that cools the main chiller, sometimes gets interrupted. If so, the PLC switches on the air-cooled backup chiller. However, this causes the LV to be switched off also. 19 September 2013 Vertex 2013 Eddy Jans 17

Safety Operation in vacuum requires immediate reacting safety systems. Three levels. Conditions 1. HW-based: interlock system 2. SW-based: warning Actions Vacuum-PLC LV Off Cooling-PLC Module temperatures Temp-boards HV Off Cooling Off Retract VELO Beam Conditions Monitor Emergency button Combined information of 4 NTCs per module are input to the FPGA, which can interlock the LV. 19 September 2013 and interlock system Vertex 2013 132 cooling parameters monitored 3 levels each: warning, error and fatal Eddy Jans 18

3. Human-based Emergency button in the LHCb control room to power off the VELO. 19 September 2013 Vertex 2013 Eddy Jans 19

Keep the detectors cold 24/7 At the tip the received fluence is 2 x 1014 neq/cm 2 and type inversion has taken place, so the sensors should always be kept cold, (below -8 o. C), in order to prevent the Vdepletion to increase due to reverse annealing. Vdepl NC + DNeff (=effective space charge density) • Short term: “Beneficial annealing (NA)” • Long term: “Reverse annealing (NY)” - time constant depends on temperature: ~ 500 years (-10°C) ~ 500 days ( 20°C) ~ 21 hours ( 60°C) The beneficial annealing budget amounts to a handful of weeks at room temperature. We try to save it till we really need it. But the conditions of the Long. Shutdown 1 period at LHCb make it hard to do so. 19 September 2013 Vertex 2013 Eddy Jans 20

The challenge is to keep the system operational 24/7. Under normal conditions the PLC deals with common problems. Goal: minimize the warm time due to • scheduled maintenance of crucial components, • repair of malfunctioning components, • unexpected problems during LS 1 and shutdown periods. 19 September 2013 Vertex 2013 Eddy Jans 21

Regular maintenance Yearly maintenance of the R 507 a chillers is performed by a specialized external company. Downtime ~0. 5 day / chiller. Yearly maintenance of the 3 CO 2 pumps is done by Nikhef-technicians. Pump is unavailable for >24 hours. Effective downtime of the system: 2 hours / pump. Repair of failing components So far no component had to be replaced, although a (redundant) pressure gauge stopped working in 2012, but miraculously reincarnated after 6 months. 19 September 2013 Vertex 2013 Eddy Jans 22

Unexpected problems How do you know a serious problem occured, causing the cooling to go off and the detector to warm up ? Especially during LS 1. Can’t rely on a PVSS-script sending a mail or sms. A modem and sms-routine have been installed in the PLC. antenna When a problem occurs every half hour a text message is sent to a list of phones numbers, until the cooling system is again in a proper state. Acted 6 times since Feb. ‘ 13 due to failing services. modem with Sunrise sim card, so works underground. 19 September 2013 Vertex 2013 Eddy Jans 23

Integrated warm time in 2012: ~1 day 18 = 8 hours 0 -10 0 year 2012 -30 19 September 2013 Vertex 2013 Eddy Jans 24

Summary • • cooling system is continuously operational since >4 years, performance is stable and according to specs, redundancy of crucial components has shown to pay off, clogging filters are annoying, good thermal insulation is less simple than it seems, superheated CO 2 can be dealt with, the warning system that sends sms-es is a great tool, thus far the integrated warm time has been ~1 day/year, so ………. 19 September 2013 Vertex 2013 Eddy Jans 25

Outlook lets keep it cool till LS 2, when the new VELO pixel detector goes in. 19 September 2013 Vertex 2013 Eddy Jans 26