OCP Cold Plate Work Session Jessica Gullbrand Cold

OCP Cold Plate Work Session Jessica Gullbrand Cold Plate Work Stream Lead Advanced Cooling Solutions

ACS Cold Plate Focus group working on ACS Cold Plate Requirements • Technology Cooling System (TCS) fluid loop - Cooling loop from CDU through rack manifold and IT equipment and back • Provide information/education, classification, and minimal requirement for OCP • Authors: - Jessica Gullbrand, Intel - Nigel Gore, Iceotope - Elizabeth Langer, CPC https: //docs. google. com/document/d/1 f 7 sup. R 2 NAMi. CMZVe. T 9 mm. S 4 -Dog. Fta-c. G_He 8 HRBWs. M 8/edit

ACS Cold Plate Requirements Focus Technology Cooling System (TCS) Examples In-Rack CDU Row Level CDU Virtual CDU Use the Facility Cooling Liquid in TCS loop Figure by Intel, C. Winkel

ACS Cold Plate Definition Hybrid Cooling – both liquid and air cooling • IT Equipment/Rack (with or without rear-door heat exchanger, RDHX) Full Liquid Cooling – all heat is rejected to liquid • Rack/IT Equipment (with or without RDHX)

ACS Cold Plate Definition Cold Plates – Heat exchangers or heat sinks with internal tubing or channels to allow cooling liquid to flow through Coolant Distribution Unit (CDU) – Circulates the cooling liquid to the IT equipment and transfer the heat to the facility cooling liquid Rack Manifold – Distributes cooling liquid within the rack Quick Disconnect Couplings – Disconnects the IT equipment or its components from the TCS loop for serviceability

Cooling Liquid Considerations Cooling Liquids Included (Single- and Two-Phase Liquids): • Water with additives • Glycol based liquids • Dielectric liquids • Refrigerants Material compatibility required between wetted materials and cooling liquid What cooling liquids are covered/considered in DCF?

Liquid Cooling Requirements Pressure Safety Requirements • The liquid cooling loop, its ingredients, and connections must be tested to the highest pressure of the safety standards: � The IEC 60950 -1 For EU and US, this standard is outgoing and is expected to be withdrawn on December 20, 2020 in favor of the IEC 62368 -1 standard. � The IEC 62368 -1 is the new safety standard requiring leak tests at 3 x under normal operating pressure and 2 x under abnormal and single fault conditions � ASME B 31. 3 “Process Piping” (2018) requires leak test at the 1. 5 x design pressure for interconnecting piping • What pressure requirements are being considered by DCF? • Standards valid on facility level? On TCS level?

Liquid Cooling Classifications IT Equipment Classifications • Hybrid Basic: CPU/GPU cold plates • Hybrid Intermediate: CPU/GPU & DIMMs cold plates • Hybrid Advanced: CPU/GPU & DIMMs & Additional (specify) cold plates • Full Liquid: cold plates on nearly all components

Liquid Cooling Classifications Liquid Rack Classifications • Hybrid Liquid Cooled Ready Rack without RDHX* • Hybrid Liquid Cooled Ready Rack with RDHX* • Full Liquid Rack Cooling without RDHX* • Full Liquid Rack Cooling with RDHX* *RDHX = Read-Door Heat Exchanger

Performance Metric

Performance Metric PUE versus TUE for liquid cooling PUE = Power usage effectiveness TUE = Total PUE, “PUE” type metric ITUE = PUE for the IT equipment

Performance Metric https: //eehpcwg. llnl. gov/assets/as_tue_a_new_look_at_pue. pdf

Performance Metric

Performance Metric

Performance Metric

Performance Metric g i

Performance Metric

Performance Metric

Future Topics of Importance • Manifold Design • Connector Specification • Leakage Detection & Mitigation • Performance Metrics • Testing/Metrology/Requirements

Backup

ACS Cold Plate Purpose: Generate an open specification and supporting documents focusing on standardization and definitions of liquid cooled solutions without preventing innovation Note: Cold Plate work stream is focusing on liquid cooled solutions using cold plates

Liquid Cooling Requirements Certification Markings • Each liquid cooled technology needs to comply with all certification regulations, which are valid for the geographic location where the liquid cooled system is used. Different regions have different requirements. • Examples: � UL and FCC markings in the US � CE certification in Europe

Cooling Liquid Considerations Water with Additives Pros Cons Good heat transfer properties (high Additives needed for reducing corrosion conductivity and specific heat) risk and bacterial growth Ongoing regular maintenance

Cooling Liquid Considerations Glycol Based Liquids Pros Cons >25% glycol no bacterial growth Changes to viscosity => changes pump power conditions including power needed Improved maintenance schedule Lower conductivity and specific heat with increasing glycol level

Cooling Liquid Considerations Dielectric Liquids Pros Cons No short circuit of electronics Adds liquid weight during potential leaks Higher Global Warming Potential (GWP) Higher cost Limited supplier availability Potential for flow instabilities and maldistribution in micro-channel cold plate using 2 -phase

Cooling Liquid Considerations Refrigerants Pros Cons Increase in thermal performance compared Environmental - Ozone Depletion Potential (ODP) and Global Warming Potential (GWP) to single-phase Potential for no and lower pumping power Potential loss of liquid under maintenance compared to single-phase conditions Specifically newer refrigerants can have Higher operating pressure than single-phase lower GWP than dielectric 2 -phase liquid Can be inert: nontoxic, nonflammable, Potential for flow instabilities and nonelectrically conductive maldistribution in micro-channel cold plate using 2 -phase Specialist knowledge required for commissioning, handling, and maintenance