LHC Maintenance Management Policy LHC MAINTENANCE POLICY AND

LHC Maintenance Management Policy LHC MAINTENANCE POLICY AND REQUIREMENTS Chamonix XIV January 2005 T. Pettersson - EDMS: 536848

LHC Maintenance Management Policy Contents • • • Introduction – LHC & TQM Cost of down-time Availability indicators Designed for availability – the Power Converter approach Cryogenics – maintenance management now! Quench protection – Energy Extraction RF Technical infrastructure maintenance Common LHC Maintenance policy? Tools available to help you… Conclusion Chamonix XIV January 2005 T. Pettersson - EDMS: 536848

LHC Maintenance Management Policy LHC and TQM • The reliability and availability of the LHC will depend not only on the quality of its components but also on the quality of the maintenance of these components… • This maintenance quality involves everyone and all the activities related to LHC. • Quality also means conformance to requirements (you should get what you ask for…). • Quality can and must be managed. • This is known in industry as Total Quality Management (TQM) and it implies: – A process for managing quality is defined; a philosophy of perpetual improvement of how we execute our activities. Chamonix XIV January 2005 T. Pettersson - EDMS: 536848

LHC Maintenance Management Policy The Principles of TQM • Quality can and must be managed. • Everyone in the LHC has a customer and is a supplier of something. • Processes, not people constitute the problems to solve. • Everybody is responsible for the quality achieved – not only the QAWG! • Problems must ideally be prevented, not just fixed after they have happened. • Quality can and must be measured. • Quality improvements may be small but must be continuous. • The quality standard is defect free. • Goals are based on requirements, not negotiated. • Life cycle costs should be taken into account, not only the initial costs… • Management must be involved and lead the process. • Plan and organize for a permanent quality improvement. Chamonix XIV January 2005 T. Pettersson - EDMS: 536848

LHC Maintenance Management Policy Cost of down-time • LHC Ct. C is ~ 3. 5 BCHF • LHC experiments ~ 1. 5 BCHF • Expected yearly operation duration ~ 160 -180 days • Expected life-time ~ 20 years • Average yearly maintenance cost of a technical installation is ~ 2% of capital cost (industry). • Physicists of course, worry much more about integrated luminosity… Chamonix XIV January 2005 T. Pettersson - EDMS: 536848

LHC Maintenance Management Policy Global availability indicator - MTTR The MTTR indicator can be split into nine parts: 1 – Production reactivity 2 – Maintenance reactivity 3 – Availability of man power 4 – Diagnostic Delay 5 – Preparation Delay 6 – Stock Logistics Delay 7 – Purchase Logistics Delay 8 – Work duration 9 – Administrative Delay Chamonix XIV January 2005 T. Pettersson - EDMS: 536848

LHC Maintenance Management Policy Some examples of approaches taken • Orbit corrector power converters (AB/PO) • Cryogenics (AT/ACR) • Quench protection and Energy extraction system (AT/MEL) • Radiofrequency (AB/RF) • Technical infrastructure (TS dept). Chamonix XIV January 2005 T. Pettersson - EDMS: 536848

LHC Maintenance Management Policy Designed for availability • 752 Power Converter units for orbit correction. Expected MTBF ~ 80. 000 hours • AB /PO expects to lose on average 1 orbit corrector power converter unit every 4: th day (100+ hours)! • Sufficient power in the remaining corrector power supplies to permit operation for 60 -90 days without stoppage. • Implies that when the machine stops – everything has to be ready for rapid intervention including logistics in the tunnel to permit rapid exchange of any failing supply (25 -65 kg). (MTTR factors) • Faulty equipment is repaired on the surface (“off-line”). Chamonix XIV January 2005 T. Pettersson - EDMS: 536848

LHC Maintenance Management Policy Cryogenics – Maintenance Management Now! • Maintenance plans, tasks and spare parts policy are being defined taking into account manufacturers information, operational experience and detailed process knowledge. • Plans and spares lists are based on industrial state-of-theart approach and methodology are prepared using the CERN asset tracking and maintenance management system (CAMMS) • Maintenance plan will be validated on the initial cryogenic installations going into operation for subsystem commissioning. • Experience will be gathered, stock levels of spares adjusted, maintenance routines (preventive and corrective) optimised. • Outsourcing of maintenance requires extensive preparations of contracts to provide clear objectives and to get reasonable performance from the contractor. Chamonix XIV January 2005 T. Pettersson - EDMS: 536848

LHC Maintenance Management Policy Cryogenics – general approach 1. Corrective maintenance as needed during the operational periods 2. Preventive maintenance: - every winter shutdown on rotating machines, oil levels, filters, inspections, etc. - instrumentation and actuators calibration every two years - safety devices verification and validation every two years – tbc 3. Major overhaul of pumps (every 20'000 hours) and compressors (every 40'000 hours). Issues: 1. How to deal with maintenance of equipments still under warranty and/or in commissioning but already critical for subsystem commissioning? 2. Definition of component criticality with no operational experience and lack of support from manufacturers? 3. Missing resources to establish a complete maintenance plan, spare parts lists, component criticality, maintenance indicator to improve and optimize spare, preventive and corrective maintenance… MTTR 3! Chamonix XIV January 2005 T. Pettersson - EDMS: 536848

LHC Maintenance Management Policy Maintenance of the Quench Protection System • The Quench Protection and Energy Extraction systems must be both reliable and available! This is reached by: – Systematic repair of faults – Analysis of data, acquired during quenches, and action – Diode voltages – Heater voltage recharging etc. – Beam Loss Pattern – Other information in Post Mortem – Analysis of data, taken during test periods, and action – Monthly ≥ 1 day maintenance with access • No LHC run may start without a check of the QP and EE Systems Chamonix XIV January 2005 T. Pettersson - EDMS: 536848

LHC Maintenance Management Policy Requirements for the repair/maintenance • Material – Spares are available and can be repaired/ replaced • Manpower issue (MTTR 3) – At HERA there are 12 persons with experience available when the need arises (24 h/24 h, 7 d/7 d) – In AT/ MEL 4 persons are available today (expecting 2 more later…) – No detailed plans have been made today, MEL is waiting for experience from the HW commissioning before commitments are made. Chamonix XIV January 2005 T. Pettersson - EDMS: 536848

LHC Maintenance Management Policy Radio Frequency System • Detailed work on maintenance management issues by the time the RF hardware is being commissioned, i. e. towards end 2006. • The RF group has a vast experience from running RF systems in other accelerators for guidance. • Spares for critical equipment are acquired as required already now. Chamonix XIV January 2005 T. Pettersson - EDMS: 536848

LHC Maintenance Management Policy Technical Infrastructure Maintenance • The old ST division was an early adapter of CAMMS and much work has been done on: – Standardised terminology! – Naming conventions…. – The Top Ten of non-disposable equipment – Maintenance cost par equipment or family – Number of on-failure (corrective) maintenance per month – Costs of Spare Parts per month – Mean Time Between Failure (MTBF) – Mean Time To Repair per equipment (MTTR) • Yearly maintenance cost ~ 1. 9% of installed value for technical equipment. Chamonix XIV January 2005 T. Pettersson - EDMS: 536848

LHC Maintenance Management Policy Outsourced maintenance • Major issues seen: – Contract specifications – needs have to be defined very carefully. – Performance – you get (sometimes) what you pay for… – Planning of activities – has to concur with the accelerator schedule. – Tracking of interventions – work orders traced in CAMMS – Inventory management. Chamonix XIV January 2005 T. Pettersson - EDMS: 536848

LHC Maintenance Management Policy Maintenance policy? • Objective is clear – no LHC downtime except scheduled! • Present status, location, stocks, maintenance procedures and interventions on all equipment installed must be managed Create dashboards for all equipment families that clearly show both costs incurred and services effectively supplied. • Identify the real processes along with potential bottlenecks, split them into sub-processes. Document the procedures and alleviate the bottlenecks where possible. • Time measurement of the sub-processes must be done automatically using a computer application – an asset tracking and maintenance management system. • Each intervention must originate an Work Order – no action without a trace! • Analyse, evaluate and compare the performance indicators (MTTRj) – otherwise they are of no use. Chamonix XIV January 2005 T. Pettersson - EDMS: 536848

LHC Maintenance Management Policy Tools available to help you… • A computerized asset tracking and maintenance management system is used to do this – we have MP 5 ( new release is called D 7 i). • LHC equipment tracked by the MTF application will automatically be accessible from MP 5/D 7 i. But maintenance procedures must still be created for each family of equipment. • The technical infrastructure is already mostly managed with MP 5/D 7 i as is parts of the cryogenics installations. Much experience exists and is available for you to profit from! Chamonix XIV January 2005 T. Pettersson - EDMS: 536848

LHC Maintenance Management Policy Tools available to help you…dashboard with maintenance performance indicators…. Chamonix XIV January 2005 T. Pettersson - EDMS: 536848

LHC Maintenance Management Policy Conclusion • Maintenance management of LHC equipment has already started! • Keeping LHC available will be taxing resources in the various accelerator and technical departments. This has been expressed by ALL groups that have contributed! • A strict policy of keeping track of all equipment in the complex, their maintenance procedures and the interventions on any equipment related to the availability of LHC is required. • Use of a tool is mandatory but manpower resources must also be made available. • Key indicators can be created to measure how well the quality aspects of the maintenance activities are managed, i. e. they will indicated how the LHC availability is being managed. • LHC QAWG will be asked to prepare general rules and advise on implementation issues for each equipment group. Chamonix XIV January 2005 T. Pettersson - EDMS: 536848

LHC Maintenance Management Policy Acknowledgements Thanks to F. Bordry, L. Serio, K-H Mess, T. Linnecar and I. Ruehl who shared their information about their aproaches to this subject. Chamonix XIV January 2005 T. Pettersson - EDMS: 536848