Maintenance and Efficiency Paul W Sampson Maintenance and

Maintenance and Efficiency Paul W. Sampson Maintenance and Accelerator Support Group Leader: CAD/RHIC Brookhaven National Laboratory

Short Background § CAD at BNL in Upton New York, USA § RHIC Large superconducting collider • 500 Ge. V cm for polarized proton running • Species to Au (U this year? ) • Dissimilar species § Many “parasitic” users on injectors: • LINAC • Tandem Van de Graff • EBIS • Booster • AGS April 13 th, 2011 Paul W. Sampson 2

April 13 th, 2011 Paul W. Sampson 3

April 13 th, 2011 Paul W. Sampson 4

Maintenance Support Group Role § Plan and coordinate work throughout the accelerator complex. § Emphasis on operations and uptime. § Integrate Professional Trades, Technical, Engineering and Physics staff activities during shutdown, failure (unscheduled maintenance) and scheduled maintenance periods. April 13 th, 2011 Paul W. Sampson 5

Goals: Maximize uptime and optimize accelerator output § Integrated Luminosity is king? • STAR, PHENIX, Overall integrated luminosity. • What cost for improvement? § Parameters vary for varying species in RHIC • Use and maintenance of different systems. • Differing beam parameters. • Safety/Radiation issues- affects on access and equipment availability. § New systems commissioning. § Internal goals: • Beam parameters. • Available time at store, setup minimization • Calendar time at store. April 13 th, 2011 Paul W. Sampson 6

Reliability formula complicated § End of life. § Installation and commissioning. • Infant mortality § “Roll up period” § Legacy systems. April 13 th, 2011 Paul W. Sampson 7

April 13 th, 2011 Paul W. Sampson 8

Shutdown § Annual occurrence. § Duration factors: • Budget. • Major work schedule. • Experiment upgrades/readiness. • Climate. § Documented utilizing data base tool. § Progress tracked and Project published. § Goals • Complete shutdown work. • Prepare for structured pre-startup checkout. • Minimize start up (time to Physics). April 13 th, 2011 Paul W. Sampson 9

Unscheduled Maintenance § Major failure or other prolonged downtime. • Goal of maximizing use of time and access while minimizing the recovery. • Coordinated by Maintenance Support Group. § Schedule are created as necessary, depending on length of downtime. § Each day run similar to a scheduled maintenance day. April 13 th, 2011 Paul W. Sampson 10

During the run: “Minor” Failure (>4 hours) § Small low impact jobs to be performed. § “Fast” experimenter access § Coordinated by on duty Operations Coordinator, with assistance and advice from Maintenance Support Group as needed. § Documented by standard methods. April 13 th, 2011 Paul W. Sampson 11

During the run: Scheduled maintenance days. § Usually 8 -12 hours, biweekly. § Driven by experimental access. § Accelerator access for: • Preventative Maintenance. • Repair • Modification • Installation • Commissioning. • Shutdown planning • Inspection/work observation • Continuation of shutdown/installation work. April 13 th, 2011 Paul W. Sampson 12

How maintenance days directly impact availability… § Scheduled maintenance days demand that the program is off. § Many required tasks require prolonged access. § Some systems have long lead time before work can be performed. Source cool down. • Cryogenic warm up. • Radiation cool down periods. • § Setup and recovery after maintenance period. § Human error: Systems left in inoperable state following maintenance period. • Failure of improperly maintained equipment or systems. • April 13 th, 2011 Paul W. Sampson 13

Reliability and Maintenance days: § Preventative maintenance periods should be focused on reducing failures. § Repairs to systems and infrastructure. § Accelerator tunnel/Radiation area access: • PM and repair • Installation, testing and commissioning of new and modified systems. § Low intensity test cycles following maintenance period: • Used for machine development. • Tests and implementation modified and modes before they are adapted as operational. • Scheduled as part of the maintenance period. April 13 th, 2011 Paul W. Sampson 14

Reliability and maintenance days § Maintenance day goals: • Complete tasks necessary to maintain current mode of running at the accelerator complex. • Prepare for any new running conditions expected. • Minimize the total duration and the time from end of maintenance to a return to smooth running. § Some tasks performed can cause unstable running and increased failure in the days that follow maintenance periods. • Vacuum bleed-up. • Ion source maintenance. • Cryogenic. • RF systems. • Super Conducting power supply systems. April 13 th, 2011 Paul W. Sampson 15

Reliability after maintenance: § New running modes can cause increased failure for various reasons: More aggressive magnetic cycles. • Increased high voltage. • Additional interlocks. • § Re-integration of repaired systems: Feedback elements. • Interlocking devices- loss monitors, chipmunks/radiation monitors, beam permit or inhibit systems. • § Failure of new or un-commissioned systems: Apertures in the beam pipe. • Improperly grounded devices. • Miss-configured interlocks or inhibits • Miss-configured devices (current loops). • April 13 th, 2011 Paul W. Sampson 16

Maintenance methods for increased reliability § § § § § Scheduled Maintenance Days Job Requests Benefit and impact analysis Prioritization Individual work planning Overall work planning and scheduling System recovery Accelerator recovery Return to physics April 13 th, 2011 Paul W. Sampson 17

Before maintenance days § Determine which tasks are actually necessary. § Avoid low return, high risk jobs. § Carefully document accelerator status before turn off. • Define parameters for recovery. § Schedule accesses for efficiency- 4 hours to secure accelerators. § Communicate with all groups • Scheduling meeting, web, CCTV, face to face … April 13 th, 2011 Paul W. Sampson 18

Accelerator Physics/Administration Integrated Facility Management Maintenance Support Accelerator systems Experimental Support and Facilities

Job Request: Create April 13 th, 2011 Paul W. Sampson 20

Pic of list April 13 th, 2011 Paul W. Sampson 21

Work planning: § Individual jobs are planned by cognizant techs or § § § engineers. Job request is submitted to the MSG for review. Primary approval given and schedule created. Final review with upper management, experimenters and run coordinators. Staff briefing. Schedule posts. April 13 th, 2011 Paul W. Sampson 22

Schedule: Time By 0600 Activity Personnel Turn down OPPIS source for maintenance (After store to last to 0800 hrs has been established) MCR 0600 Apply CA LOTO to AGS Maintenance Begins (items outside the ring) RHIC areas to RA, RHIC Maintenance Begins LINAC Maintenance begins CAS CAD CA Access into the AGS Access complete RHIC sweeps begin in select areas PHENIX tunnel sweep Hysteresis ramp, PHENIX IR sweep Physics setup in RHIC MCR/CAS MCR/CAS MCR/AP… 0800 0900 1130 1400 1430 1500 April 13 th, 2011 Paul W. Sampson 23 MSG LINAC

Submission: Individual Jobs Initial review: Task Analysis, job approval (denial) Initial Schedule Proposal Final Schedule publication, review with staff Execution of scheduled and Emergent tasks System Recovery, Test ramp Paul. Physics W. Sampson Return to April 13 th, 2011 24

Execution § Complex formally handed over to Maintenance Support Group § MCR staffing: • Maintenance Coordinator (MC) to run activities, normally from MCR. • Utilize Operations Coordinator, Operators and support staff. § Leads keep MC informed of progress, emergent issues and job completion. § MSG leader appraised of major change, facilitates review, approval and schedules changes as necessary. § Maintenance coordinator executes changes. April 13 th, 2011 Paul W. Sampson 25

Communication: § During the maintenance, it is communication is essential. • CCTV • Web, elogs, Job request system. § Radio, pagers and telephone… April 13 th, 2011 Paul W. Sampson 26

End of maintenance: § Job closeout statements reviewed by MC before final closure. § Tiered access allows for recovery of systems to begin as soon as they are ready. • Injectors • RF • Power supply • Vacuum • Instrumentation § Specialist confirms with MSG that systems have been returned to operational condition. § Specialist remains on alert until program is recovered April 13 th, 2011 Paul W. Sampson 27

Return to Physics § Accelerators are returned to operating condition as available. • Schedule work accordingly. § After injectors have recovered, RHIC test ramp performed. § Structured hand off to MCR for routine operation. • Maintenance Coordinator’s Journal closeout. • Operations Coordinator’s journal entry. • Maintenance day summary/closeout. April 13 th, 2011 Paul W. Sampson 29

April 13 th, 2011 Paul W. Sampson 30

April 13 th, 2011 Paul W. Sampson 31

April 13 th, 2011 Paul W. Sampson 32

Results § Since structured systems introduction in 2005: • The methods described have produced a marked improvement in performance on and after maintenance periods. • Return to productive running has steadily improved. • In many instances, machine performance is actually better immediately following maintenance periods. • Could maintenance have actually helped? ? !! April 13 th, 2011 Paul W. Sampson 33

April 13 th, 2011 Paul W. Sampson 34

Maintenance Effectiveness Trend 70% 65% 60% 55% 50% Maintenance Weeks 45% non-Maintenance Weeks 40% 35% 30% 25% 20% FY 2005 FY 2006 April 13 th, 2011 FY 2007 FY 2008 FY 2009 Paul W. Sampson FY 2010 35 *FY 2011

Other comments: § Maintenance and repairs done behind physics § § § stores. Software releases. Systems maintenance: BPMs. Systems commissioning, improvement. Studies following Maintenance days. Dual mode days- Maintenance on one machine and science on another. April 13 th, 2011 Paul W. Sampson 36

Acknowledgements: § Availability, “Top Ten” and integrated luminosity plots adapted from presentation “Operating Statistics” by P. F. Ingrassia, Brookhaven National Laboratory. § “RHIC from space” courtesy of BNL Photo archives. April 13 th, 2011 Paul W. Sampson 37

Thank you April 13 th, 2011 Paul W. Sampson 38

April 13 th, 2011 Paul W. Sampson 39

April 13 th, 2011 Paul W. Sampson 40

April 13 th, 2011 Paul W. Sampson 41