Operational Experience SNS Karen S White Controls Group

Operational Experience @ SNS Karen S. White Controls Group Leader 10/11/10

The Spallation Neutron Source The world’s most powerful pulsed neutron source 2 Managed by UT-Battelle for the U. S. Department of Energy EPICS Collaboration Meeting - October 2010

Outline • Machine Performance • Controls Performance • Issues • New Stuff 3 Managed by UT-Battelle for the U. S. Department of Energy EPICS Collaboration Meeting - October 2010

Accelerator Daily Performance 4 Managed by UT-Battelle for the U. S. Department of Energy EPICS Collaboration Meeting - October 2010

Accelerator Lifetime Performance 5 Managed by UT-Battelle for the U. S. Department of Energy EPICS Collaboration Meeting - October 2010

Focus on Availability • Availability considered more important than increasing the beam power • Machine designed for 1. 4 MW, but routinely operates at ~1 MW to maximize availability and production hours 6 Managed by UT-Battelle for the U. S. Department of Energy EPICS Collaboration Meeting - October 2010

Accelerator Availability 7 Managed by UT-Battelle for the U. S. Department of Energy EPICS Collaboration Meeting - October 2010

Controls Availability FY 07 – FY 10 8 Managed by UT-Battelle for the U. S. Department of Energy EPICS Collaboration Meeting - October 2010

Controls Downtime • Includes MPS and PPS • FY 09 = 116. 8 hours – Dominated by PLC and PLC/IOC communications problems ~60 hr – PPS ~26 hr – MPS ~17 hr – Radiation Monitors ~14 hr • FY 10 = 40. 8 hours – – 9 Device. Net ~13 hr PPS ~12 hr MPS ~6 Radiation Monitors ~3 hr Managed by UT-Battelle for the U. S. Department of Energy EPICS Collaboration Meeting - October 2010

PLC and PLC/IOC Communication • Late 2008, increase in PLC module failures and communication problems • Two failures impacted CHL resulting in trips and the downtime associated with 4 K to 2 K transition (16. 3 and 19. 4 hours) • Failure analysis revealed eight different contributing problems • Worked with Allen-Bradley engineers to understand failures and plan improvements • Analyzed control system response to failures in CHL and implemented improvements 10 Managed by UT-Battelle for the U. S. Department of Energy EPICS Collaboration Meeting - October 2010

PLC/IOC Failure Causes and Mitigation Cause 11 Mitigation HW failures in PLC processors HW failure in PLC Ethernet modules HW failure in PLC I/O modules Upgraded processors (99) Damaged fiber optic cable Replaced cable Memory corruption bug in IOC/PLC communication driver Failing Automatic Transfer Switch PLC Firmware bugs Bug fix installed on all IOCs Upgraded Ethernet modules (50) Replaced failed modules Replaced ATS Firmware upgraded; monitoring AB bulletins Single memory bit error in Upgraded processors EPICS Collaboration Meeting - October 2010 Managed by UT-Battelle for the U. S. Department of Energy

Machine Protection System • Easy to “shoot the messenger” • Most downtime due to noisy input signals • Testing verified beam coupling related noise in the Ring Service Building • Several improvements were made to reduce the noise and help the MPS better tolerate the noise • Timing measurements showed many MPS nodes did not trip the beam within the required 20 ms • Primary causes for MPS delay 12 – Large internal capacitors installed for noise filtering – Open collector sublink drive circuit slow response Managed by UT-Battelle for the U. S. Department of Energy EPICS Collaboration Meeting - October 2010

Performance Issues – LLRF Controls • LLRF IOCs run automatic feed forward algorithm • Increasing beam power -> longer beam pulses -> more data • MVME 2100 CPU could not processing all data for each 60 Hz pulse • Upgrading CPUs to MVME 5500 13 Managed by UT-Battelle for the U. S. Department of Energy EPICS Collaboration Meeting - October 2010

Increasing Controls Scope • Increasing support for Instruments – Applying accelerator controls technology where it makes sense for instrument construction and support • Instrument Personnel Protection Systems • Instrument CF*, Vacuum, selected device controls • Sharing engineering expertise for problem areas – Instrument work not in original Controls scope • Infrastructure not in place • Standards not in place – propagating accelerator controls standards 14 *Lon. Works talk Wednesday 9: 30 am Managed by UT-Battelle for the U. S. Department of Energy EPICS Collaboration Meeting - October 2010

Lessons • Take data – listen to what it tells you • Sometimes what appears to be “a problem” has more than one cause • Send clear messages for faults or errors • Monitor and evaluate bug reports; apply patches as needed • Plan for how the control system should respond to various types of failures • Important to strike a balance between new technology and need for high availability 15 Managed by UT-Battelle for the U. S. Department of Energy EPICS Collaboration Meeting - October 2010

CSS Tools • CSS applications for archiving (BEAUTY) and alarm handling (the BEAST) deployed in ops – Configured using RDB – Data and logging also stored in RDB 16 • New DM, BOY under development to interoperate with B&B Managed by UT-Battelle for the U. S. Department of Energy BOY – Thursday 11 am and 3 pm The BEAST Thursday 5 pm EPICS Collaboration Meeting - October 2010

New Stuff • We now have the SNS Café in our building • Spring 2011, a nearby Guest House will open for visitors • Working on CD-2 for Power Upgrade Project • Working on CD-1 for Second Target Station 17 Managed by UT-Battelle for the U. S. Department of Energy EPICS Collaboration Meeting - October 2010