Control System Overview Bob Dalesio Control Group Leader

Control System Overview Bob Dalesio, Control Group Leader NSLS-II CD-3 DOE Review September 30 – October 2, 2008 1 BROOKHAVEN SCIENCE

Outline • Design Requirements • Progress since CD-2 DOE Review • Construction Readiness • Conclusions • Technical Requirements & Specifications • Cost & Schedule Baseline • Design Maturity Requirements for CD-3 Approval • • Staffing Standards Technical Development Cost & Schedule Performance • Design Maturity Assessment • Remaining Design Issues and Risks 2 BROOKHAVEN SCIENCE

Control System Requirements – 1 of 2 Preliminary requirements complete – detailed numbers in 2009 • • • Bunch Length 1 -40 psecs 2. 6 usec ring revolution Top off every 1 minute Top off bunch train 140 -300 nsec Top off damping time 10 -50 msecs (no extraction) • • Manual control of orbit trims, quadrupoles, sextupoles, and insertion devices are asynchronous ~10 Hz write/read is suitable for “turning knobs” for a power supply 5 Hz updates to operators of up to 1000 chosen parameters Archive up to 6000 parameters at a rate of 2 Hz continually • • Must scale to support 150, 000 physical I/O connections and 400, 000 computed variables 99. 7% availability during operation 3 BROOKHAVEN SCIENCE

Control System Requirements – 2 of 2 • Transient Recording • • • Provide data for all control aspects 5 KHz RF Feedback on beam phase 10 k. Hz orbit feedback, (100 usec loop time) • • • 20 msec equipment protection mitigation 1 Hz model based control 10 k. Hz power supply read backs triggered from timing sys 10’s of Hz Data Collection for RF loop correction. 80 psecs pulse to pulse timing jitter. • Provide electron detector as event for beam line • Take coherent turn by turn orbit data for up to 800 channels 1024 turns • Latch the last 10 seconds of data from all parameters in the storage ring • Beam line needs 1 msec archiving over 1 minute for temperatures and positions • 300 BPMs (10 per cell) • 2 * 120 Corrector PS in 30 I/O Controllers (IOC) • During top off, some beam lines will need 1. 1 - 1. 8 psecs of timing jitter 4 BROOKHAVEN SCIENCE

Cost & Schedule Baseline • Baseline • • Manpower • 4 High Level Application Engineers • 2 RDB Architects • 8 Project Engineers • 1 EPICS Expert • 1 System Administrator Hardware • 156 IOCs w/ timing hardware • $1 M network hardware & 360 K timing distribution • $400 K Servers and Control Room Consoles Software Licenses $300 K Schedule • • Build synchronous distributed device controllers by end 2009 Install Physics Environment for Application development 2009 Install First release of component. lattice, and wiring RDB tools 2009 Prototype subsystems and detailed design by end 2009 Procure and implement subsystems in 2010 Install and test Subsystems in 2011/2012 Improve and manage RDB tools over the project to support design, installation, test, and maintenance. Extend Physics Environment to support installation, test, commissioning and operation. 5 BROOKHAVEN SCIENCE

Design Maturity Requirements for CD-3 Approval Control System Design is 50% Complete 6 BROOKHAVEN SCIENCE

Progress • • • Staffing Standards Developments 7 BROOKHAVEN SCIENCE

Staffing • High Level Applications • Relational Database (IRMIS) • Project Engineers • EPICS • Nikolay Malitsky and Guobao Shen • Two positions open to fill in early FY 09: several likely candidates identified. • Don Dohan and Gabriele Carcassi • Contracts are being used to enhance our capabilities early in the project • • Yuke Tian 2 post docs starting early in FY 09 on 2 year contracts 4 positions to fill in late FY 09 and early FY 10 Contract employees, sabbaticals, and purchase orders are being used to augment efforts. • Tools and System Administration Position to be filled in the next 2 years • Core software developer to be filled in the next 2 years 8 BROOKHAVEN SCIENCE

Control System Standards - 1 of 4 Nomenclature Standard is in place Psy: PI-Ssy: SITsy: TI<Dev: DI>Sg: Sg. I-SD System Device Signal Control System Naming Examples Comment S: C 30 -VA: G 1<SGV: A>Pos: 1 -Opn cell 30, vacuum, girder 1, sector gate valve A (upstream), open S: C 20 -FE: B<FV>Pos: 1 -Cls cell 20 BM front end fast valve S: C 20 -VA: G 1<IP: A>E: 1 -RB 1 st sputter ion pump at C 20 girder #1 S: C 20 -VA: G 4<CHMA: B> 2 nd aluminum chamber at C 20 girder #4 S: C 20 -VA: G 4<BLWR> S: C 24 -RF: G 1<TMP: B> S: C 24 -RF: G 1<CCG: B>P S: C 24 -VA: G 1<CCG: D>P rf shielded bellows at C 20 girder #4 TMP station at 2 nd SC cavity insulating vacuum cold cathode gauge at 2 nd SC cavity beam vacuum 9 BROOKHAVEN SCIENCE

Control System Standards - 2 of 4 Software standards are being evaluated • • • EPICS based IRMIS tools used for all project configuration data: lattice, components, wiring etc. Embedded Real-Time Operating System choices: RTEMS (LCLS, Spear, CLS) or Vx. Works (APS, SNS, Diamond, SLS) Linux Workstations running Debian Use of standard EPICS engineering tools: Extensible Display Manager, Channel Archiver, Striptool, Alarm Handler Evaluate the possibility of participating in the CSS development Visual Database Configuration Tool w/ Modifications for Table Entry Use of physics applications: Matlab Middle Layer Toolkit – MMLT (many light sources), e. Xtensible Accelerator Language - XAL (SNS/SLAC), Elegant (APS), Tracy online simulation (Diamond) Evaluating Use of Beam Line Applications: SPEC, Blu. Ice, IDL, Syn. Apps (APS), GDA (Diamond) BROOKHAVEN SCIENCE 10

Control System Standards - 3 of 4 Architecture for instrumentation is being prototyped Operator stations: Displays, Archiving, Alarm Management, Strip charts, Save/Restore Utility Events/ / Timing Data Shared Memory – Ethernet Hardware w/ Synchronous Protocol Ethernet – EPICS Channel Access Protocol C P U E V R M T R S Read Remote Ethernet and serial I devices. High density IO, Motor Control C E Position Control / P V O U G C EL L C N ET I / O A L S N ET E V R C EL L …. . Field I/O …. . C EL L C P U E V R C EL L PS IOCs DCCT BPM Fast Equipment BPM Protection Signal BPM PS Pr. Mon BPM Field I/O PLCS , Slow I/O, High Reliability, Low Accuracy, High Density Vacuum, PPS, MPS, Non FOFB PS, Cryo. , Facility control E V R PS IOCs PS BPM Field I/O C P U BPM IOCs BPM Timing Master Instrumentation Ethernet P L C C P U 11 BPM PS PS Non FOFB Diags. BPM BROOKHAVEN SCIENCE

Control System Standards - 4 of 4 • • • Hardware Components are being selected Dell Linux development workstations PLC Solutions Siemans Allen Bradley Control Logix Building Automation ALS Needs fast Ethernet based interface board VME crates Rittan 4 slot VME 64 x-2 U 4 S-PS 300 C-SM 4, 395. 00 Rittan 7 slot VME 64 x-4 U 7 S-PS 900 C-SM 5, 510. 00 Weiner 9 slot VME 195 x. PO 6, 050. 00 Weiner 21 slot VME 6023/611_JL 7, 271. 00 CPU Boards Motorola MVME 5500 3, 000. 00 PP 410 3, 500. 00 Motor Controllers Hytec 8601 with motor driver 32 axis ~1, 000/axis Newport XPS Intelligent motion controllers 12 BROOKHAVEN SCIENCE

Development Opportunities • • • Embedded Controllers need an open standard for high speed, deterministic functions. Work with other labs and board manufacturers to develop one. High Level Applications currently tie together functions through data or file structures. To make the components of High Level Applications modular and distributed, a client/server architecture is needed. Relational Databases support data management through the life of a project. Development of adequate tools to enter and report this data is required early. 13 BROOKHAVEN SCIENCE

Embedded Device Control – 1 of 3 Physics Applications: Correlation plots, orbit calculations, beam dump data Events/ / Timing Data – Beam Trip Event, Synch Data Event 2 GB Shared Memory – Communicates Cell Data @ 20 usecs Ethernet – EPICS Channel Access Protocol C P U E V G C EL L Timing Master C P U E V R C EL L BPM IOCs Fast Equipment Protection Signal C P U E V R C EL L Compute Fast Orbit Feedback PS IOCs BPM PS BPM 14 BROOKHAVEN SCIENCE

Embedded Device Control – 2 of 3 RF Clock Distribution 45 MHz +T 0 + - Settling time on BPMs Fiducial Distribution 1 Hz +0. 0 µsecs - BPM to Compute Controller 512 bits = 64 bits * 8 BPMs 5. 18 usecs = 518 bits over 100 MBit enet +6. 5 µsecs - Compute Controllers to each other 15, 360 bits = 30 nodes * 512 bits BPM Controller /OC EPICS PCIx Interface BPM Controller Prev Cell Core Controller BPM Controller I/OC EPICS PCIx Interface 15. 4 usecs = 15, 360 bits over 2 GB enet +21. 9 µsecs – Compute local matrix 0 usecs + 21. 9 µsecs - Communicate t Power Supply Controllers 256 bits = 4 PS * 64 bits each (v and h) 2. 56 usecs +25. 8 µsecs – loop complete settling time for magnets PS Controller Core Controller I/OC EPICS PCIx Interface PS Controller communicate diagnostic waveforms etc… 100 MB ENET +200 µsecs – start again 15 Next Cell Core Controller 2 GBit ENET BROOKHAVEN SCIENCE

Embedded Device Control – 3 of 3 • FY 08 • Status: – Redundant 1. 1 GByte Fibers transmitting – Master clock is distributed over grey wires in the test rig – 163 nsecs of latency to transmit through the board – Check character every 8 kb, no lost packets. • • Carryover 60% of ‘ 08 money from late start FY 09 • Cell Controller Development – Integrate and test communication core – Develop the interface from the cell controller to a processor – for integration into EPICS – Integrate external clock and fiducial • Device Controller Development – Develop the 100 Mbit interface on Cell Controllers – Test on device controller core with 100 MBit comm lines – Begin integration of these device controllers - Libera - Power supply control - LLRF control 16 BROOKHAVEN SCIENCE

High Level Applications – 1 of 4 Design Goal Lattice Physics Applications (Thin Client) Optics Resp. Matrix (S) Model Server Physics Applications (Thick Client) Measured Orbit Differences Conversions Name Mapping Configuration Parameters Optics Deviations Gradient Errs & Corrections Beam R. M. Diff’s Data Server Mid Level Data Client/Server Application / Family Channel Access Computed Data EPICS Client/Server Need to port Need to develop Under development Distributed IOC Process Databases Existing Dipole Quad Sext. Corr. BPM RF Designed lattice & installed hardware seq 17 BROOKHAVEN SCIENCE

High Level Applications – 2 of 4 Online simulation and physics tools are operational MMLT/AT XAL/Tracy Python/Tracy High level Low level EPICS Client/Server VIOC Real Machine Tracy Simulation Engine Distributed IOC process databases Dipole Quad Sext. Corr. BPM RF 18 RF BROOKHAVEN SCIENCE

High Level Applications – 3 of 4 MMLT Setorbit – orbit correction (against VIOC) Kick beam: 1 e-5 for 1 st H&V Run setorbitgui (with 5 iterations) Orbit reset to 0 after correction 19 BROOKHAVEN SCIENCE

High Level Applications – 4 of 4 • • • High Level Application environment is available in XAL, Matlab Middle Layer Toolkit and Python An online simulation is running under the IOC in Tracy 3 Libraries for HLA for online model will be done for AT and Tracy 3 DDS open source data transport protocol installed and under evaluation Direction • • • Evaluate alternative communication protocols: Extend Channel Access or Tine Choose a protocol for the high level application environment (12 months) Implement communication 20 • identify the client/server architecture that is required for orbit correction • Create the XML data grams that would be required for this application • Prototype the communication using DDS • Simulate the Servers that would be required for Orbit Correction BROOKHAVEN SCIENCE

IRMIS – 1 of 3 Java Client API Server XML protocol (REST style WS) Data Service layer Few database utilities: backup, consistency check, etc… Applets and Widgets Integration with external tools (i. e. physcs) Java. Script bridge 3 rd party Perl/Pyton scripts Client Web applications 3 rd party Java applications Application Architecture Supports Independent Development Database layer 21 BROOKHAVEN SCIENCE

IRMIS- 2 of 3 Web Based Reports Editor Component Type Component Lattice EPICS Database Name Mapping Wiring Scripts Component Type EPICS Database Components Lattice EPICS Database Wiring EPICS Database Name Mapping Wiring Component Lattice Name Mapping Files for control Lattice EPICS Database Complete FY 08 Complete FY 09 22 Name Mapping BROOKHAVEN SCIENCE

IRMIS – 3 of 3 23 BROOKHAVEN SCIENCE

Design Maturity • Hardware standards are being evaluated for functionality, reliability, and cost for crates, processor boards, PLC family, and motion control. • Open Architecture Hardware standards are being developed to accomplish high speed, low latency applications such as fast orbit feedback, timing, and machine protection. Commercial solutions can provide some level of functionality if a fallback position is needed. We are working with manufacturers and in-house board developers to keep them appraised of the design and progress with the prototype. • Hardware for fast and high precision applications will be standardized as late as possible to take advantage of the latest hardware developments that can meet our requirements in the most cost effective manner. A good example of this is high speed digitizers for the ring current monitor – where the cost of good solutions have gone from $130 K down to $30 K and threaten to fall further. • Final design is scheduled to be done in 2009 as the detailed requirements for each of the subsystems are documented and the prototypes are developed and proven. 24 BROOKHAVEN SCIENCE

Design Maturity Requirements for CD-3 Approval Control System Design is 50% Complete All tools are developed in the EPICS community to provide all major engineering functions Real Time OS – does not affect any code. Software supported on RTEMS, vx. Works, Linux etc. . Hardware standards exist to meet all requirements Development being done provides alternate solutions that reduce cost and improve productivity Prototypes being developed in FY 09 will be used to make the final design Detailed device lists developed in FY 09 will be used to verify communications design and prepare for execution in FY 09. 25 BROOKHAVEN SCIENCE

Design Maturity Requirements for CD-3 Approval Control System Design is 50% Complete 26 BROOKHAVEN SCIENCE

Remaining Design Issues • • Remaining design issues for construction start of Conventional Facilities: None. Other Remaining Design Issues: None. 27 BROOKHAVEN SCIENCE

Near Term Plans • • • Prototype subsystem test stands for asynchronous applications • Vacuum, facility control, machine protection, and personnel protection. Prototype subsystem test stands for synchronous applications • Diagnostics, power supplies, and LLRF Develop first two physics application in a client/server architecture – orbit display and correction. Develop tools for lattice and wiring entry and browsing. Develop tools for exporting lattice to different platforms. Verify network design against all data requirements. 28 BROOKHAVEN SCIENCE

Concluding Remarks • • • Hardware standardization is being aggressively pursued with PLCs, processors, and crates being evaluated. (*) All subsystems should be prototyped in FY 09, early FY 10 time frame. We have critical mass to develop and prototype critical items in all areas of development and we continue to recruit and hire as budget authority permits (*). Early Success in Staffing. Aggressive recruiting continues. To address this risk we are using aggressive recruiting and staff augmentation through the use of contract labor from companies and individuals with good EPICS experience. Design Maturity is adequate for controls to assure delivery while taking advantage of the rapidly changing market. (*) Comments from the previous review that are being addressed. Also, coverage of wiring costs was confirmed. 29 BROOKHAVEN SCIENCE

EVMS – RF Controls (1 of 9) Controls is within its budet and performance parameters 30 BROOKHAVEN SCIENCE

EVMS – RF Controls (2 of 9) RF Subsystem Work Postponed to FY ‘ 09 31 BROOKHAVEN SCIENCE

EVMS Diagnostics Controls (3 of 9) Diagnostic Subsystem Postponed to FY ’ 09 – small bit of design done 32 BROOKHAVEN SCIENCE

EVMS EPS Control (4 of 9) EPS Subsystem Postponed to FY ’ 09 – no money to speak of 33 BROOKHAVEN SCIENCE

EVMS Control Room Costs (5 of 9) Control Room Postponed to FY ’ 11 – This was in the wrong year 34 BROOKHAVEN SCIENCE

EVMS Timing Control (6 of 9) Timing Subsystem Postponed to FY ’ 09 – small bit of design done 35 BROOKHAVEN SCIENCE

EVMS Operations Tools (7 of 9) IRMIS Development work is on track. The extra costs reflects 1. 3. 5. 13 work done here…. 36 BROOKHAVEN SCIENCE

EVMS High Level Applications (8 of 9) High Level Application Work – Great progress, costs went to 1. 3. 5. 11 37 BROOKHAVEN SCIENCE

EVMS Control Management and Travel (9 of 9) Management and travel costs 10% underestimated. Some costs to subsystem in future 38 BROOKHAVEN SCIENCE

Backup – Subsystem Interface Design • • • Power Supply Conventional Facilities Vacuum RF Beam Lines Diagnostics 39 BROOKHAVEN SCIENCE

Power Supply Interface Operator stations: Displays, Archiving, Alarm Management, Strip charts, Save/Restore Utility Events/ / Timing Data Shared Memory Ethernet – EPICS Channel Access Protocol PS IOC E V R C P U C EL L PS IOC 90 Storage ………. Ring 4 Booster Ring 2 LINAC MPS PLC E V R C P U C EL L C E L L PS PS PS PS PS PS Fast Correctors Slow Correctors Fast Correctors 40 MPS PLC Slow Correctors BROOKHAVEN SCIENCE

Conventional Facility Interface Operator stations: Displays, Archiving, Alarm Management, Strip charts, Save/Restore Utility Events/ / Timing Data Ethernet – EPICS Channel Access Protocol C P U 5 1 per SR Equipment Building 1 for the injection Equipment Bldg. 1 for the cryo building E V R Instrumentation Ethernet A L C Field I/O N ET ME-LGR Multi Equipment - LAN Gate Router. . …. . Field I/O 41 BROOKHAVEN SCIENCE

Vacuum Interface Operator stations: Displays, Archiving, Alarm Management, Strip charts, Save/Restore Utility Events/ / Timing Data Ethernet – EPICS Channel Access Protocol 5 1 per SR Equipment Building 1 for the injection Equipment Bldg. 1 for the cryo building E V R C P U Instrumentation Ethernet 24 Port Digi P L C C N ET Field I/O I / O …. . I / O Vacuum Device Controllers Field I/O 42 BROOKHAVEN SCIENCE

RF Interface Operator stations: Displays, Archiving, Alarm Management, Strip charts, Save/Restore Utility Events/ / Timing Data Shared Memory Ethernet – EPICS Channel Access Protocol RF IOC C P U E V R Cav C EL L 1 Storage Ring ………. 1 Booster Ring 1 LINAC HP PLCs RF IOC C P U E V R C EL L HP PLCs Cav Cav LLRF 1 (up to 6) Storage Ring 1 Booster Ring 3 (or 4) LINAC LLRF 43 BROOKHAVEN SCIENCE

Beamline Interface Operator stations: Displays, Archiving, Alarm Management, Strip charts, Save/Restore Utility Events/ / Timing Data Shared Memory Ethernet – EPICS Channel Access Protocol BL IOC C P U MPS PLC E V R M TR LV DT PS IOC 6 -1………. per Beamline C P U MPS PLC E V R M TR LV DT MTR 44 BROOKHAVEN SCIENCE

Diagnostics Interface – BPMs / Loss Mon. Operator stations: Displays, Archiving, Alarm Management, Strip charts, Save/Restore Utility Events/ / Timing Data Shared Memory Ethernet – EPICS Channel Access Protocol Diag IOC C P U E V R C EL L Diag IOC 30 Storage ………. Ring 4 Booster Ring 2 LINAC A D C C P U E V R C EL L Instrumentation Ethernet BPM 2 Loss Monitors Instrumentation Ethernet BPM Photon BPMs BPM BPM BPM 2 Loss Monitors Per ID Photon BPMs BPM BPM ADC - At only 5 locations BPM BPM A D C BPM Per ID 45 BROOKHAVEN SCIENCE