Control System Overview Control System Group Bob Dalesio

Control System Overview Control System Group – Bob Dalesio presenting NSLS-II ASAC Review March 26 – March 27, 2009 1 BROOKHAVEN SCIENCE

Outline • Design Requirements • Technical Requirements & Specifications • Cost & Schedule Baseline • Status • Staffing • Standards • Technical Development • Recent Accomplishments • Near Term Plans • Conclusions 2 BROOKHAVEN SCIENCE

Control System Requirements – 1 of 2 • • • Bunch Length 1 -40 psecs 2. 6 usec ring revolution Top off every 1 minute Top off bunch train 140 -300 nsec Top off damping time 10 -50 msecs (no extraction) • Slow control of orbit trims, quadrupoles, sextupoles, and insertion devices are slow but can be set synchronous to the orbit reference. 5 Hz updates to operators of up to 1000 chosen parameters Archive up to 20, 000 parameters at a rate of 1 Hz continually • • • Must scale to support 150, 000 physical I/O connections and 400, 000 computed variables 3 BROOKHAVEN SCIENCE

Control System Requirements – 2 of 2 • Transient Recording • • • Provide data for all control aspects (no hidden parameters) 5 KHz RF Feedback on beam phase 10 k. Hz orbit feedback, (100 usec loop time) • • • 20 msec equipment protection mitigation 1 Hz model based control 10 k. Hz power supply read backs triggered from timing sys 10’s of Hz Data Collection for RF loop correction. 80 psecs pulse to pulse timing jitter. • Take coherent turn by turn orbit data for up to 800 channels for 1024 turns • Latch the last 10 seconds of data from all parameters in the storage ring • Beam line needs 1 msec archiving over 1 minute for temperatures and positions • 360 BPMs (12 per cell) • 120 Corrector PS (4 per cell) • All data available to system with revolution identifier for turn by turn data correlation. • During top off, some beam lines will need 1. 1 - 1. 8 psecs of timing jitter 2 nsec timing resolution 4 BROOKHAVEN SCIENCE

Cost & Schedule Baseline • Baseline • • Manpower by FY 09 • 4 High Level Application Engineers • 2 RDB Architects • 9 Project Engineers • 1 EPICS Expert • 1 System Administrator Hardware • 156 IOCs w/ timing hardware • $1 M network hardware & 360 K timing distribution • $400 K Servers and Control Room Consoles Software Licenses $300 K Schedule • • Build synchronous distributed device controllers by end 2009 Install Physics Environment for Application development 2009 Install First release of component. lattice, and wiring RDB tools 2009 Prototype subsystems and detailed design by end 2009 Procure and implement subsystems in 2010 Install and test Subsystems in 2011/2012 Improve and manage RDB tools over the project to support design, installation, test, and maintenance. Extend Physics Environment to support installation, test, commissioning and operation. 5 BROOKHAVEN SCIENCE

Staffing • High Level Applications • Relational Database (IRMIS) • Project Engineers • EPICS • Nikolay Malitsky, Guobao Shen, and Jayesh Shah • Contract work or new hire • Don Dohan and Gabriele Carcassi • Contracts are being used to enhance our capabilities early in the project • • • Yuke Tian, TBD – power supplies Huijuan Xu – vacuum Rob Petkus - network Yong Hu, Kiman Ha – diagnostics Michael Davidsaver - RF David Dudley – facility and EPS control TBD – timing Daron Chabot, TBD – beam line control Contracts with other groups: Larry Doolittle et. al. LBL, Joe Meade BNL Contract employees for support (Sheng Peng, Steve Lewis, Steve Hunt, Cosylab, Observatory Sciences) • Ralph Lange (BESSY), Mauro Gianchini (INFN) temporary for 12/6 months • Position to fill 6 BROOKHAVEN SCIENCE

Control System Standards - 1 of 5 Nomenclature Standard is in place Psy: PI-Ssy: SITsy: TI<Dev: DI>Sg: Sg. I-SD System Device Signal Control System Naming Examples Comment S: C 30 -VA: G 1<SGV: A>Pos: 1 -Opn cell 30, vacuum, girder 1, sector gate valve A (upstream), open S: C 20 -FE: B<FV>Pos: 1 -Cls cell 20 BM front end fast valve S: C 20 -VA: G 1<IP: A>E: 1 -RB 1 st sputter ion pump at C 20 girder #1 S: C 20 -VA: G 4<CHMA: B> 2 nd aluminum chamber at C 20 girder #4 S: C 20 -VA: G 4<BLWR> S: C 24 -RF: G 1<TMP: B> S: C 24 -RF: G 1<CCG: B>P S: C 24 -VA: G 1<CCG: D>P rf shielded bellows at C 20 girder #4 TMP station at 2 nd SC cavity insulating vacuum cold cathode gauge at 2 nd SC cavity beam vacuum 7 BROOKHAVEN SCIENCE

Control System Standards 2 of 5 Archive Viewing Archive Analysis User Tools DM/MEDM/DM 2 K/EDM/JDM Alarm Handler / strip. Tool/ knob. Manager Connection Server WAN/Local Archive Access Archiving Channel Access Client (CAC) Connection Data Transfers Program Interfaces C, C++, Fortran, JAVA IDL / Matlab / Mathematica Active. X / DDE / Vis. Basic SDDS / SAD / tcl / PERL / Python State Notation Lang / FSQT EPICS Tools Reduce Programming Connection Data Transfers Configuration tools provide: Interfaces exist to add: Channel Access Server (CAS) data acquisition drivers for new hardware DB Engine Types Device Support Driver Support I/O Controller supervisory control state control steady state control model based control operator displays alarm management 8 BROOKHAVEN SCIENCE

Control System Standards – 3 of 5 Software standards are being evaluated • • IRMIS tools are used for all configuration data: lattice, components, wiring etc. Embedded Real-Time Operating System choice: RTEMS (LCLS, Spear, CLS) or Vx. Works (APS, SNS, Diamond, SLS) Linux Workstations running Debian Evaluate the possibility of participating in the CSS development Visual Database Configuration Tool w/ Modifications for Table Entry Use physics applications: Matlab Middle Layer Toolkit – MMLT (many light sources), e. Xtensible Accelerator Language - XAL (SNS/SLAC), Elegant (APS), Tracy online simulation (Diamond) Evaluate Use of Experiment Control Tools: SPEC (ubiquitous), Blu. Ice (SLAC), IDL, Syn. Apps (APS/EPICS), GDA (Diamond), CBass (BNL), Scientific Studio 9 BROOKHAVEN SCIENCE

Control System Standards – 4 of 5 Architecture for instrumentation is being prototyped Operator stations: Displays, Archiving, Alarm Management, Strip charts, Save/Restore Utility Events/ / Timing Data Shared Memory – Ethernet Hardware w/ Synchronous Protocol Ethernet – EPICS Channel Access Protocol C P U E V R M T R S I / O FOFB IOC Read Remote Ethernet and serial devices. High density IO, Motor Control C E Position Control P V G U C N ET I / O A L S N ET C P U C EL L E V R C EL L FC 1 RFBPM 1 PS IOCs DCCT FC 2 Fast Equipment Protection FC 3 Signal RFBPM 2 Pr. Mon Timing Master Instrumentation Ethernet P L C C P U C EL L RFBPM 8 SC 1 Field I/O …. . SC 2 XRay BPM 1 XRay BPM 2 Field I/O PLCS , Slow I/O, High Reliability, Low Accuracy, High Density Vacuum, PPS, MPS, Non FOFB PS, Cryo. , Facility control 10 Non FOFB Diags. SC 6 Correctors XRay BPM 4 BROOKHAVEN SCIENCE

Control System Standards – 5 of 5 • • • Hardware Components are being selected Dell Linux development workstations PLC Solutions – having pricing, checking reliability, prototyping test stands. Siemens Allen Bradley Control Logix Building Automation ALS Needs fast Ethernet based interface board VME crates Rittal 7 slot VME 64 x-4 U 7 S-PS 900 C-SM 4, 100. 00 Weiner 9 slot VME 195 x. PO 4, 100. 00 V-Ross u. System u. TCA w/MCH & Intel CPU 5, 000. 00 CPU Boards Motorola MVME 3100 1, 400. 00 Motor Controllers Oregon Micro. Systems OMS Max. V 1, 600. 00 11 BROOKHAVEN SCIENCE

Development Opportunities • • • A Synchronous Device Interface that implements an open standard for high speed, deterministic functions provides a modular platform on which to develop a Fast Orbit Feedback system. High Level Applications currently tie together functions through data or file structures. To make the components of High Level Applications modular and distributed (and therefore reusable), a client/server architecture is needed. Implement asset management in the relational database. Development of adequate tools to enter, report, and track 12 equipment and BROOKHAVEN SCIENCE

Fast Feedback (1 of 4) 13 BROOKHAVEN SCIENCE

Fast Feedback (2 of 4) 14 BROOKHAVEN SCIENCE

Fast Feedback (3 of 4) Latency calculations: Data per BPM Data per cell Total data to be distributed GTP 1. 25 Gbps 8 -bit datawidth GTP 2. 5 Gbps 16 -bit datawidth 10 bytes (X: 4; Y: 4; Status: 2) 120 Byte (8 BPM 4 XBPM) 120 * 30 = 3. 6 Kbyte 3. 6 Kbye/1 Gbps = 28. 8 us 3. 6 Kbye/2 Gbps = 14. 4 us 3. 6 Kbye/2 Gbps =14. 4 us FPGA: 125 MHZ FPGA: 250 MHZ FPGA: 125 MHZ 15 BROOKHAVEN SCIENCE

Fast Feedback (4 of 4) SDI core design: LBL BNL: BPM measurement, GTP interface test, Integration GTP interface test 1. fiber loop back two SFT ports 2. Data is internal counter 16 BROOKHAVEN SCIENCE

High Level Applications – 1 of 4 Design Goal Lattice Physics Applications (Thin Client) Optics Resp. Matrix (S) Model Server Physics Applications (Thick Client) Measured Orbit Differences Conversions Name Mapping Configuration Parameters Optics Deviations Gradient Errs & Corrections Beam R. M. Diff’s Data Server Mid Level Data Client/Server Application / Family Channel Access Computed Data EPICS Client/Server Need to port Need to develop Under development Distributed IOC Process Databases Existing Dipole Quad Sext. Corr. BPM RF Designed lattice & installed hardware seq 17 BROOKHAVEN SCIENCE

High Level Applications – 2 of 4 Online simulation and physics tools are operational MMLT Elegant AT Tracy XAL/Tracy Python/Tracy High level Low level Real Machine Sext Corr. BPM . VIOC Model API Distributed IOC process databases Dipole Quad EPICS Client/Server Elegant Simulation Engine Tracy Simulation Engine RF Dipole Quad Sext Corr. BPM . 18 rf Dipole Quad Sext Corr. BPM . BROOKHAVEN SCIENCE rf

High Level Applications – 3 of 4 MMLT Setorbit – orbit correction (against VIOC) Kick beam: 1 e-5 for 1 st H&V Run setorbitgui (with 5 iterations) Orbit reset to 0 after correction 19 BROOKHAVEN SCIENCE

High Level Applications – 4 of 4 • • • High level application development environment is available for developing commissioning tools based on MMLT, XAL, or Python. Online simulations of Tracy 3 and Elegant provide a solid EPICS environment for testing these tools. The API for model servers is being formulated while these online simulations are brought online. The DDS API is being prototyped as the interface for these applications. Channel Access is being overloaded in version 3 and extended in version 4 to provide a client/server environment for these applications. 20 BROOKHAVEN SCIENCE

IRMIS – 1 of 4 Java Client API Server XML protocol (REST style WS) Data Service layer Few database utilities: backup, consistency check, etc… Applets and Widgets Integration with external tools (i. e. physcs) Java. Script bridge 3 rd party Perl/Pyton scripts Client Web applications 3 rd party Java applications Application Architecture Supports Independent Development Database layer 21 BROOKHAVEN SCIENCE

Database Status – 2 of 4 Web Based Reports Configuration Tools Component Type Component Lattice Component Type EPICS Database Component Name Mapping Lattice Wiring EPICS Database Name Mapping Scripts Component Type Component Lattice EPICS Database Name Mapping Wiring Develop Use Cases Lattice EPICS Database Name Mapping Wiring Files for control Lattice EPICS Database Complete Release 1 Complete FY 09 Complete FY 10 Name Mapping Tools develop in subsequent releases as users are added. 22 BROOKHAVEN SCIENCE

IRMIS – 3 of 4 23 BROOKHAVEN SCIENCE

IRMIS 4 of 4 Application prototype Lattice Deck manager using IRMIS HLA App Deck Input Deck to IRMIS Object Mapping IRMIS Data Service Deck Parser Tracy/Elegant HLA App Deck Generator Other Lattice Information IRMIS to Deck Object Mapping Tracy Elegant Deck Tracy Deck Elegant Simulation Server (Elegant) Simulation Server (Tracy) 24 BROOKHAVEN SCIENCE

Recent Accomplishments • Prototype subsystems started for slow applications • Prototype subsystems started for beam correlated applications • • • Vacuum, beam line and personnel protection. • Diagnostics, power supplies, and LLRF Prototype protocols running for High Level Applications architecture DDS deployed as an interface to some channel access mechanisms. Standard interface for orbit control running. IRMIS API for lattice entry and report is running and integrated into the online model environment. Preliminary design documents for vacuum, power supply, diagnostics, and network started. (*) 25 BROOKHAVEN SCIENCE

Near Term Plans • • Prototype subsystem test stands for facility control and personnel protection. Prototype subsystem test stands for diagnostics, timing and LLRF. Expand DDS interface to all channel access mechanisms. Prototype HLA communication requirements on CA V 4 Prototype the Twiss Server. Develop tools for lattice and wiring for entry and browsing. Complete preliminary design documents for vacuum, power supply, diagnostics, and network (*). Write the preliminary design documents for timing, 26 BROOKHAVEN SCIENCE

Concluding Remarks • • We have had good success in staffing. We are putting staff augmentation capability in place through the use of contract labor and temporary employees. Choosing EPICS reduces the programming needed to accomplish the engineering control tasks. The control team is developing the preliminary design documents and interface control documents for subsystems that have project engineers on board. The others will follow shortly(*) Hardware tests will be in place for crates and processors in the next 2 months. (*) All subsystems will be prototyped by early FY 10. Development in the relational database is keeping pace with the needs of the users of the relational database. Development of the fast orbit feedback hardware now has reasonable momentum. A full working prototype based on the Avnet board should be operational in August. The high level application environment already supports several models and sets of commissioning tools. The development of a client/server architecture is progressing well. (*) Comments from the previous review that are being addressed. Also, coverage of wiring costs was confirmed. 27 BROOKHAVEN SCIENCE

Backup – High Level Applications • DDS open standard as an API for the high level applications client and server. 28 BROOKHAVEN SCIENCE

DDS-Based High Level Application Environment Conceptual Solution: Start the implementation of the DDS specification in the form of the EPICS extension based on the Channel Access protocol EPICS-DDS 29 BROOKHAVEN SCIENCE

Client/Server Architecture for High Level Apps High Level Application Environment – Matlab Middle Layer Toolkit, XAL Client API – CAV 3: what is used now for channel data not enough TINE: is used for channel data – large interface DDS: superset of what is needed. Evaluate for packing/unpacking data CAV 4: will be applied to the interface definition Protocol – CAV 3: could overload array packet for early tests TINE: could also overload array packet for early tests Open-DDS: evaluated and deemed too slow – based on Corba ICE: evaluated and deemed to complicated to deploy CAV 4: will be applied to the payloads we determine are needed for HLA Server API – CAV 3: what is used now for channel data not enough TINE: is used for channel data – large interface DDS: superset of what is needed. Evaluate for packing/unpacking data CAV 4: will be applied to the interface definition 30 BROOKHAVEN SCIENCE

Status Prototype / working solutions of the following aspects: http: //sourceforge. net/projects/epics-dds • Middle-layer server [ ca. Example. App ] • Application-specific fixed structures based on the EPICS waveform record, array of characters [ Twiss. App ] • Synchronous access [ ca. Example. App ] • Asynchronous access [ ca. Monitor. App ] • Benchmark use case [ ca. Time ] 31 BROOKHAVEN SCIENCE

Backup – Subsystem Interface Design • • Network Power Supply Conventional Facilities Vacuum RF Beam Lines Diagnostics 32 BROOKHAVEN SCIENCE

Subsystem I/F - Network 33 BROOKHAVEN SCIENCE

Subsystem I/F - Network Any. LAN Pre-tapped, redundant STAR topology • Pre-spliced = >$ material, <$ installation • On-site splicing = <$ material, >$installation • Little margin for error • (2) 60 -fibre bundles in each direction = 120 x 2 • (1) tap/cell, (4) fibres/tap, Optitip -> LC harness • (2) spare fibres/cell = security, growth 34 BROOKHAVEN SCIENCE

Subsystem I/F – Fast Orbit Feedback Interface Operator stations: Displays, Archiving, Alarm Management, Strip charts, Save/Restore Utility Events/ / Timing Data Fast Orbit Feedback Synchronous Bus Ethernet – EPICS Channel Access Protocol FOFB IOC C P U C EL L 30 Storage ………. Ring 2 Injection FC 1 RFBPM 1 FC 2 RFBPM 2 FC 3 DCPS 01 DCPS 02 RFBPM 8 SC 1 SC 2 XRay BPM 1 XRay BPM 2 SC 6 Correctors DCPS 39 DC Power Supplies XRay BPM 4 35 BROOKHAVEN SCIENCE

Subsystem I/F – Conventional Facilities Operator stations: Displays, Archiving, Alarm Management, Strip charts, Save/Restore Utility Ethernet – EPICS Channel Access Protocol 5 1 per SR Equipment Building 1 for the injection Equipment Bldg. 1 for the cryo building C P U Instrumentation Ethernet A L C Field I/O N ET ME-LGR Multi Equipment - LAN Gate Router. . …. . Field I/O 36 BROOKHAVEN SCIENCE

Subsystem I/F – Vacuum Operator stations: Displays, Archiving, Alarm Management, Strip charts, Save/Restore Utility Ethernet – EPICS Channel Access Protocol 5 1 per SR Equipment Building 1 for the injection Equipment Bldg. 1 for the cryo building C P U Instrumentation Ethernet 24 Port Digi Per Cell P L C C N ET Field I/O I / O …. . I / O Vacuum Device Controllers 30 x Field I/O 37 BROOKHAVEN SCIENCE

Subsystem I/F - Vacuum • Super period Vacuum - Even Cell S 1 LS S 2 S 3 S 4 S 5 S 6 Tota l IP TSP NEG- TM CC -C S P G 2 2 2 1 1 8 1 1 1 1 7 TC G 1 1 RG A 1 GV 2 1 1 2 38 1 1 4 2 2 2 BROOKHAVEN SCIENCE

Subsystem I/F - Vacuum • Super period Odd Cell S 1 SS S 2 S 3 S 4 S 5 S 6 Total IP TSP NEG- TM -C S P 1 1 1 2 1 1 7 1 1 1 1 1 6 CC G 2 TC G 1 1 RG A 1 GV 2 1 1 2 39 1 1 4 2 2 2 BROOKHAVEN SCIENCE

Subsystem I/F - Vacuum • Injection device vacuum - Even Cell IDFE IP TSP NEG TM CC TC -C -S P G G 4 4 4 1 4 2 RG A 2 GV FV 2 1 • Injection device vacuum - Odd Cell IDFE IP TSP NEG TM CC TC -C -S P G G 4 4 1 4 2 40 RG A 2 GV FV 2 1 BROOKHAVEN SCIENCE

Subsystem I/F - Vacuum • Beam line front end Vacuum BMFE IP TS NEG- NEG P C -S 4 4 TM CC P G 1 4 TC G 2 RG A 2 GV FV 2 1 • Note: • IP, TSP will be installed as a unit • NEG-C and NEG-S will be activated during maintenance period using portable power supply • RV will be operated manually and will not be monitored by control system 41 BROOKHAVEN SCIENCE

Subsystem I/F - RF Operator stations: Displays, Archiving, Alarm Management, Strip charts, Save/Restore Utility Events/ / Timing Data Phase Control Synchronous Bus Ethernet – EPICS Channel Access Protocol RF IOC C P U CFC C EL L 4 Storage Ring ………. 1 Booster Ring 7 LINAC HP PLC RFP CFC RFP DSP 42 BROOKHAVEN SCIENCE

Subsystem I/F – Beam line Operator stations: Displays, Archiving, Alarm Management, Strip charts, Save/Restore Utility Events/ / Timing Data Ethernet – EPICS Channel Access Protocol BL IOC C P U MPS PLC E V R M TR LV DT PS IOC 6 -1………. per Beamline C P U MPS PLC E V R M TR LV DT MTR 43 BROOKHAVEN SCIENCE

Subsystem I/F – SR Diagnostics Operator stations: Displays, Archiving, Alarm Management, Strip charts, Save/Restore Utility Events/ / Timing Data Ethernet – EPICS Channel Access Protocol C P U E V R D C CT F CT C P U E V R F I R E C P U E V R Camera BLM 00 Camera BLM 01 Camera BLM 02 Camera BLM 59 Camera 44 BROOKHAVEN SCIENCE

Subsystem I/F – SR Diagnostics Operator stations: Displays, Archiving, Alarm Management, Strip charts, Save/Restore Utility Events/ / Timing Data Ethernet – EPICS Channel Access Protocol C P U E V R M TR C P U E V R F I R E C P U E V R Camera TUNE Camera OSCILLATOR 45 BROOKHAVEN SCIENCE

Design of Diagnostics Controls for Injector (Linac & Lt. B) Linac: Lt. B transport line : • IOCs for Linac & Lt. B diagnostics (2 in total) • • One IOC (PCIe/Linux) for 3 WCMs + 2 FCTs One IOC (micro. IOC/Linux) for 7 Screens via firewire & repeaters (longer distances than 4. 5 m) 46 BROOKHAVEN SCIENCE

Design of Diagnostics Controls for Injector (Booster & Bt. S) Booster: Bt. S transport line : • IOCs for Booster & Bt. S diagnostics (3 in total) • • • One IOC (PCIe/Linux) for 1 DCCT + 2 FCTs One IOC (micro. IOC/Linux) for 12 Fluorescent screens via firewire & repeaters One IOC (PC/Linux) for 29 BPMs in injector 47 BROOKHAVEN SCIENCE

Design of Diagnostics Controls for Storage ring 48 BROOKHAVEN SCIENCE