Modern Accelerator Controls Modern Accelerator Control Systems Kazuro

Modern Accelerator Controls Modern Accelerator Control Systems Kazuro Furukawa, KEK for KEKB Control Group and Linac Control Group <kazuro. furukawa@kek. jp> Jun. 26. 2007. PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 1

Modern Accelerator Controls u. Accelerator Controls at KEKB and Linac u. Operational Software u. Considerations on Accelerator Controls in General u. Available Technologies u. Adaptive Reliabilities u. Summary PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 2

Modern Accelerator Controls KEKB and Linac PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 3

Accelerator / Control Systems Modern Accelerator Controls Mt. Tsukuba J-PARC KEKB PF-AR ATF STF PF Linac PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 4

Accelerator / Control Systems Modern Accelerator Controls Control Systems in KEK u. Operational Presently v. Linac, PF-AR, ATF, KEKB u. Under Construction v. J-PARC, STF u. EPICS v. KEKB, … Linac J-PARC PF PAC 2007, Albuquerque, NM, US PF-AR EPICS Group KEKB ATF / STF Kazuro Furukawa, KEK, Jun. 2007. 5

KEKB and Linac Accelerator Modern Accelerator Controls Increase of the Luminosity Feb. 2005 Continuous Injections May. 2000 Apr. 2003 Dual Bunch e+ PAC 2007, Albuquerque, NM, US Now Collision with Crab Cavities Kazuro Furukawa, KEK, Jun. 2007. 6

KEKB Controls Modern Accelerator Controls KEKB Control System (Hardware) u. Gb. E Fiber Optic Networks v. Single Broadcast Domain v. Central Control Room and 26 Local Control Rooms u. VME/IOC v~100 VME/IOC mostly with Power. PC CPU u. Field bus v~200 VXI thru MXI for BPM Instrumentations v~50 CAMAC for rf and Vacuum (inherited from TRISTAN) v~200 Arc. Net network segments for Magnet Power Supplies, and other field Controllers v. GPIB for Instrumentations, RS 232 C, Modbus+ for PLCs u. Host Computers v. HP-UX/PA-Risc, Linux/x 86 Controls Server v 3 Tru 64/Alpha with Tru. Cluster v. Several Linux v. Many Mac. OSX v(Solaris/Sparc for Vx. Works) PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 7

KEKB Controls Modern Accelerator Controls KEKB Control System (Software) u. EPICS 3. 1 and 3. 14. 6, 8 u. Vx. Works 5. 3. 1 mainly, and 5. 5. 1 v. Hope to upgrade EPICS/Vx. Works Shortly u. IOC Development v. Cap. Fast, (VDCT) Perl, SADscript for Database Configuration v. Oracle as a backend Database Management ³Migration towards Postgresql u. Operational Application Development v. MEDM(DM 2 k) for Startup v. Python/Tk for Equipment Controls v. SADScript/Tk for Beam Operation, etc PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 8

KEKB Controls Modern Accelerator Controls KEKBLOG and ZLOG u. KEKBlog/kblog Archiver is Used from the Beginning of the Commissioning v. Just less than 2 GB / day v. Several Viewer Tools ³Very often Used to Analyze the Operation Status u. Zlog Operation Log v. Zope, Python, Postgre. SQL ³Most of the operation logs ³In Mostly Japanese ³Figure Storing Integration w ex. Screen shot of operational Panels PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 9

Linac Controls Modern Accelerator Controls Linac; Physical Structure u. Multi-tier, Multi-hardware, Multi-client, … PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 10

Linac Controls Modern Accelerator Controls Linac; Multi-tier Logical Structure PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 11

Linac Controls Modern Accelerator Controls Software Architecture u Base control software structure for Multi-platform v v v any Unix, OS 9, Lynx. OS (Realtime), VMS, DOS, Windows, Mac. OS TCP - UDP General Communication Library Shared-Memory, Semaphore Library Simple Home-grown RPC (Remote Procedure Call) Library Memory-resident Hash Database Library u Control Server software v v Lower-layer servers (UDP-RPC) for control hardware Upper-layer server (TCP-RPC) for accelerator equipment Read-only Information on Distributed Shared Memory Works redundantly on multiple servers u Client Applications v Established applications in C language with RPC v Many of the beam operation software in scripting language, ³ Tcl/Tk ³ SADscript/Tk PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 12

Modern Accelerator Controls Operation PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 13

KEKB and Linac Operation Modern Accelerator Controls KEKB Commissioning Groups u. Formation of Commissioning Group (KCG) v. Linac Commissioning (LCG) ³ 7 from Linac ³~10 from Ring v. KEKB Ring Commissioning Group (KCG) ³All LCG ³~20 from Ring ³Several from Detector (BCG) v. Commissioning software base was formed during Linac Commissioning (1997~) Tcl/Tk , Python/Tk, SADscript/Tk PAC 2007, Albuquerque, NM, US KEKB Commissioning Linac Group Commissioning Group KEKB Ring Linac Kazuro Furukawa, KEK, Jun. 2007. 14

KEKB and Linac Operation Modern Accelerator Controls SADScript u. Mathematica-like Language v. Not Real Symbolic Manipulation (Fast) v. EPICS CA (Synchronous and Asynchronous) Ca. Read/Ca. Write[ ], Ca. Monitor[ ], etc. v(Oracle Database) v. Tk Widget v. Canvas Draw and Plot v. KBFrame on top of Tk v. Data Processing (Fit, FFT, …) v. Inter-Process Communication (Exec, Pipe, etc) System[ ], Open. Read/Write[ ], Bidirectional. Pipe[ ], etc. v. Greek Letter v. Full Accelerator Modeling Capability v. Also Used for non-Accelerator Applications v. Comparable to XAL, but very different architecture PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 15

Modern Accelerator Controls Virtual Accelerator in KEKB u. For Example in KEKB vmost Beam Optics Condition is maintained in the Optics Panel v. Other Panels Manipulate Parameters Communicating with the Optics Panel (Oide, Koiso, Ohnishi et al) ===> Tune Measurement/Changer PAC 2007, Albuquerque, NM, US Optics Panel Kazuro Furukawa, KEK, Jun. 2007. 16

Modern Accelerator Controls Beam Optics Database u. Repository of Inputs to Simulation Codes? u. XSIF Extended Standard Input Format v. Many Simulation Codes utilize it v. SAD does not v. Currently a Conversion Tool is Used to for These Input Formats v. XSIF (LIBXSIF) inclusion in SAD? u. Yet another Generalized Input Format? v. Separation between Beamline Geometry (relatively static) and Beam Optics (more varying) v. Could be structured into XML u. Relational information to each Hardware Components v. We do not prefer complicated relations by Oide PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 17

Modern Accelerator Controls PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 18

Accelerator Controls Modern Accelerator Controls u. Definition and goal v. Specified only after technical details of the accelerator is decided ³Of course the final goal is the science achievement v. Often change after commissioning ³Many prefer to flexibility as well as to robustness (depending on the purpose) ³Should support rapid development to realize novel ideas immediately v. Unfortunately we don’t have general accelerator controls ³We may have to make something PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 19

Accelerator Controls Modern Accelerator Controls History u. Discussion of accelerator controls v. At ICALEPCS conferences ³After some success of NODAL at SPS/CERN ³Needs for more general software tools v. NODAL was chosen at TRISTAN v. SLC/SLAC used Micros + VMS v. Standard model ³Field-network + VME + Unix + X 11 v. Software sharing ³Definition of a Class to represent whole accelerator w. Which was impossible v. More common control system with extended API ³nc. RPC/CERN, TACL/CEBAF, ACNET/Tevatron, etc ³EPICS got popular maybe because of the selection at SSC, APS, CEBAF, BESSY, … v. Then more object oriented software (naturally after RPC) ³More computer aided development possible ³CICERO/CERN, TANGO, CORBA+Java, CERN, … ³Windows/Microsoft, … PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 20

Accelerator Controls Modern Accelerator Controls No common controls yet v. Balance between many available technologies u. Object-oriented vs. Channel-oriented v. Object-oriented technology ³More support benefits from software engineering ³Extendable, clearer definitions ³Different people have different ideas on control objects v. Channel-oriented technology ³Flat (one-layer structure), simple, scalable ³Not much support from software engineering ³Easy to make gateways PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 21

Accelerator Controls Modern Accelerator Controls More balances u. Compiled language vs. interpretive language v. Two level languages ³Interpretive language for rapid prototyping ³Compiled language for established algorithms v. After too much success of NODAL v. Compiled languages programmed by expert ³Documentation, maintenance, policy-driven ³Manageable, then reliable v. Interpretive/scripting languages ³Rapid development w. Realization of novel ideas in hours ³Everyone attends the construction of operation environment ³Another level of management/maintenance required PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 22

Accelerator Controls Modern Accelerator Controls More balances u. Best & aggressive vs. moderate & conservative v. New technology is attractive ³But can be a “fad” ³Can we justify the choice? v. For longer life-span, which is better? ³Life of accelerator is often very long compared with w. User facilities w. Commercially available software/communication technologies ³Operational performance continuously advances v. Accumulation of operation knowledge base ³Stored mainly as software and database in the control system w. Beam stabilization algorithms, hardware startup procedures, etc v. It is valuable treasure ³There should be mechanism to keep such resources w. With longer life-span PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 23

Accelerator Controls Modern Accelerator Controls More balances u. International vs. de-facto standards v. International organizations pursue ideal solutions ³Sometimes they don’t become de-facto standards ³Selection of one of many standards is difficult v. Watching the market ³TCP/IP network, Unix/Windows operating system, VME boxes v. Advantages of de-facto standards ³Economical advantage to select products out of markets ³Save man-power avoiding proprietary development ³Solutions will be provided for the old standard in the next generation ³As a whole, it is good for long life-span PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 24

Modern Accelerator Controls Available Technologies PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 25

Available Technologies Modern Accelerator Controls PLC u. Programmable Logic Controllers (PLC) v. Rule-based algorithms can be well-adopted for simple controls v. IP network for the both controls and management were preferable ³Especially at KEK/Linac which has a policy of IP only field network v~150 PLCs at Linac since 1993, and also many at J-PARC v. Isolated/separated development becomes easy ³Outsourcing oriented v. Equipment developer oriented ³Many maintenance capabilities were implemented v. IEC 61131 -3 Standards ³ 5 languages, with emphasis on naming ³Not so popular in Japan ³Effort to make common development environment ³XML representation of resources ³Should be paid more attention v. Redundancy PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 26

Available Technologies Modern Accelerator Controls Network with only IP/Ethernet u. The policy chosen when we upgrade Linac in 1993 v. Make network management simpler ³Faster switches, routing, network-booting, etc. v. Avoid Hardware failure and analysis effort with old field network ³Home-grown field networks need much dedicated man-power v. Cost for optical Ethernet went down at around 1995 ³Linac has high-power modulator stations, noise source v. Nowadays many facilities have this policy with Gb. E ³J-PARC controls basically followed this v. More and more intelligent network devices ³ex. Oscilloscopes with Windows/3 GHz-Pentium built-in ³Even EPICS IOC, MATLAB, or others can be embedded v. Network components can be replaced one-by-one v. Security consideration will be more and more important PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 27

Available Technologies Modern Accelerator Controls FPGA u. Another “everywhere” after IP network v. Digital circuit and software can be embedded in to one chip ³Even CPU core is embedded ³Flexible and robust, wonderful platform for local controls w. Sometime terrible source of bugs v. Nano-second level timing v. More and more gates, memory, pins, etc v. More software support PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 28

Available Technologies Modern Accelerator Controls ATCA and TCA u. Advanced telecommunications computing architecture v. Accommodate several 100 ohm serial buses v. Gb. E or PCI-express, 10 Gb. E, etc v. Typically 14 slots in 19” and 12 -unit height v. Shelf manager manages healthiness of the system ³through Intelligent Platform Management Interface (IPMI) v. Many reliability improving facilities, redundancy, hot-swap, etc u. Micro. TCA v. More recently defined in 2006, based on Advanced. MC Mezzanine Card defined in ATCA v. Begin to have many facilities from ATCA PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 29

Available Technologies Modern Accelerator Controls EPICS u. Now is a kind standard, but … u. Object-oriented design support v. Naming scheme, and/or design of new record v. More software-engineering support favored ³Several different efforts to provide better environment w. Java IOC (M. Kraimer), Control system studio (M. Clausen), Data access (R. Lange) u. Security mechanisms v. User, Host-based protection available v. More security ³Dynamic controls of security ³Access logging u. Dynamic configuration of database v. Dynamic creation / loading of records v. Dynamic removal of records ³Maybe some part of the codes can be shared with redundant-IOC project PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 30

Available Technologies Modern Accelerator Controls Magnet Controls u. It is typical controls and still many things to do u. Many magnets and many power supplies ³No one-to-one correspondence v. Which hardware interface to use u. Procedures v. Interlock status, on/off, analog with some precision, etc v. Energy, kick - field - current conversions ³How to represent those conversion curves v. Timing synchronous operation ³for tune change, orbit correction, etc. v. Standardization PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 31

Available Technologies Modern Accelerator Controls Timing Event System u. Present Timing System v. Provides ~3 pico-second Timings to ~150 Devices v. Only 4 Events can be Distinguished v. VME(x 6) and CAMAC(x 10) u. Diamond Event System v. Single Fiber can Transfer Clock, Delayed. Timings, Events (256), Data Buffers (2 k-bytes) u. New IOC v. MVME 5500 v. RTEMS (developed at BNL) ³(May migrate to Vx. Works if KEKB upgrades Vxworks) v. EPICS Driver/Device Support from SLS/Diamond/SLAC/LANL PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 32

Modern Accelerator Controls Reliability PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 33

Reliability Modern Accelerator Controls Reliability u. The end user expect rigid reliable operations u. Inner layers need flexibilities ³Because of daily improvement v. Compromise between ³Practical or ideal solutions ³Aggressive and conservative ³Under restrictions of w. Time, safety, budget, man-power v. Here we think about adaptive reliability PAC 2007, Albuquerque, NM, US hardware Interface equipment controls beam controls linac ring accelerator physics beam delivery detector data acquisition computing physics, chemistry, medical treatment Kazuro Furukawa, KEK, Jun. 2007. 34

Reliability Modern Accelerator Controls Reliability Increase without much Cost u. There should be “right way” v. We hope to have it some day, but for now we need interims u. Surveillance for everything v. Well-arranged system does not need this, but… u. Testing framework v. Hardware/Middleware tests just before Beam v. Software tests when installed u. Redundancy v. In Many Hardware/Software components v. Of course some of them are Expensive, but… PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 35

Reliability Modern Accelerator Controls Surveillance for everything u. We have written too many pieces of software vwhich assume certain circumstances unfortunately ³which will fail some day vin scripting languages too rapidly and too easily ³without documentations u. We manage too many computers v. If only one, I’m almost sure I can make it stable ³But in reality even hostname can be mis-labeled u. We installed too many network components vwithout good network database etc ³which sometimes has bad routing information, etc PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 36

Reliability Modern Accelerator Controls Surveillance for everything u. If certain installation of (software/hardware) was not ideal v. Find out ³What is the most important feature of the installation? ³What is the easiest test for its healthiness? v. Routine test is carried automatically ³by cron or continuous scripts ³If an anomaly found, w. Alarm, e-Mail to the author, make error log w. Restart related software, if not critical w. Report to the human operator, if critical v. Not ideal, but effective under limited human resources PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 37

Reliability Modern Accelerator Controls Software Testing u. Moving operating environment v. For better resource performance ³We tend to do it because of the pressure from budget restrictions v. May lead to malfunctions ³We knew they may happen u. Automatic software (hardware) tests preferable v. Under new environment (machine, compiler, network, etc) ³Many kinds of important free software does them ³Language systems, Linux Test Project u. We do some tests v. But sometimes not enough v. More thoroughly prepared tests needed PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 38

Reliability Modern Accelerator Controls Testing Framework u. When we introduce new environment v. Unit test ³We don’t do it much yet ³EPICS began to have it, “make runtests” w. Collecting existent test cases w. User can provide tests in Perl/Test framework ³Hope to have for SAD and SADscripts v. Regression tests ³We have something, but not thorough, not exhaustive ³Difficult to collect cases v. Stress tests ³We do it during operation (? ) ³We know computers rarely fail, but network/network-devices do w. Find solution w. Development of surveillances w. Installation of failure-recovery or failover procedures PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 39

Reliability Modern Accelerator Controls Testing Framework u. When we start new run v. New software/hardware ³We test unit by unit ³But not through operational tools prepared v. Maintenance works ³We often forget to restore/initialize cables, switches, variables ³Power-stop may bring another annoyance u. We need routine procedures which include ³Hardware tests ³Name/ID matching ³Database tests ³Software component tests ³Software/Hardware simulation tests v. Before beam operation v. We do it mostly by operator observations based on written procedures v. CERN did some efforts PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 40

Reliability Modern Accelerator Controls Redundancy u. Do we need redundancy? ³Redundancy may be the last-resort measure ³It may cost v. Centralized facilities are easier to manage ³If I have only one server, my life is much easier v. But they become complicated monsters ³Nobody understand everything u. Especially useful for maintenance v. Not only for failure-recovery ³Redundant systems of complicated system; (complicated)2 u. Anyway we may have to prepare backups v. Then automatic failover is just around the corner ³And … PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 41

Reliability Modern Accelerator Controls File server redundancy u. RAID and Mirror-disks are used everywhere now u. We began to use Cluster software before KEKB v. DECsafe, Tru. Cluster for Unix v. Life. Keeper, Redhat-AS, Rose-HA for Linux v. Net. App u. It works at least for Hardware troubles; but sometimes for Software troubles Server-1 u. Maintenance and Scheduling became easier Server-2 RAID PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 42

Reliability Modern Accelerator Controls Network Redundancy u. Mostly established technologies w. Wide acceptance of Ethernet and IP v> 10 years ago ³Redundant Transceivers v. More recently Standards available ³Hsrp or Vrrp and Rapid spanning tree PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 43

Reliability Modern Accelerator Controls Redundant PLC’s u. CPU built-in redundancy is already used in several vendors v. Dual main memory with checksum at every-cycle v. ROM as well as flash memory ³Bad circumstances at field forced them to implement it u. We just started to evaluate redundant CPU’s u. Redundant PLC’s are used at CERN v. Siemens S 7, slightly expensive u. Several possibilities in architecture v. Single vs. dual backplane v. Power-supply, CPU, Network-interface v. I/O (? ) PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 44

Reliability Modern Accelerator Controls Redundant EPICS IOC u. Redundant controllers are favorable was in PLCs v. The project was started at DESY (M. Clausen) ³Redundancy monitor task (RMT) w. Monitors healthiness of controllers w. Manages primary redundancy resource (PRR) ³Continuous control executive (CCE) w. Synchronizes internal states ³Modifications for several others PRR’s w. Scan tasks, Channel access server tasks, Sequencer, Drivers w. Possibly user tasks v. KEK joined in for wider applications ³Linux (OSI) port ³Gateway applications v. ATCA implementation possible ³For ILC (? ), micro. TCA (? ) PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 45

Reliability Modern Accelerator Controls Software redundancy u. EPICS IOC redundancy is slightly complicated v. Since it has name resolution facility v. More advanced u. Linac/KEK controls is simpler v. Normally we run several middle-layer control servers ³on separate machines v. For EPICS gateway ³We need redundant IOC technology u. Other existent servers v. Recently more careful in redundancy ³Like dchpd ³Redundancy and replications PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 46

Modern Accelerator Controls Summary PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 47

Summary Modern Accelerator Controls Phronesis u. Aristotle’s view of wisdom. u. Contrary to Sophia; the ability to understand the universal truth u. Phronesis is the ability to find a way to achieve an overall goodness PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 48

Summary Modern Accelerator Controls Summary u. EPICS and SAD made KEKB a great success, but other accelerators have different criteria u. Accelerator controls design needs a balance between many aspects u. There are many good technologies waiting to be utilized u. Also more reliability features needed u. Share more experiences u. Phronesis PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 49

Modern Accelerator Controls Thank you PAC 2007, Albuquerque, NM, US Kazuro Furukawa, KEK, Jun. 2007. 50

Linac Controls PAC 2007, Albuquerque, NM, US Modern Accelerator Controls Kazuro Furukawa, KEK, Jun. 2007. 51