KEKB Accelerator Control System KEKB Controls Group 2001
KEKB Accelerator Control System KEKB Controls Group 2001 -08 -29 by T. Katoh
Control System Control Computer System l Timing Control System l Beam Gate Control System l Personnel Safety Control System l Communication System l Control Room and Operators Consoles l
Contents Construction History of KEKB Accelerator Control System l KEKB Control System Design l System Hardware Configuration l System Software Configuration l Central Control Room l
Construction History 1994. 4 Started System Design for KEKB 1995. 2 Controls Systems Evaluation 1995. 5 Committee Recommendation to adopt EPICS 1995. 8 First Version of Computer System Specifications 1996. 1 Started to Dismantle TRISTAN Main-Ring 1996. 3 Final Version of Specifications 1996. 6 Opened the Bids 1997. 3 abco 1 and 15 IOCs were Installed 1997. 4 Hardware Specifications were Fixed 1997. 10 BT Lines were Commissioned 1998. 3 The Rest of IOCs were Installed
Construction History(contd. ) 1998. 4 Started Software Installation with Link. Persons and Engineers from the Company 1998. 12 Commissioning of KEKB Accelerators 1999. 3 Commissioning with the BELLE Detector 1999. 4 Started Designing PF-AR Control System 2000. 10 Installation of abco 2 and 10 IOCs for PFAR 2001. 2 Started to Dismantle PF-AR Devices 2001. 7 Started to Install or Re-install Equipment 2002. 1 Commissioning of PF-AR
KEKB Control System Design l System Requirements l Constraints l Basic Concepts l System Architecture
System Requirements l Sources – Equipment Groups • • • Magnet and Power Supply RF Beam Monitor Vacuum Beam Transport Feedback, Facilities, Physics, etc. – Operations Group – Accelerator Physicists or Commissioning Group – Controls Group
System Requirements(contd. ) l Requirements – All the data that are possible to take should be taken. – All the data that are taken should be saved for later analyses. – All the operation should be recorded for later inspection. – All the machine parameters and information about the machine components should be stored in the database.
System Requirements(contd. ) l Requirements(contd. ) – The Man-Machine Interface should be Operator. Friendly. – The Programming Environment should be Programmer-Friendly. – The Overall Response Time to an Operator’s Request should be less than a Second unless the Progress of the Process is Indicated.
Constraints l Use CAMAC as an Equipment Interface l Schedule – – – Control Systems Final Design January, 1995 Removal of MR equipment January, 1996 Control Systems Installation September, 1996 Completed Hardware Installation March, 1997 KEKB Commissioning December, 1998
Constraints(contd. ) l Limited Man-Power – 9 KEKB Controls Group Members • 3 Physicists and 6 Engineers – 12 Link-Persons from Other Groups • • • 2 from Magnet and Power Supplies 2 from RF 2 from Vacuum 2 from Beam Transport 2 from Beam Monitor and Feedback 1 from Linac, 1 from Physics
Basic Concepts l Standard Model Architecture l International Standards – CAMAC, VME, VXI, GPIB, FDDI, etc. l Existing Software Environment : EPICS l Separate Computer-bus and Field-buses l “Link-person” System : Applications l “Out-Sourcing” : 5 People from Companies
Standard Model Architecture l l l Presentation Layer (Server) – Man-Machine Interface(OPI) – Logging, Analyses, Alarm Displays, Database, Global Feedbacks Equipment Control Layer (IOCs) – Data Acquisition, Sequence Control, Local Feedbacks Device Interface Layer (Field-buses) – Hardware Interfaces
The most Important Points l Presentation Layer – Software Development Tools • User/Programmer Friendly Tools – Ergonomic Consoles • LCDs, Macintosh’s, PCs, PDPs, Flat Desks l Equipment Control Layer – VMEbus : High Reliability, Flexibility l Device Interface Layer – CAMAC : High Reliability, Well-known
Presentation Layer l l l l Operator’s Consoles : X-Terminals Database Management : ORACLE 7 Alarm Generation/Recording Data Logging Data Display : CATV Network Simulation : SAD Program High-Speed Network – FDDI, Distributed Shared-Memory Network, – 100 Base/TX and 10 Base/T l Gateway to KEK Laboratory Network : acsad
Equipment Control Layer l Provides Standard Interfaces – CAMAC Serial Highway Drivers(Hytec) – MXI-bus Drivers for VXI Main Frames(HP) – ARCNET Drivers, GPIB Drivers(NI), etc. l Computers : VMEbus based IOCs – – l FORCE CPU-40(MC 68040) FORCE CPU-60(MC 68060) FORCE Power. Core 6603(PPC 603 e) FORCE Power. Core 6750(PPC 750) Operating System : Vx. Works
Device Interface Layer l Standard Interfaces – CAMAC : RF and Vacuum – VXI : Beam Position Monitors – GPIB : RF, Vacuum, Magnet Readouts, etc. – ARCNET : Magnet Power Supplies TV Signal Switches – MODBUS+ : PLCs – RS 232: Vacuum Measuring Instruments
System Architecture(1) l Functional Configuration Laboratory Network CATV Gateway Display Alarm Logging Simulation Console Database High-Speed Network Console Presentatikon Layer VME Equipment Control Layer CAMAC CAMAC Device Interface Layer
System Architecture(2) l Actual Configuration Laboratory Network CATV Gateway Display Alarm, Logging, Database, Console, Simulation FDDI Network Switch X-Term Consoles F/E VME X-Term Presentation Layer Equipment Control Layer CAMAC CAMAC Device Interface Layer
System Configuration Consoles & Peripherals Server Workstation FDDI Switch Central Control Building Local Control Building CPU Module RS 232 Module CAMAC Serial Highway Driver Module VME-MXIbus Driver Module GPIB Controller Module ARCNET Driver Module Power Supply Unit VXI Main Frame Measuring Instruments Magnet Power Supplies Various Equipment CAMAC Crates
Central Control Room X Terminals for Consoles Cisco Catalyst 1200 Bridge Phaser 550 JX 2 QMS 2425 Lattis System 810 M Cisco Catalyst 1200 Bridge Cisco C 1400 Concentrator Server abco 1 ME/RK 460 26 Local Control Rooms FDDI GIGA Switch VME IOC Timing, etc.
abco 1 Server Workstation l PA-RISC 7200 Architecture – 120 MHz CPU Clock – 4 CPUs – 2 GB of Main Memory – 4 GB Hard Disk Drives – FDDI Interface – 20 GB RAID Disk
acsad Server Workstations l Compaq Alpha Server – 4 Alpha CPUs of 440 MHz – 6 Alpha CPUs of 330 MHz – 1 GB Memory – 50 GB RAID – FDDI Network Interface – True 64 Unix Operating System
Former KEKB Control System Laboratory Network PCs Operators’ Consoles acsad 94 IOCs KEKB Accelerator Operation Beam Optics Simulation KEKB Control Network Real-time Control abco 1 KEKB Accelerator Operation EPICS Software Development Relational Database
KEKB/PF-AR Control System Laboratory Network PCs Operators’ Consoles KEKB Accelerator Operation Beam Optics Simulation acsad 104 IOCs KEKB Control Network Real-time Control abco 1 abco 2 KEKB & PF-AR Accelerator Operation EPICS Software Development Relational Database
abco 2 Server Workstation l PA-RISC 8500 Architecture – 440 MHz CPU Clock – 2 CPUs – 1 GB of Main Memory – 36 GB Hard Disk Drives – FDDI Interface – 140 GB RAID Disk
IOC Configuration From CCR Ethernet HUB FDDIーEthernet Bridge Terminal Server Lattis 810 M Cisco Catalyst 1200 Cisco 2509/2511 Ethernet RS-232 C X Terminals CPU Module(Power. Core 6750) System Monitor Module RS-232 C Module CAMAC SD(HYTEC VSD 2992) GPIB Controller(NI GPIB 1014) MXIbus Interface Module ARCNET Driver Power Supply Module VXI Main Frame Measuring Instruments Magnet Power Supplies Measuring Instruments CAMAC Crates
VME IOC System l Subrack – Schroff 16 Slots Subrack l Power Supply Module – 259 Watts, 5 V 35 A, 12 V 5 A, -12 V 2 A – 14 HP Wide, Schroff MPS 8 -7746 l System Monitor Module – Mitsubishi DRSJ-01 • Remote System Reset • DC Power Lines Monitor
VME Modules Used l CPUs – FORCE CPU-40 MC 68040 33 MHz – FORCE CPU-60 MC 68060 66 MHz – FORCE Power. Core 6603 e PPC 603 e – FORCE Power. Core 6750 PPC-750 266 MHz – FORCE Power. Core 6750 PPC-750 400 MHz l CAMAC Serial Highway Driver – HYTEC VSD 2992
VME Modules Used (contd. ) l GPIB Controller – National Instruments GPIB 1014 l ARCNET Driver – Advanet ARCNET-4 l VME-MXI Driver – Hewlett Packard VME-MXI – National Instruments VME-MXI II l MODBUS+ Interface Controller
Number of IOCs and VME Modules MG RF MO VA BT MV EN TI PH LI Tota l 96 IOC 8 8 20 12 4 20 20 2 1 1 CAMAC - 8 - 12 2 - - 2 - 1 12 6 20 20 - - - - - GPIB 8 - - RS-232 C 1 - - ARCNET 96 - - - 4 - - - MXI - - 20 - 1 - - - 10 1 2 2 5 6 6 1 3 10 0 2 1 22
Equipment Connected l 2, 517 Magnet Power Supplies – 176 ARCNET Segments l 800 Beam Position Monitors
EPICS Records on IOCs 242, 597 EPICS Records on 94 IOCs l 25, 147 EPICS Records on IOCMGD 06 l 2, 788. 5 EPICS Records in Average l l 48, 149 k. B Max. Allocated Memory 1, 324 k. B Min. Allocated Memory 12, 142 k. B Ave. Allocated Memory
Hardware Summary l Use Standard Buses – Reliability : VME, Compact PCI, CAMAC • Not ISA, EISA, PCI, Proprietary Buses – Separate CPU Bus from Field Buses • Isolation, Analog Signal Handling l Choose Components Carefully – Power Supplies – Capacitors, Fans, Connectors l Use Ergonomic Equipment
Control Consoles
TRISTAN Consoles
KEKB and PF-AR Consoles
KEKB Consoles l l Voice Generator using Power Macintosh Multiple Screen Display for Consoles – – l DELL Optiplex NX 1 Intel Pentium II 400 MHz, 1 GHz Colorgraphic Evolution 4 Macintosh with Multiple Video Cards X-terminal using IBM Network Station – Power PC 406(133 MHz) – 48 MB Memory – Disk-less / Fan-less System
Control Consoles l l 18” TFT LCDs(1280 x 1024 pixels) 16” TFT LCDs(1280 x 1024 pixels) 14” TFT LCDs(1024 x 800 pixels) Book-type Personal Computers – DELL Optiplex NX 1 and IBM Network Station – Multi-Screen Display Controllers – Wireless Keyboard/Mouse l l l Power Macintosh with 2/4 Video Controllers Six 40” Plasma Displays as TV Monitors Easy to change Configuration
Software
Software l Two Language Architecture – TRISTAN Experiences : NODAL and PCL – Interpretive Language for Applications • • SAD and python Short Turn-around Time Safe Debuging Environment Not Dedicated Application Software Programmer – Compiler Language for Low-Level Software • • C or C++ for Low-Level Software Quick Response Time Runs Fast Realizes Everything You Want
Software (contd. ) l Relational Database Software – From the Designing Stage – Put All the Accelerator Information • Cable Connection Lists • Equipment Parameters – Physical Parameters : Sizes, Weight, … – Calibration Constants, Fitting Curve Parameters, … • Histories – Installation Dates, Repair Histories, Costs, …. • Property Numbers and more … – Generates EPICS Database Automatically
Software (contd. ) l EPICS Software Toolkit – medm • Generated from Relational Database – Ah : Alarm – Ar : Archiving – SNL : State Notation Language
EPICS Software Used medm l SAD l python l
medm Applications
python Applications
SAD Applications
EPICS Databse Files
Linac Portable CA Server Linac Console Linac Main Controls KEKB CA Clients Linac CA Server Linac Sub-Controls KEKB IOCs Linac Equipment KEKB Equipment
Linac Portable Channel Access
Application Software in Use SAD l medm l python l Miscellaneous l Total l 141 74 42 6 263
EPICS Records on IOCs 242, 597 EPICS Records on 94 IOCs l 25, 147 EPICS Records on IOCMGD 06 l 2, 788. 5 EPICS Records in Average l l 48, 149 k. B Max. Allocated Memory 1, 324 k. B Min. Allocated Memory 12, 142 k. B Ave. Allocated Memory
Software Summary l l l l l EPICS R 3. 13 Toolkit on HP-UX 10. 2 medm for Man-Machine Interface X-Window Server Software on the Terminals python for Application Programming SAD for Accelerator Simulation / Operation Vx. Works on IOCs Tornado Cross Software Development Sys. CAPFAST for EPICS Runtime Database Dev. ORACLE 7 for Relational Database Portable Channel Access Server for Linac Controls
VME CPU Board Benchmarks
CPU Board Benchmark Test FORCE Pcore 6604 FORCE CPU-40 CPU Power PC 604 e MC 68040 Clock 200 MHz 25 MHz Memory 16 MB L 2 Cache 512 k. B --
CPU Load Arising from Scanning Database 1. 0 sec 0. 5 sec 0. 2 sec 0. 1 sec PCore 6604 CPU-40 0. 60 % 1. 2 % 3. 0 % 6. 1 % 5. 8 % 12 % 27 % 56 %
Transactions Time Required for Channel Access PCore 6604 ca_search 953 us ca_put 23 us ca_get 58 us ca_put&ca_get 76 us CPU-40 1, 189 us 109 us 118 us 246 us
VME-MXI Problems
VME-MXI Problem HP VME-MXI Driver Module : National Instruments Products l FORCE Power. Core 6603 or 6750 CPU Modules : Power PC l Tandra Universe or Universe II Chips : PCI-VME Bus Bridge Chips l CPU Module Halts and Never Restarts Responding to the Reset Signal without Power is switched off l
Improvements in ARCNET A HUB Box contains 3 ARCNET HUBs l 33 HUB Boxes were Installed for 97 ARCNET Lines l Decreases 102/Day/Line ARCNET “Reconfiguration” Errors down to Zero l
Configuration of the Test Bench VME-MXI module System Monitor module CPU module FORCE Power. Core 6750: PPC 750, 266 MHz, Univ. II FORCE Power. Core 6750: PPC 750, 400 MHz, Univ. IIB MPX module FORCE Power. Core 6603: PPC 603, Universe Chip RF Voltmeter FORCE CPU 64: MC 68060, 66 MHz MPX module FORCE CPU 40: MC 68040, 33 MHz VXI-MXI Controller CPU Modules tested
Normal and Abnormal Bus Cycles Normal Bus Cycle Abnormal Bus Cycle
Abnormal Bus Cycle
Normal and Abnormal DTACK* Signals Power. Core 6603, with Universe Chip Normal Abnormal
Abnormal DTACK* Signal Power. Core 6750, 266 MHz, with Universe II Normal Abnormal
Realtime Linux
Linux l Disk I/O causes 20 - 30 ms of Latency, possibly Latency can be around 100 ms or more. l Cause: – Non-preemptive Kernel – Scheduling Algorithm – Interrupt Disabling
Non-Preemptive Kernel Interrupt Kernel High Priority Low Priority Latency
Preemptive Kernel Interrupt Kernel High Priority Low Priority
EPICS ioc. Core Network Channel Access Run-time Database Device Access Layer I/O Buses (VME/PCI/ISA)
EPICS under L 4 -Linux Server X MEDM ioc. Core L 4 real-time micro-kernel
Linux System Calls Interrupt Linux Kernel L 4 Kernel ( IPC ) Linux Server Process Standard Linux Process L 4 -Linux
RT-thread Preempts Linux Interrupt L 4 Kernel RT- thread Linux Server Process
RT-thread Calls Linux Interrupt L 4 Kernel RT- thread Linux Server Process
What Happens if RT-thread Calls Preempted Linux? Interrupt L 4 Kernel RT- thread Linux Server Process Latency
- Slides: 75