Hardware Platforms for ESS ICS Timo Korhonen ICS

Hardware Platforms for ESS ICS Timo Korhonen ICS www. europeanspallationsource. se Nov. . 5, 2014

Overview • • Scope of the hardware platform decision(s) Goals Development strategies Platforms • High-speed digital platform • Middle range I/O • Industrial I/O • Conclusions 2

Scope of the decision(s) • Control systems have a lot of interfaces • Need to cater for a spectrum of I/O requirements • Fast, real-time signal processing • State-of-the-art technology, evolving fast • FPGA-based processing • MHz to GHz range of signal acquisition • Middle-range I/O • requires synchronization, k. Hz range I/O • Non-real time industrial I/O • Typically PLC-based • Off-the-shelf devices • Serial, Ethernet or other fieldbus devices • No single platform can cover this cost-efficiently! 3

Goals • Sufficient performance for the task at hand • Data processing & transport capabilities • Time synchronization • Cost-efficiency • Looking at the total cost, not only component cost • Software (and firmware) dominates cost • Roadmap for evolution • Beware of obsoleteness traps • Project timespan – prepare for change • Efficiency of development • ICS provides a platform for system development • Division of work – development and integration 4

Development strategies • Use standardized equipment whenever possible • Select platforms to support and promote them (via a harmonization committee: E 2 H 2 C) • Benefit from experience in the community • Prefer standards that are used in the community • Benefit from collaborations • Open sharing of work – direct peer-to-peer • Use proven solutions whenever possible • Sometimes new solutions are required, though • Use modular systems • Parts can be replaced without redoing the whole system • Interfaces need to be clearly defined 5

Platforms • It is not reasonable to do everything with a single platform • In the past, VME and CAMAC have been such ones • There will be many industrial-type systems • We need to be flexible (one size does not fit all) • High-end, digital processing platform • For beam instrumentation and LLRF applications • Acquisition in MHz and above range • Response times in microseconds • Middle range I/O • Often forgotten but important • Not highest speed but real-time capable and flexible • Industrial I/O • PLC is the standard way in industry 6

High-speed Digital Platform • Performance requirements (very shortly) • Support 14 Hz operation • Good integration with timing • Acquisition (ADC sampling) at 88 MSPS or faster • 2. 86 ms long pulses: a lot of data to be handled • Local pre-processing of data • Interfaces to machine protection/interlock • Post-mortem abilities • In fact, any (packaging/bus) platform could provide this! • The real questions are elsewhere: • Maintainability • Development and long-term sustainability • Collaborations and support for IKC partners 7

High-speed Digital Platform – the box • Today, there is no clear single choice • c. PCI, VME, custom (aka “pizza”) box, MTCA. 4, . . . ? • Serial links have overtaken parallel buses • Intelligence is in the interfaces and protocols • The “box” supplies infrastructure (power, cooling, monitoring, connectivity, some convenience features) • Take a proven solution or go for something new? • Several labs are looking at MTCA. 4 • DESY (main promoter), SLAC, many others • Packaging standard supports a modular approach • Different components can be mixed (interoperability!) • Serviceability, component upgrades, etc. • Potentially a long lifecycle (if enough deployment) 8

High-speed Digital Platform • What to take, then? • Decide on one (until now, c. PCI and MTCA) • (quite) some work has been done on MTCA • Using a single card (SIS 8300) + RTMs • MTCA. 4 native timing receiver in preparation • Seems prudent to continue on the MTCA path • Decided on a platform – fine. What then? • Platform alone does not provide any functionality • Systems and processes have to be built around • Here lies the major investment! • We have some components developed but • Deployment will bring up many issues • Systems need to be maintained for a long time 9

High-speed Digital Platform – and beyond • Lifetime issues • ESS project timespan > 10 years • Obsolecense issues will arise during the project • Need to be ready for change! • Biggest cost factor is in integration • Software, firmware, maintenance processes • How could that be modularized? • Development of a full lifecycle system is costly • Has somebody addressed the deployment issues? • How to do gradual upgrades? • How to secure maintainability? • Deciding the packaging standard is not enough • A more comprehensive framework is needed 10

High-speed Digital Platform – blocks • Processing in FPGA is the standard way today • Monolithic designs have been common • Reuse of components is hard • Needs special skills • FMC (FPGA Mezzanine Card), VITA-57 standard addresses these issues • Separate analog (application-specific) and digital parts with a defined interface • One digital board can be used for several purposes • Reduce hardware stock complexity • Re-use of firmware libraries, software, integration • Modularization at the level of the major investment • Firmware, software, operational procedures 11 • Separation of work (defined interfaces)

High-speed Digital Platform - framework • Set up a collaboration with PSI (ESS partner lab) • In the last 3 years, PSI invested a lot into development of a framework that addresses the issues of: • Performance and flexibility • Maintainability and integration • Lifecycle management and modularity • Done with industry (small company) collaboration • We could benefit from that • Even if the form factor is different – majority of effort is in the infrastructure for applications • Over 10 person-years worth of work • Used in production (PSI LLRF, diagnostics, etc. ) • Interest also from other labs (Daresbury, DESY. . ) 12

Framework proposal - development • Port the existing framework to MTCA. 4 hardware • Hardware implementation by the company IOx. OS SA • Reuse existing designs, with upgrades • Rapid hardware development • Soft- and firmware can be retained close to 100% • Open collaboration with PSI and IOx. OS • Full documentation and source code sharing • Most tricky problems have already been solved • Interrupts, DMA handling, etc • Collaboration in the first line, can develop into IKC • In the mid-term, port previous work to this platform • All previous work can be used in the short term • Minimize distruptive sudden changes 13

Middle-range I/O • High-end platform is expensive and centralized • Use only where needed • Some applications still need time synchronization • Especially in a pulsed machine! • PLCs are not ideal for this (asynchronous cycle times) • Ether. CAT is a good solution for this range • Uses Ethernet cabling, special protocol • Can run on a PC or a “hard” IOC • Loop times of several k. Hz possible • Distributed I/O (reduce cabling) • Cost effective • Many I/O modules, several manufacturers • Gaining popularity in the EPICS community 14

Industrial I/O • PLC is the standard choice for most industrial-type I/O • High reliability • Widely known and deployed in partner labs and industries • A huge selection of I/O modules available • Ideally, select one manufacturer • Reduce development and maintenance costs • One skillset, reduced stock of spares • Select a manufacturer with domain expertise • Local support for IKC partners • Procure via an open call for tender • Not quite straightforward, though – but in progress 15

“Other” I/O • There is still a large number of devices not covered yet • Serial devices • Use MOXA – seems to be almost de-facto standard • Ethernet-based devices • Ethernet as a fieldbus, use Stream. Device & ASYN • Cameras • Gig. E, 10 GBE, Camera. Link(HS) • Very much provider-dependent – take what the company gives • EPICS Area. Detector supports a large device base • Motion control • Working together with our NSS colleagues 16

Summary • A lot of discussion revolves around the high-end platform • But that is only a part of the story • work-intensive and critical part, but still • Other areas tend to get too little attention • try to set up a comprehensive set of solutions • We have selected MTCA. 4 for the high-end • Knowing that there will be issues to solve • But they can be solved • We try to establish and leverage collaborations • Mutual help between institutes • Other areas are equally, if not even more urgent • Need to keep going there, too • Start prototyping (in our new lab) 17

Questions? 18

Reserve slides 19

System interfaces (PSI board for IS&LEBT) Ethernet CPU (EPICS) PCIe switch FPGA Event Receiver • PCI express for internal (and external) communication • Timing (Event Receiver) onboard via PCI • EPICS running locally • Ethernet to the outside world FMC (signal interfaces) Timing distribution 20

PSI board: a picture tells more than 1 k. Words CPU Power. PC P 2020 • 1. 2 GHz dual core • runs real-time Linux / EPICS • Boots over LAN, u. SD card, or on-board FLASH FPGA Virtex-6 LX 130 T • TOSCA-II PCI-express Network on-chip IP • connects: 512 MB shared memory (DDR 3) • connetcs: VME, user logic, FMC sites, VME_P 2 • User FPGA code PCI-express GEN 2 switch • central interconnect between CPU / FPGA / XMC / VME_P 0 • contains Non-Transparent (NT) function 21