General FPGA to EPICS IOC Communication Protocol Yuke

General FPGA to EPICS IOC Communication Protocol Yuke Tian, Kiman Ha, Joseph Mead Brookhaven National Lab EPICS Collaboration Meeting, Hsinchu, Taiwan, June, 2011

Outline • • • Common requirements for FPGA-based equipments Resources • FPGA • EPICS General IOC-FPGA communication protocol • IOC side • FPGA side Test results on PSC Expected through put for BPM/Cell controller EPICS Collaboration Meeting, Hsinchu, Taiwan, June, 2011

Common requirements for FPGA-based equipments • Why do we need FPGA-based equipments: FPGA vs ASIC ? FPGA Design Advantages ASIC Design Advantages Faster time-to-market - no layout, Full custom capability - for design masks or since other manufacturing steps are needed device is manufactured to design specs No upfront NRE (non recurring Lower unit costs - for very high expenses) volume costs typically associated with an ASIC designs design Simpler design cycle - due to software that Smaller form factor - since device is handles much of the routing, manufactured to design specs placement, and timing More predictable project cycle - due to Higher raw internal clock speeds elimination of potential re-spins, wafer capacities, etc. Field reprogramability - a new bitstream can be uploaded remotely For experimental physics equipment, FPGA is a better choices. EPICS Collaboration Meeting, Hsinchu, Taiwan, June, 2011

Common requirements for FPGA-based equipments FPGA Logic Control (V 6: up to 100 K slices) DSP (V 6: up to 800 DSP 48) High speed serial link Embedded CPU (V 6: up to 72 GTX /GTH transceiver, PCIe) (microblaze: 32 RISC 1 K slice) 0. 5 Tbps External logic (ADC, DAC, etc) Other system Analog (V 7: 1 Mbps ADC) Standard CPU peripheral (Ethernet, RS 232 etc) External world FPGA for So. C EPICS Collaboration Meeting, Hsinchu, Taiwan, June, 2011 Analog signal

Common requirements for FPGA-based equipments • Why do we use embedded CPU in FPGA (instead of a separate CPU outside FPAG) ? The embedded CPU and the user defined logic are on the same FPGA chip. They have fast bus interface. All user registers and block memory are just part of the CPU memory space. CPU can easily access them without any other logic control. The embedded CPU already has many IO peripheral that are ready to communicate with the outside world, such as EPICS IOC. At NSLS-II, all our custom designed equipments (BPM, cell controller, power supply controller) use the same Xilinx soft-core CPU (microblaze). We need to find a general way for the CPU to communicate with EPICS IOC. The nature choice is through Ethernet port. Since this is an embedded light weight CPU system, we need to find some Collaboration Meeting, Hsinchu, Taiwan, June, 2011 main task is to protocol simple and. EPICS reliable. In fact, the embedded CPU’s

Common requirements for FPGA-based equipments CA Client (Physics applications) Channel Access EPICS IOC Small Size Data Bulk. Tx Data Bulk. Rx Data (ai, ao, di, do etc) (waveform, etc) BPM: gain, calbration BPM: TBT, ADC raw BPM: filter coefficients CC: matrix selection 10 KHz orbit data CC: reverse response matrix TCP/IP FPGA (micro. Blaze / PSC: Xilkernel / LWIP PSC: setpoints, commands Booster ramping function TCP/IP) ADC readbacks registe rs SRA M MPM C CC: PSC: DDR 2/DD R 3 How do we design a simple/reliable protocol to transfer data between IOC /FPGA quickly ? EPICS Collaboration Meeting, Hsinchu, Taiwan, June, 2011

Resources on FPGA Embedded softcore CPU micro. Blaze Speed Cache Memory BPM/CC (Virtex 6) 125 MHz Available (32 K/32 K) A lot. (total DDR 3: 1 GB, 64 bit wide) PSC (Spartan 3 A) 50 MHz Available (2 K/2 K) A lot. (total DDR 2: 256 MB, 8 bit wide) Network resource EMAC PHY BPM/CC 1 Gbps hardcore TEMAC 1 Gbps PHY PSC 100 Mbps soft EMAC 100 Mbps PHY EPICS Collaboration Meeting, Hsinchu, Taiwan, June, 2011

Resources on FPGA Xilkernel: Xilinx kernel supports multithreads, scheduling, semaphore, message queue, buffer memory. EPICS Collaboration Meeting, Hsinchu, Taiwan, June, 2011

Resources on FPGA LWIP TCP/IP stack: Free TCP/IP stack from LWIP community. It support multiple TCP connections, DHCPetc. TCP window size, TX/RX checksum offload, jumbo frame, ARP, EPICS Collaboration Meeting, Hsinchu, Taiwan, June, 2011

Separate three data traffic into three sockets CA Client (Physics applications) Channel Access EPICS IOC Small Data Bulk. Tx Data Bulk. Rx Data (ai, ao, di, do etc) TCP/IP port 1 (waveform, sub. Arrary etc) TCP/IP port 3 TCP/IP port 2 FPGA (micro. Blaze / Xilkernel / LWIP TCP/IP) registe rs SRA M MPM C DDR 2/DD R 3 How do we design a simple/reliable protocol to transfer data between IOC /FPGA quickly ? EPICS Collaboration Meeting, Hsinchu, Taiwan, June, 2011

Resources from EPICS There are many device drivers to support TCP/IP communication between IOC and network equipments. Once commonly used is asyn. Driver. TCP/IP driver EPICS Collaboration Meeting, Hsinchu, Taiwan, June, 2011

General IOC-FPGA communication protocol: IOC side EPICS IOC Small Data Bulk. Rx Data (ai, ao, di, do etc) (waveform, a. Sub etc) Bulk. Tx (waveform, a. Sub etc) Out: Combined small data into one MTU by using a. Sub. In: parse one MTU into small data. record/asyn. Int 8 Array. In 1 MT U U (10 H z) ID waveform record/asyn. Int 8 Array. Out Large frame (optiona l) amount data ID frame waveform Large amount data asyn port 1: asyn port 2: asyn port 3: drv. Asyn. IPPort. Configure (“Normal. Rx. Tx", "192. 168. 1. 10: 7 TCP", 0, 0, 0) (“Bulk. Rx", "192. 168. 1. 10: 18 TCP", 0, 0, 0) (“Bulk. Tx", "192. 168. 1. 10: 20 TCP", 0, 0, 0) EPICS Collaboration Meeting, Hsinchu, Taiwan, June, 2011

General IOC-FPGA communication protocol: FPGA side asyn port 1: asyn port 2: asyn port 3: drv. Asyn. IPPort. Configure (“Normal. Rx. Tx", "192. 168. 1. 10: 7 TCP", 0, 0, 0) (“Bulk. Rx", "192. 168. 1. 10: 5001 TCP", 0, 0, 0) (“Bulk. Tx", "192. 168. 1. 10: 5000 TCP", 0, 0, ID 1 MT U frame (10 H z) (optiona l) Large amount data ID frame Large amount data 1. Create socket for "192. 168. 1. 10: 7” port 1. Create socket for "192. 168. 1. 10: 18” port 1. Create socket for "192. 168. 1. 10: 20” port 2. Listen to the socket 3. If new packet coming, create a thread to process it. 4. In process thread, keep writing from CPU mem of to socket. 4. In process thread, read data, copy it, 4. In process thread, keep reading data outspace socket and copying it to CPU memory space. write response data to socket. Total C codes running on FPGA should be simple socket programming. Since we develop it from scratch, we clearly understand it and we have full control of the kernel and each thread. EPICS Collaboration Meeting, Hsinchu, Taiwan, June, 2011

General IOC-FPGA communication protocol: FPGA side Psudo codes: Init_platform(); //start xilinx platform xilkernel_init(); //init Xil_kernel xmk_add_static_thread(main_thread, 1); //threw main thread xilkernel_start(); //start Xil_kernel Main_thread(): lwip_init(); //initialize lwip TCP/IP stack sys_thread_new("NW_THREAD", network_thread, NULL, THREAD_STACKSIZE, DEFAULT_THREAD_PRIO); //create return 0; Network_thread(): IP 4_ADDR(&ipaddr, 192, 168, 1, 10); IP 4_ADDR(&netmask, 255, 0); IP 4_ADDR(&gw, 192, 168, 1, 1); //set IP, netmask, gateway xemac_add(netlist, . . ); netif_set_default(netif); netif_set_up(netif); //add network to netlist, set it as default, setup netlist sys_thread_new("xemacif_input_thread", (void(*)(void*))xemacif_input_thread, netif, THREAD_STACKSIZE, DEFAULT_THREAD_PRI sys_thread_new(“normal. Rx. Tx", normal. Rx. Tx_thread, 0, THREAD_STACKSIZE, DEFAULT_THREAD_PRIO); sys_thread_new(“Bulk. Rx", rx_thread, 0, THREAD_STACKSIZE, DEFAULT_THREAD_PRIO); sys_thread_new(“Bulk. Tx", tx_thread, 0, THREAD_STACKSIZE, DEFAULT_THREAD_PRIO); return 0; EPICS Collaboration Meeting, Hsinchu, Taiwan, June, 2011

General IOC-FPGA communication protocol: FPGA side Psudo codes: normal. Rx. Tx_thread(): lwip_socket(AF_INET, SOCK_STREAM, 0); //create socket lwip_bind(sock, (struct sockaddr *)&address, sizeof (address)) < 0); //assign socket address/port to socket lwip_listen(sock, 5); //willing to listen and queue length if ((new_sd = lwip_accept(sock, (struct sockaddr *)&remote, (socklen_t *)&size)) > 0) a thread to process it //if new connection established, create sys_thread_new("echo_server", process_normal. Rx. Tx_request, (void*) new_sd, THREAD_STACKSIZE, DEFAULT_THREAD_PRIO); Process_normal. Rx. Tx_thread(): Lwip_read(sd, rx. Tx. Buffer, RECV_BUF_SIZE); //read the received data //depending the ID frame, we copy the received data into different location. This is FPGA memory-map dependent switch (rx. Tx. Buffer[0] {case 1: memcpy(add. RX, rx. Tx. Buffer, read. Number); case 2: …}; write. Number = lwip_write(sd, rx. Tx. Buffer, read. Number); //write back to socket Bulk. Rx_thread is similar to normal. Rx. Tx_thread except: -- It binds to different port; -- In the process thread, it keep read the data from the socket using lwip_read, then copy to some CPU memory space. Bulk. Tx_thread is similar to normal. Rx. Tx_thread except: -- It binds to different port; -- In the process thread, it keep copy data from CPU memory space , and write the data to socket using lwip_write. EPICS Collaboration Meeting, Hsinchu, Taiwan, June, 2011

Test results on PSC Only one port (Bulk. Tx) is connected. Throughput (PSC to IOC): 10*1400*8 bits/0. 2 ms = 5. 6 Mbps EPICS Collaboration Meeting, Hsinchu, Taiwan, June, 2011

Test results on PSC Last packet from PSC. EPICS Collaboration Meeting, Hsinchu, Taiwan, June, 2011

Test results on PSC In psc_test. cmd file: drv. Asyn. IPPort. Configure ("psc. Normal. Rx. Tx", "192. 168. 1. 10: 7 TCP", 0, 0, 0) drv. Asyn. IPPort. Configure ("psc. Rx", "192. 168. 1. 10: 18 TCP", 0, 0, 0) drv. Asyn. IPPort. Configure ("psc. Tx", "192. 168. 1. 10: 20 TCP", 0, 0, 0) Bulk. Tx port (FPGA to IOC) normal. Rx. Tx port Bulk. Rx port (IOC to FPGA) All three ports are connected. EPICS Collaboration Meeting, Hsinchu, Taiwan, June, 2011

Test results on PSC All three ports are connected. EPICS Collaboration Meeting, Hsinchu, Taiwan, June, 2011

Expected throughput for BPM/Cell Controller With cache turned on, use jumbo frame, and TX/RX checksum offload, BPM can get TX/RX at 80 Mbps throughput with lwip socket mode. This is tested with iperf Our new protocol is similar to iperf. We hope to get this throughput between IOC and BPM/Cell controller. If so, that means: We might be able to get turn-by-turn data (x/y position) in real time (not on demond anymore). TBT data: 378 KHz * 4 byte * 2 (for x, y position) * 8 = 24. 2 Mbps On IOC side, 24. 2 Mbps * 8 BPM/cell = 193. 6 Mbps. Our IOC has true Gig. E connection. For cell controller, we can get 10 KHz data in real time from several cell controllers 10 KHz orbit data: 10 KHz * 4 byte * 2 (for x, y position) * 240 * 8 = 153 Mbps We need to use a few (for example 5) cell controllers (each one output 20% of the data) to get the 10 KHz orbit data. Higher throughput to EPICS is always good for physics applications. More applications (LLRF, etc) can be done through this new protocol. Suggestions from both EPICS and FPGA experts are welcome. EPICS Collaboration Meeting, Hsinchu, Taiwan, June, 2011

Summary FPGA is a common choice for accelerator and experimental physics to carry out control system hardware design. To use embedded CPU in FPGA will simplify the data delivery to EPICS IOC from the hardware level registers (such as ADC, DAC data, or memory data, etc). Using TCP/IP socket programming protocol provides a simple and reliable data communication between FPGA embedded CPU and EPICS IOC. On the FPGA side, the protocol is easy to implement. asyn. Driver provides a perfect solution. On the EPIOC side, EPICS Collaboration Meeting, Hsinchu, Taiwan, June, 2011