Data Backend System The data backend system receives

Data Backend System? The data backend system - receives the detector’s data - aggregates raw detector data to images or frames - passes the images on downstream - buffers the detector data to decouple the downstream receivers from the upstream detector X-Ray Detectors for Synchrotron Applications. 5 th-7 th July Zurich and Villigen Data backend system for fast multi megapixel detectors. Heiner Billich, Seite 1

Requirements for data backend systems Keep up with the data stream from the detector, the data backend must never limit the detector. Allow to scale out as the detector scales out. Don’t drop any data, especially no UDP datagrams. X-Ray Detectors for Synchrotron Applications. 5 th-7 th July Zurich and Villigen Data backend system for fast multi megapixel detectors. Heiner Billich, Seite 2

Outline Detectors – Eiger 9 M and Giga-FROST Approach, Strategy Hardware setup and scale-out Will it work? Some numbers UDP datagramm losses and how to prevent them Message passing Storage Future (hardware) options X-Ray Detectors for Synchrotron Applications. 5 th-7 th July Zurich and Villigen Data backend system for fast multi megapixel detectors. Heiner Billich, Seite 3

Giga-FROST – Real-Time Tomography single quadrant • 2 x 10 Gb. E whole camera • 4 quadrants • 8 x 10 Gb. E • 80 Gbit/s max. throughput • sends UDP datagrams, 6 k. B data payload each • board can saturate 10 Gb. E links See Ultrafast Data Processing Pipeline for Real-Time Tomographic Imaging at TOMCAT. Federica Marone, SRI 2012 Detector requirements for synchrotron-based X-ray tomographic microscopy: A customized high-speed data interface for subsecond temporal resolution at TOMCAT Federica Marone at 09: 00 today X-Ray Detectors for Synchrotron Applications. 5 th-7 th July Zurich and Villigen Data backend system for fast multi megapixel detectors. Heiner Billich, Seite 4

Eiger 9 M half module • 256 x 1024 pixels • 1 x 10 Gb. E • 1 x 4 sensor chips Full detector = 18 modules • 36 x half module • 9 M pixels • 36 x 10 Gb. E • 360 Gbit/s max. throughput module = 2 x half module • 2 x 10 Gb. E • 2 x 4 sensor chips sends UDP datagrams, 4 k. B payload • can saturate 10 Gb. E • See Eiger, a fast framing large area pixel detector for X-ray applications Ian Johnson, Friday 09: 50 X-Ray Detectors for Synchrotron Applications. 5 th-7 th July Zurich and Villigen Data backend system for fast multi megapixel detectors. Heiner Billich, Seite 5

Approach, Strategy No special hardware but up-to-date components as used to build compute clusters. pattern Stay parallel to allow to scale-out Attach sufficient header information Overprovision bandwidth everywhere to all messages to mark them to allow to catch up if someone falls uniquely. behind. Use scripting as glue (python) Current hardware and adequate Be prepared to handle losses and libraries provide the required out-of-order datagrams. performance and throughput (and Don’t mix controls – Epics, Tango, … most times not our own code). d - and the data backend: don’t talk to Save to memory and such buffer the detector … detector data as early as possible. Aggregate the tiny UDP datagrams to larger messages as early as possible. Use message passing, a data-flow X-Ray Detectors for Synchrotron Applications. 5 th-7 th July Zurich and Villigen Data backend system for fast multi megapixel detectors. Heiner Billich, Seite 6

Hardware setup (simplified) Storage X-Ray Detectors for Synchrotron Applications. 5 th-7 th July Zurich and Villigen Data backend system for fast multi megapixel detectors. Heiner Billich, Seite 7

Hardware setup – scale-out Storage X-Ray Detectors for Synchrotron Applications. 5 th-7 th July Zurich and Villigen Data backend system for fast multi megapixel detectors. Heiner Billich, Seite 8

Will it work? – some numbers Eiger half module – single 10 Gb. E link to backend 256*1024 pixels, assume 16 bit@1000 Hz 500 Mi. B/s 128’ 000 UDP datagram/s ✔ ? ? bandwidth on current x 86 hardware 35 GB/s memory total 3. 2 GB/s PCIe 2. 0 8 x 3. 2 GB/s QDR Infini. Band 1. 1 GB/s 10 Gb. E X-Ray Detectors for Synchrotron Applications. 5 th-7 th July Zurich and Villigen ✔ ✔ Data backend system for fast multi megapixel detectors. Heiner Billich, Seite 9

UDP datagram losses UDP is an unreliable service – out of order, duplicated or missing datagrams are allowed. Where do we miss datagrams? 1. On the network card or operating system level 2. In the switch, if we use one 3. In between: bad cables or connectors X-Ray Detectors for Synchrotron Applications. 5 th-7 th July Zurich and Villigen Data backend system for fast multi megapixel detectors. Heiner Billich, Seite 10

UDP datagram losses - tests Single server, 2 x 10 Gb. E, direct attached receiving from FPGA based UDP test generator 29 Ti. B in batches of 290 Gi. B and 38 M datagrams, 8 k. B payload ~ 4*109 datagrams ~ 2’ 300 Mi. B/s ~ 300’ 000 datagram/s no datagrams lost 12 Ti. B in batches of 128 Gi. B and 38 M datagrams, 3. 5 k. B payload ~ 3. 7*109 datagrams ~ 2’ 300 Mi. B/s ~ 660’ 000 datagram/s (!) no datagrams lost X-Ray Detectors for Synchrotron Applications. 5 th-7 th July Zurich and Villigen Data backend system for fast multi megapixel detectors. Heiner Billich, Seite 11

UDP datagram losses – how to prevent Employ standard tuning as described in 10 Gb Ethernet Mark Wagner Red Hat Summit, Boston, 2008 Tuning 10 Gb network cards on Linux. A basic introduction to concepts used to tune fast network cards Breno Henrique Leitao, IBM Proceedings of the Linux Symposium July 13 th– 17 th, 2009 Montreal, Quebec Canada and use several receiving UDP sockets per network port as bucket chain. Typically all datagrams of single image and sender go to the same socket. X-Ray Detectors for Synchrotron Applications. 5 th-7 th July Zurich and Villigen Data backend system for fast multi megapixel detectors. Heiner Billich, Seite 12

UDP datagram losses – Ethernet switches How do we avoid oversubscription? - By static one-to-one assignments of receiving and sending ports (Giga-FROST, just 8 lines) - By tokens circulating among the detector modules: Allow just a single detector port to send to a specific receiving port at any time (Eiger 9 M, 36 lines) I plan to do first tests with switches later this year. X-Ray Detectors for Synchrotron Applications. 5 th-7 th July Zurich and Villigen Data backend system for fast multi megapixel detectors. Heiner Billich, Seite 13

Message passing MPI – tight coupling inside the backend system Zero. MQ – loose coupling to the downstream receivers Focus on throughput, not message rate – use large messages. X-Ray Detectors for Synchrotron Applications. 5 th-7 th July Zurich and Villigen Data backend system for fast multi megapixel detectors. Heiner Billich, Seite 14

Storage Giga-FROST No compression available, need to write at 8 GB/s to disk. Initially I’ll buffer whole datasets to RAM. Later we may get more disks, about 450 disks at 7’ 200 rpm to get 8 GB/s sustained write throughput. Eiger Compression may reduce the data rate to disks by factors. Dectris and PSI work with The HDF Group to allow parallel high throughput compression with HDF 5. X-Ray Detectors for Synchrotron Applications. 5 th-7 th July Zurich and Villigen Data backend system for fast multi megapixel detectors. Heiner Billich, Seite 15

Future options -Intel Xeon E 5 cpus (Sandy Bridge) -PCIe Gen 3 -FDR Infiniband 56 Gbit/s -40 Gb. E host cards and switches -40 Gb. E over single mode fiber -Break-out copper or fiber cables to merge 4 x 10 Gb. E into 1 x 40 Gb. E on switch X-Ray Detectors for Synchrotron Applications. 5 th-7 th July Zurich and Villigen Data backend system for fast multi megapixel detectors. Heiner Billich, Seite 16

Thank You! X-Ray Detectors for Synchrotron Applications. 5 th-7 th July Zurich and Villigen Data backend system for fast multi megapixel detectors. Heiner Billich, Seite 17