Contents Glance over SACLAXFEL Overview of DAQ system

Contents • • • Glance over SACLA/XFEL Overview of DAQ system DAQ Front-end Future perspective Summary 次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28

DAQ for Large Image Sensors : >5 Gbps >5 Gbps in total CPU board “K Supercomputer” 10 Pflops, 2012. 4~ Custom-made VME Camera Link Board Camera FPG Camera Link Camera AFPG デー Camera. Linkカード Link Camera AFPG デー Link カード Camera AFPG デー > 2 Gbps/connectionタ Link カード Camera AFPG タ Camera デー 圧縮 Link Pr. PMC カード AFPG タ デー 圧縮 Link Pr. PMC カード A Grabber タ デー 圧縮 Pr. PMC カード タ 圧縮 Pr. PMC 圧縮 Pr. PMC Linux FPGA Lossless Compression CCD array 4 M pixels (8 sensor modules) 60 Hz control 10 Gbps Ethernet Reduced data for on-line monitor High Speed Storage Data-handling Servers 次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28 PC Cluster Preprocess 10 Tflops

SACLA : XFEL facility at SPring-8 (SPring-8 Angstrom Compact free electron LAser) SPring-8 ~700 m SACLA Experimental Hall e- • Began operation in February 2011. – Aiming for < 1Å of wavelength of FEL by 8 -Ge. V linac. • First lasing achieved on June 2011 at 1. 2Å. • User experiment will begin on April 2012. 次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28

Courtesy of M. Yabashi Anticipating a lot of Type of Experiments Generation of extreme state High peak brilliance x 109 Atomic-level imaging for noncrystalline objects XFEL phage flagella, cilia Spatial coherence 100% Probing of ultrafast chemical reaction & phase transition Ultrafast pulse < 100 fs 5 次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28

Coherent X-ray Diffraction e. g. Coherent X-ray Diffraction Imaging Differential Cross Section A lot of images must be taken Structure Factorby a large pixel sensor to reconstruct the r (r) : Electron Density Distribution original structure. Sample X t en r e oh C Courtesy of Y. Nishino s y -ra Phase Retrieval Sample Image Fourier Transform Reconstruction without the aid of Lenses Unstained Human Chromosome 次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28

Overview of DAQ system

Requirements for Data Acquisition(DAQ) • Shot-by-shot data acquisition in synchronization with the beam operation cycle. – In order to correlate the beam characteristics with the data. – 60 Hz at present, higher for future. • Accumulate data of large image sensors into storage without data loss. ⇒ >5 Gbps of data rate with 6 M pixels at 60 Hz. – Data compression to reduce the bandwidth and storage size. • On-line data-quality monitor for efficient data collection. • Any type of users instrument specific to their experiment can be attached to our DAQ to work with. – Like a Plug-and-play device. 次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28

Schematic of DAQ • Network distributed system. • MADOCA as the control framework. Front-end Electronics Front-end Computer Image Sensors CCD cameras VME : Solaris PC : Linux On-line Database DAQ Control My. SQL GUI Data Buffering Beam line Instruments Monitors Users Experimental Instruments Detectors VME : Solaris PC : Linux Data Buffering Windows Lab. VIEW TCP socket client vi 京(K) Super Computer High Speed Storage 1 G/10 G Ethernet GUI Accelerator Status Database On-line Data Monitor Sybase 次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28 PC farm for Analysis

Schematic of DAQ • Data transfer via TCP/IP over 1 - and 10 -Gbps Ethernet. • FIFO buffering for non-real-time components. Front-end Electronics Front-end Computer Image Sensors CCD cameras VME : Solaris PC : Linux Data Buffering Beam line Instruments Monitors Users Experimental Instruments Detectors VME : Solaris PC : Linux Data Buffering Windows Lab. VIEW TCP socket client vi On-line Database DAQ Control My. SQL 京(K) Super Computer GUI TCP/IP High Speed Storage 1 G/10 G Ethernet GUI Accelerator Status Database On-line Data Monitor Sybase 次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28 PC farm for Analysis

Event Synchronization: Tag Number • To reconstruct the data set in the same beam shot, a “tag number” is recorded in every data to identify which one is related to which beam shot. – Sequential number of master trigger pulse. – Counted by the trigger-counter module at each station. • Delivered to users instruments as well. – Encoded serial pulse-train, parallel bit-pattern. Master Trigger 1, 2, 3, . . Trigger Counter Gun 60 Hz Pulse 1, 2, 3, . . Trigger Counter Accelerator Components 1, 2, 3, . . Trigger Counter Monitor System 1, 2, 3, . . Trigger Counter 1, 2, 3, . . Trigger Counter Beamline Experimental Users Components Detectors Instruments 次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28

DAQ Front-end

DAQ Front-end Trigger tag counter # of pixel Depth Repetition rate Total bit rate 500 k pixel 16 bit 60 Hz 480 Mbps On-the-fly lossless compression Safety margin < 300 Mbps Cameralink (base configuration) < 2. 04 Gbps DAQ front-end Gigabit Ethernet << 1 Gbps - Protocol overhead - Congestion control Interface with CCD camera and transfer the data to storage system 次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28

Implementation of DAQ front-end • general purpose FPGA board with custom user logic executable. FPGA: VIRTEX-5 (XC 5 VFX 70 T) • CPU board with Gb. E I/F CPU: Power. QUICC III • Cameralink board. Designed for a RF-beam monitor system and an x-ray diamond monitor system of SPring-8 次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28

Implementation Cameralink CPU board Bandwidth Limit: 760 Mbps Repetition limit is ~90 Hz 次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28

High Speed Storage : DDN + Stor. Next • Parallel-writing to a single file system – Achieve over 5 Gbps in total. – Convenient to handle data files in a centralized repository. 1 G/10 G Ethernet SAN >100 Gbps Data Direct Network S 2 A 9900 Quantum Stor. Next file system Camera. Link Data of CCD sensor #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 additional #11 sensor #12 Data file servers ~180 TB >GB/sec Single file system > 5 Gbps in total 次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28

DAQ System Worked as Expected • Intensively used during beam tuning of SACLA to view the beam profile. • Ready for user experiments. June 7, 2011 First lasing at SACLA Beam energy: 7 Ge. V Wavelength: 1. 2Å MPCCD image 1 mm 次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28

Future perspective

Upgrade plan In near future: Requirement for next gen. DAQ system • Image sensor: 6 M ⇒ 76 M pixels • Beam repetition: 60 Hz ⇒ 120 Hz ⇒ 300 Hz ⇒ Data rate ~ x 26 : ~150 Gbps ! ~ x 65 : ~380 Gbps ! Target specification of DAQ front-end Number of Pixel Depth Repitation Total bit rate 1. 92 M 24 bit+metadata 60 Hz 2. 77 Gbps※ 1 st phase Number of Pixel 1. 92 M Depth 24 bit+metadata Repitation 300 Hz Total bit rate 13. 85 Gbps※ Final target 次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28

Latest action and future plan - Features of several VISION technologies - Max speed / single Cameralink HS Coa. XPress 10 Gig. E Vision 2. 04 Gbps(Base) 3. 125 Gbps 6. 25 Gbps 10 Gbps Max speed / several 5. 44 Gbps(Full) 3. 125 Gbps x N 6. 25 Gbps x N 40 Gbps (4 lanes) Max length 7~10 m 15 m 40~130 m 37~100 m Connector/Cable SDR CX 4 BNC/coaxial RJ 45/Cat 6 A- Trigger Yes Yes ? Optical option Yes not yet ? Yes done not yet Standardization Because progress of vision technology is very fast, we can wait until a market is growing up and specification will be settled down. 次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28

Latest action and future plan - Features of several generic serial link technologies XAUI (10 Gb. E) Ser. Des(Rocket. IO) s. RIO PCIe Max speed / single 6~10 Gbps 3. 125~6. 5 Gbps※ 6. 25 Gbps 5~40 Gbps Max speed / several - Max length 3. 125~6. 5 Gbps x N 6. 25 Gbps x N - several 100 m / optical fiber Connector/Cable SFP+ (Q)SFP+ Trigger Yes Yes Optical option Yes Yes Standardization Yes Yes Cost of IP core Free Some ※ depend on transceiver We started preparing test stand for XAUI and Rocket. IO, and will finish evaluation within this fiscal year. 次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28

Latest action and future plan - Platform design - FMC (FPGA Mezzanine Card) separate a logic and I/O from a platform. change suitable I/F enough performance. ~20 Gbps Study items Actual bandwidth between FMCs. Actual bandwidth of DMA transfer at PCIe Hardware implementation FPGA Logic implementation 次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28

Summary • We have constructed an event-synchronized data acquisition system for >5 Gbps of data rate for the SACLA experiment. – Shot-by-shot data acquisition up to 60 Hz. – Expandable for user instruments. ⇒ Ready for user experiments. • Further upgrades for larger detector size and higher repetition rate will be necessary in future. – Faster data transfer and storage system. – Effective data compression / suppression. ⇒ Start fighting them. 次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28