Data Acquisition Systems for Big Science Dr John

Data Acquisition Systems for Big Science Dr John Coughlan STFC Rutherford Appleton Laboratory Presentation for the NMI January 23 rd 2009 John Coughlan

Talk Outline • STFC Technology, Who we are • Data Acquisition for Big Science, CERN LHC • LHC-CMS Data Acquisition boards. • Next generation systems Presentation for the NMI January 23 rd 2009 John Coughlan

STFC, Rutherford Appleton Laboratory, Oxfordshire UK • Science and Technology Facilities Council, STFC • Largest of 7 UK Scientific Research Councils reporting to DIUS • 2, 000 scientists & engineers at main site Rutherford Appleton Laboratory Presentation for the NMI January 23 rd 2009 John Coughlan

STFC, Rutherford Appleton Laboratory, Oxfordshire UK • Space Science, Lasers, X-Ray and Neutron sources, Particle Physics +… • Operate several world-class large-scale research facilities on site, e. g. VULCAN Laser, ISIS Neutron source, Diamond X-Ray Light Source. • UK Hub supporting activities of University researchers on other international facilities, e. g. ESA, CERN Geneva Presentation for the NMI January 23 rd 2009 John Coughlan

Major Site Developments STFC, Rutherford Appleton Laboratory, Oxfordshire UK • Harwell Science and Innovation Campus… • New Detector Systems centre 2010/11 • Agreement for new ESA research centre Presentation for the NMI January 23 rd 2009 John Coughlan

STFC Technology • STFC Technology Department • • 300 Electronic & Mechanical Engineers with strong Scientific Expertise Developing Instrumentation for Large Scale Science Experiments • Telescopes to Medical Imaging, Micro MEMS to Major engineering structures, Novel sensors to Cryogenic Superconducting Magnets, Electronics and Microelectronics… • Europe wide Microelectronics design support. EUROPRACTICE. Presentation for the NMI January 23 rd 2009 John Coughlan

STFC Technology Electronics • • PCB Board and System Design FPGA Design. (Xilinx and Altera) Embedded systems. Board and System Test and System Integration • Microelectronics, ASICs, MAPS sensors • • Current generation of Large Scale Projects for CERN completed, LHC New projects starting R&D phase, European Free Electron Laser XFEL Presentation for the NMI January 23 rd 2009 John Coughlan

CERN European Laboratory for Particle Physics. Geneva Big Science Large Hadron Collider LHC Understanding the Origin of the Universe Started Operation (briefly!) in 2008. No Black Holes yet… Presentation for the NMI January 23 rd 2009 John Coughlan

CMS Experiment at LHC 2, 500 Scientists & Engineers in collaboration 12, 000 tons apparatus ~ 50 million electronic readout channels DAQ rate ~ 100 k. Hz Presentation for the NMI January 23 rd 2009 John Coughlan

CMS Installation 200 metres Underground STFC Microelectronics 80, 000 x Analogue Pipeline ASICs Extremely Radiation Hard VME Racks Water Cooled Presentation for the NMI January 23 rd 2009 John Coughlan

LHC Data Acquisition One Collision 40 Million Collisions every second. n Keep images from 100, 000 collisions every second. n Each image built up from 10 million analogue sensors. n Rarest new particles HIGGs ~ few dozen per year? n Need for Massively Parallel (FPGA) based Processing n ~ 1 TERA-Bytes / sec n Presentation for the NMI January 23 rd 2009 John Coughlan

LHC-CMS Data Acquisition Custom COTS Presentation for the NMI January 23 rd 2009 John Coughlan

LHC DAQ Requirements • • • Performance driven by science & experiments. Large channel counts. Large Form Factor PCBs High bandwidths. Custom data protocols. • Specific Functions with Flexibility unknown science. FPGAs • • Electronics developed in collaboration with scientists. Large communities. In parallel with detector development. Loose specifications. Fully custom systems where performance and cost justify design investment • • Long development cycles. Long operating times. 10+10 Years Larger systems. Designs frozen early. Minimise risk. Presentation for the NMI January 23 rd 2009 John Coughlan

LHC-CMS DAQ Board • Board Features – – – 9 U x 440 mm (VME mechanics) Optical Inputs, Analogue 96 ADC Channels 10 bit @ 40 MHz 10 x XC 2 V 2000 24 x XC 2 V 40 – 14 Layers – Double sided PCB • System has – – • 500 boards (large nr for PP) 24 Crates 8 Racks 50 KW Delivered to CERN 2006 Data In ~ 3 GByte/s. Data Out ~ 200 MByte/s Presentation for the NMI January 23 rd 2009 John Coughlan

LHC-CMS DAQ Board Development • • • Large boards. 9 U VME Mechanics. High I/O count Analogue & Digital Large nr FPGA BGAs. 676 pins 1 mm pitch Concerns for BGA assembly Design for Test. Boundary Scan • • • Design started in 2001. Prototypes (25 off) in 2003/4 before production (500 off) in 2005/6. FPGA Virtex II. Fix technology early. DCMs for channel synchronisation DCI useful but power hungry Double Data Rate I/O Presentation for the NMI January 23 rd 2009 John Coughlan

LHC-CMS DAQ Board Specs Analogue components repeated on 2 nd side Board parameters: - 9 U x 440 mm VME 64 x form factor - Optical/Analogue/Digital logic ; 96 ADC @ 40 MHz channels - 14 layers (incl. 6 power). FR 4. - Double-sided (secondary side with half of analogue components) - 6, 500 components (most passives are 0402). Surface mount - ENIG metal finish, for BGAs. Pb/Sn - 20, 000 connections ; 14, 000 vias - 100 micron tracking, some 75 micron diff imp, min gap 100 micron, - 37 BGAs (larger FPGA 676 pins on 1 mm pitch). All BGAs located on primary side. - Controlled impedance - Boundary Scan JTAG all Digital devices Presentation for the NMI January 23 rd 2009 John Coughlan

FPGA Assembly on PCBs 3 rd Batch Prototypes 2003 ALL 6 failed Boundary Scan on several BGAs. Shorts under BGAs. Rework failed too. 17, 000 BGAs in system EU Tender process for PCB & Assembly. Exception Ltd 1 out of 500 production boards failed Tests 2006 Presentation for the NMI January 23 rd 2009 John Coughlan

LHC CMS CERN Industry Awards Gold Award : Exception Ltd Presentation for the NMI January 23 rd 2009 John Coughlan New Electronics Cover Story

Other LHC-DAQ Boards • FPGA 130 -90 nm generation • 1 Gbps serial data. • DDR 2 • VME Form Factors • Conventional PCB manufacture. • Step change design required for Next Generation of Projects Presentation for the NMI January 23 rd 2009 John Coughlan

Next Generation Systems European X-Ray Free Electron Laser XFEL DESY Laboratory, Hamburg Operating in 2013 Presentation for the NMI January 23 rd 2009 John Coughlan

Single Protein Molecules X-ray Imaging Just before XRay pulse X-ray Diffraction Pattern During the pulse After pulse 30, 000 images per second each up to 16 MPixels Presentation for the NMI January 23 rd 2009 John Coughlan

X-RAY Pixel Detectors DAQ For XFEL Detector Pixel Sensors FPGA SFP+ Image Builders 10 G Fibre 1 30 m 10 GBytes/sec x N cards • • 16 M + tiled pixel detector at 30 K frames/s -> 1 – 6 TByte/s 128 x FPGA 40 nm + 10 G links Off detector DAQ next gen Advanced Telecoms ATCA crates 2008 - 2013… Scientific requirements ? Presentation for the NMI January 23 rd 2009 John Coughlan 8

Image Builder for XFEL Demonstrator Advanced Mezzanine Card KEEP OUT FMC KEEP OUT C O N N E C T O R RAM K E E P O U T FPGA M M G T G T M G T T R X X FPGA RAM KEEP OUT 180 mm Presentation for the NMI January 23 rd 2009 John Coughlan Xpoint T X R X C O N N E C T O R • • AMC Form Factor. Migrating to 8 FPGAs on 8 U ATCA? • FPGA ~ 16 x 3 -6 Gbps serial links • Analogue Cross Point for Image Building. 72 x 72 @ 3 -6 Gbps • • DDR 2/3 ~ 2 -4 GBytes B/W 1 -2 GBytes/sec In & Out • • • VITA 57 FMC Mezzanine I/O 2 x SFP+ opto TRx 10 Gbps (XAUI or RXAUI PHY) • m. TCA serial backplane

Next Generation Board Issues • High speed diff pairs 3 -6 Gbps. BGA pitch < 1 mm? • • Reduce Fabrication Risk. Advanced PCB design and construction techniques. Vias in pad, micro vias, Laser drill. Incremental build up layer PCB. • • • FPGA 40 nm generation FPGA to Memory interface. SO-DIMMs WASSO Memory controllers Hard/Soft IP from FPGA vendors • • 10 Gbps optical interfaces 3 -6 Gbps Serial Backplanes. • • Power. Multiple POL. Analogue. Decoupling caps. next gen FPGA packages. Pb Free manufacture? Tools Signal Integrity analysis, how to measure eye diagrams on board? Presentation for the NMI January 23 rd 2009 John Coughlan

Design Tools • PCB Design Tool Flow based on CADENCE. • Fast serial design Signal Integrity analysis, Hyper. Lynx • FPGA Design Flow Mentor Graphics • • How to integrate PCB and FPGA design flow? Pin outs. Need realistic FPGA designs to guide PCB layout, e. g. memory interfaces • Our expert (Paul Hardy) is here today Presentation for the NMI January 23 rd 2009 John Coughlan

Summary • STFC Technology, Who we are • Data Acquisition for Big Science, CERN LHC • LHC Data Acquisition boards. • Next generation systems, XFEL Presentation for the NMI January 23 rd 2009 John Coughlan

STFC, Rutherford Appleton Laboratory, Oxfordshire UK http: //www. scitech. ac. uk/ Presentation for the NMI January 23 rd 2009 John Coughlan

Spare Slides Presentation for the NMI January 23 rd 2009 John Coughlan