Ongoing Developments for the ATLAS LAr Subdetector Readout
Ongoing Developments for the ATLAS LAr Subdetector Readout Possible replacement of VME for the upgrades of ATLAS 12 July 2012 Guy Perrot on behalf of the LAr upgrade collaboration
Outline • Context • US Developments (BNL/Stony Brook & U. of Arizona) – ATCA Development Blade – AMC ROD Injector • European Developments (Dresden & LAPP) – 10 Gb. E on Fabric interface – IPM Controller & ATCA Test Blade – ROD Evaluator • LAr Today’s Path for 2018 • Answers to your questions • Summary 12/07/2012 Possible VME Replacement for the ATLAS upgrades G. Perrot 2
Current & Future Architectures Phase. III 2021 2018 Phase FEBDigital & RODTrigger Change Current 128 cells/FEB 320 scells/LTDB system ~1600 FEBs ~120 LTDBs 1600 FEBs ~100 ATCA ~ 30 ATCA 192 RODs ROD blades ? DPS Blades ~120 TBBs 150 ~24 Tbps. IN IN 0. 4 Tbps OUT ~24 Tbps OUT Up to 1. 5 Tbps Monitoring 12/07/2012 Possible VME Replacement for the ATLAS upgrades G. Perrot 3
US developments (1) • BNL ATCA Development Blade Zone 3 (to transition module) LVL 1 communication Update Channel Global Clocks 12 fiber inputs SNAP 12 format DDR 2 SODIMM Memory Virtex 5 FXT FPGA Serial Debug Port 12/07/2012 Power Conversion Base Interface (Gigabit Ethernet) Fabric Interface (10 Gig Ethernet) -48 v Power Input and IPMI interface – No IPM Controller (was supposed to be DESY Firmware but lack of time) – Shortcuts to get power on. – Room for 2 Mezzanines (not AMC) • Used by Dresden to test 10 Gb. E on Fabric Possible VME Replacement for the ATLAS upgrades G. Perrot 4
US developments (2) • BNL DPS/ROD s. PU AMC – In design phase – DESY/CPPM/CERM MMC (ATMEL Processor) – Virtex 7 FPGA – 48 10 Gbps links • U. of Arizona ROD Injector AMC – 2 x 12 channels @ 6. 25 Gbps (Front) – Stratix EP 4 SGX 230 KF 40 – 12 channels @ 5 Gbps (Gb. E or PCIe) on Fabric – MMC implemented using an ARM Processor – Future AMCs to use CPPM/CERN MMC – AMC only used in u. TCA Shelf 12/07/2012 Possible VME Replacement for the ATLAS upgrades G. Perrot 5
European developments (TUD) • Technische Universitat Dresden: 10 Gb. E through ATCA Fabric Interface – ATCA shelf: 20 port 10 Gb. E Switch, BNL Sub. ROD (Virtex 5 FPGA) – Server: Dual 4 -Core Xeon, 4 x Dual Port Myricom 10 Gb. E NIC – Link: optical between shelf and server • Results: – 80 byte UDP packets: ~400 Mbps only! with packet losses – 1500 byte packets: ~4. 4 Gbps with packet losses – 9000 byte jumbo packets: ~10 Gbps without packet loss 12/07/2012 Possible VME Replacement for the ATLAS upgrades G. Perrot 6
European Developments (LAPP) ATCA Controller Mezzanine • FMC Mezzanine Power Supplies Ethernet µC LM 3 S 9 B 92 JTAG FPGA Spartan 6 JTAG EEPROM connector USB bus IO • I 2 C IPMBus A&B • µC IO • 69 mm FPGA • 76. 5 mm 12/07/2012 IPM Controller – Communication with Shelf manager through IPMBus A & B – Hot Swap, Power management etc. . ATCA blade manager – Communication via Ethernet (front panel or ATCA Base Interface) – User functions • Firmware Upgrade • ATCA blade monitoring & configuration • User stuff…. Software libraries available FMC (FPGA Mezzanine Card) compliant – up to 160 customizable links( including 74 differential links) – Low cost Features: – ARM cortex M 3 processor – Xilinx Spartan 6 for highly configurable user IO – Ethernet / USB / JTAG interfaces Possible VME Replacement for the ATLAS upgrades G. Perrot 7
European Developments (LAPP) ATCA Test Blade • ATCA Controller Mezzanine tests – IO connections (JTAG boundary scan tests) – IPMI management with the Shelf manager – Ethernet communication through ATCA Base Interface • ROD Evaluator tests – Check Blade configuration with the ATCA Controller Mezzanine (Firmware upgrade, Configuration upload etc. . ) – Test ATCA compliant power supplies – Check FPGA Design (communications with DDR 3, Flash, configuration with Flash) 12/07/2012 Possible VME Replacement for the ATLAS upgrades G. Perrot 8
European Developments (LAPP) ATCA Controller Mezzanine Version 2 • • • Smaller size: DDR 3 VLP Mini-DIMM Based on 2 µC ARM cortex M 4 from ST Microelectronics IPMC features – – – – • JTAG Master – • • JTAG master via Ethernet (ATCA board debugging, firmware upgrade) Custom interface – 18. 30 mm IPMB_0 with on board buffers, Hardware address detection Hot Swap management with ATCA Leds and front panel switch Management of up to 8 AMC + RTM On board Event LOG FRU & SDR via I 2 C Access to ATCA board sensors via I 2 C Configurable User Signals for Payload management Possibility to have a custom interface between the Mezzanine and the ATCA board for custom functionalities Other – USB and UART interfaces (debugging etc. . ). 82. 127 mm 12/07/2012 Possible VME Replacement for the ATLAS upgrades G. Perrot 9
European Developments (LAPP) ATCA Test Board Version 2 PWR Mgt AMC 1 switch Base Interface ETh Mezzanine Led, switch User IO RSVD FRU FPGA 12/07/2012 M M C PWR Mgt AMC 4 RJ 45 • ATCA Blade with 4 AMC slots. • MMC mezzanine • Designed to test all the functionalities of the ATCA controller mezzanine. • Could be used as “Reference Design” for users. I 2 C HA Etc. . T° DC/DC Power Mgt Possible VME Replacement for the ATLAS upgrades G. Perrot 10
European Developments (LAPP) ROD Evaluator SNAP 12 Rx (Carrier) DDR 3 FPGA FLASH CPLD SNAP 12 Tx (Carrier) DC/DC : 0. 9 V DC/DC : 3. 3 V, 2. 5 V, 1. 8 V, 1. 5 V ATCA IPMC • • Fully compliant ATCA blade. To evaluate high speed and high density optical & electrical transmissions. To evaluate high power DSP cells from FPGA Board management trough Ethernet (Firmware upgrade, DSP configuration…) 12/07/2012 ATCA DC/D C (FMC carrier) - - 48 optical Rx links @8. 5 Gbit/s 48 optical Tx links @8. 5 Gbit/s 8 Electrical Rx/Tx links @8. 5 Gbit/s 16 Electrical Rx/Tx LVDS links @ 1. 6 Gbit/s between FPGAs Connection to Fabric & Base interface Possible VME Replacement for the ATLAS upgrades G. Perrot 11
Phase I Architecture Phase I 2018 Digital Trigger 320 scells/LTDB ~120 LTDBs ~ 30 ATCA DPS Blades (LDPB) ~24 Tbps IN ~24 Tbps OUT Up to 1. 5 Tbps Monitoring 12/07/2012 Possible VME Replacement for the ATLAS upgrades G. Perrot 12
LDPB Data Path Block Diagram 820 Gbps 24. 2 Tbps DATA 40 MHz FE LTDB 205 Gbps 820 Gbps 24. 2 Tbps Data Processing RESULTS 40 MHz L 1 Calo FEX 49 Gbps (8 Gbps Res. only) 1. 45 Tbps (240 Gbps) DATA + RESULTS 400 KHz Monitoring Data Collector Memory DATA L 0 400 KHz Latency 2. 5 -10 us Data Processing LTDB (LAr Trigger Digitizer Board) : 118 Modules LDPB (LAr Digital Processing Blade) : 29 ½ Blades (4 LTDB / LDPB) FE (Front End) & FEX (Feature Extractor) Monitoring Data Collector 12/07/2012 Possible VME Replacement for the ATLAS upgrades G. Perrot Memory RESULTS 100 KHz Monitoring Data Collector 2 Gbps 60 Gbps L 1 100 KHz Latency 100 s of us 13
Back-End Block Diagram (In Work) Shelf Scells Raw data L T TTC, Control, D Monitoring B F E X Results to FEX L 1 Calo Shelf Manager Raw data? , result on L 0 Blade: LDPB Switch Gb. E Raw data? , result on L 1 AMC Data Processing & Memory Control, Monitoring TTC Switch Gb. E Switch 10 Gb. E AMC Data Processing & Memory Blade Ctrl. & Monitor. TTC 12/07/2012 Blade: Switch S w G i h b t E c Switch 10 Gb. E PM PC Monitoring Data Collector IPMC TTC Distrib ution Possible VME Replacement for the ATLAS upgrades G. Perrot TTC LDPS Partition (NEW!) 14 Busy
Answers (1) • What is the expected usage of the backplane? – ATCA IPM control of blades. – Blade & FE LTDB configuration & monitoring via base interface. – Data collection via fabric interface. • Do you use the fat pipes. If yes which protocol? – For the time being we are thinking 10 Gb. E. • Would you agree for a control through Ethernet? – Yes, we plan it that way! • Do you use synchronisation signals and common clocks? – No common clock for operation of blades. – TTC for clock transmission to FE and Evt signature at FE & BE. • Do you plan to use redundancy and/or hot swapping – Hot swapping YES, redundancy NO. • Do you plan full mesh or dual star backplane? – Full mesh, though we are using a dual star architecture, as it can do both. 12/07/2012 Possible VME Replacement for the ATLAS upgrades G. Perrot 15
Answers (2) • Do you need a local processor? – For the time being we don’t think so. • Which shelf manager do you plan to use? – The one that will come with the shelf (we have Pigeon Point). • Which IPMC do you plan to use? – We are developing an IPMC and are planning to use it. • Typical expected power consumption? – No idea yet, but probably quite a lot: 200 W/blade => 2. 8 KW ? ? ? • Which crate size to you plan to use (i. e. how many slots available)? – ATCA format with 14 slots + shelf manager. • Do you plan implementation with AMC mezzanine and a mother board? – We are planning to have 4 AMCs on an ATCA carrier blade. 12/07/2012 Possible VME Replacement for the ATLAS upgrades G. Perrot 16
Answers (3) • How sure are you that you need ATCA. Could you envisage µTCA? – We went straight for ATCA as we have a very big number of channels. – A study would be needed to understand data throughput on the u. TCA backplane compared to ATCA, nb of slots needed, x. TCA switches and external switches. We haven’t done that! • Do you need zero, one or more ATCA switch blades in your ATCA shelf? Which kind of switch? – To get the bandwidth for data collection we need 2 switch blades (10 Gb. E) 12 in (fabric) / 12 out (front). – Gb. E for the control of blades via base interface. – TTC distribution ? ? ? ? ? • Which MMC do you want to use on your AMC or in u. TCA? – We are planning to use the DESY/CPPM/CERN design. 12/07/2012 Possible VME Replacement for the ATLAS upgrades G. Perrot 17
Summary • A basic BE system will be installed in 2013 -14 using our ROD Evaluator (2 blades). • The Digital Trigger system in 2018 (Phase I) will allow us to develop, test and master all the technologies needed for the Phase II developments (replacement of FEBs and RODs) • It will also allow us to understand what is the best compromise in term of density, power consumption, data collection. 12/07/2012 Possible VME Replacement for the ATLAS upgrades G. Perrot 18
- Slides: 18