The GBT Project Paulo Moreira TWEPP 24 th

The GBT Project Paulo Moreira TWEPP 24 th September 2009 http: //cern. ch/proj-gbt

GBT – People Luiz Amaral – CERN, Switzerland Mohsine Menouni – CPPM, France Sophie Baron – CERN, Switzerland Alessandro Marchioro – CERN, Switzerland Sandro Bonacini – CERN, Switzerland Frederic Marin – CPPM, France Jean-Pierre Cachemiche – CPPM, France Stefano Meroli – INFN, Italy Bruno Checcucci – INFN, Italy Paulo Moreira – CERN, Switzerland Jorgen Christiansen – CERN, Switzerland Christian Paillard – CERN, Switzerland Ozgur Cobanoglu – CERN, Switzerland Nataly Pico – SMU, USA Federico Faccio – CERN, Switzerland Antonio Ranieri – INFN, Italy Philippe Farthouat – CERN, Switzerland Giuseppe De Robertis – INFN, Italy Tim Fedorov – SMU, USA Angelo Rivetti – INFN, Italy Rui Francisco – CERN, Switzerland Sergio Silva – CERN, Switzerland Alessandro Gabrielli – INFN, Italy Csaba Soos – CERN, Switzerland Tullio Grassi – University of Maryland Filipe Sousa – CERN, Switzerland Ping Gui – SMU, USA Jan Troska – CERN, Switzerland Paul Hartin – SMU, USA Francois Vasey – CERN, Switzerland Kostas Kloukinas – CERN, Switzerland Ken Wyllie – CERN, Switzerland Gianni Mazza – INFN, Italy Bryan Yu – SMU, USA http: //cern. ch/proj-gbt Paulo. Moreira@cern. ch 2

Outline § GBT at TWEPP 2009 § Radiation Hard Optical Link Architecture • GBTX-TO-FRONTEND: • • § The GBT Chipset: • • • § Parallel Modes E-Link Modes Gigabit Laser Driver (GBLD) Gigabit Trans-Impedance Amplifier (GBTIA) Gigabit – Serializer/De-serializer (GBT-SERDES) E-Links • • • E-Port Scalable Low-Voltage Signalling (SLVS) SLVS Transceiver Macrocell § GBT on FPGAs § GBT Specifications § GBT project schedule http: //cern. ch/proj-gbt Paulo. Moreira@cern. ch 3

The GBT at TWEPP 2009 Thursday, 24 September 2009 Friday, 25 September 2009 § § B 5 - Optoelectronics and Links • 11: 00 - 11: 25 (GBT) Paulo MOREIRA, • § • A 5 – ASICS: • 12: 15 - 12: 40 (GBT) Giovanni MAZZA, • § “The GBT project” 11: 25 - 11: 50 (VL) Jan TROSKA, “The Versatile Transceiver Proof of Concept” Session 6 - Programmable Logic, Boards, Crates and Systems • 10: 10 - 10: 35 (GBT) Frederic MARIN, “Implementing the GBT data transmission protocol in FPGAs” 10: 35 - 11: 00 (VL) Csaba SOOS, “FPGA-based Bit-Error-Ratio Tester for SEUhardened Optical Links” “A 5 Gb/s Radiation Tolerant Laser Driver in 0. 13 um CMOS technology” 12: 40 - 13: 05 (GBT) Mohsine MENOUNI, “The GBTIA, a 5 Gbit/s radiation-hard optical receiver for the SLHC upgrades” POSTERS SESSION • Poster 19 (GBT) Ozgur COBANOGLU, • • • “A Radiation Tolerant 4. 8 Gb/s Serializer for the Giga-Bit Transceiver” Poster 28 (GBT) Sandro BONACINI, “e-link: A Radiation-Hard Low-Power Electrical Link for Chip-to-Chip Communication” Poster 64 (GBT) Alessandro GABRIELLI, “The GBT-SCA, a radiation tolerant ASIC for detector control applications in SLHC experiments” Poster 115 (VL) Sergio SILVA, Characterization of Semiconductor Lasers for Radiation Hard High Speed Transceivers http: //cern. ch/proj-gbt Paulo. Moreira@cern. ch 4

Radiation Hard Optical Link Architecture Defined in the “DG White Paper” Radiation Hard Optical Link: § § “Work Package 3 -1” • Objective: • Development of an high speed bidirectional radiation hard optical link • Deliverable: • Duration: • • Tested and qualified radiation hard optical link 4 years (2008 – 2011) § Versatile link project: • • GBT project: • • On-Detector Custom Electronics & Packaging Radiation Hard http: //cern. ch/proj-gbt Opto-electronics Radiation hardness Functionality testing Packaging Paulo. Moreira@cern. ch ASIC design Verification Functionality testing Packaging Off-Detector Commercial Off-The-Shelf (COTS) Custom Protocol 5

GBTX-TO-FRONTEND: Parallel Modes § P-Bus Mode: • • § B-Bus Mode: • • § Simple parallel interface 40 -bit wide bus Bidirectional Double Data Rate (DDR) A byte-bus mode is also available Up to five independent buses can be used simultaneously Electrical levels: • SLVS electrical level: • • 100 W termination 400 m. V differential 200 m. V common mode ILOAD = ± 2 m. A JEDEC standard, JESD 8 -13 Scalable Low-Voltage Signalling for 400 m. V (SLVS-400) http: //www. jedec. org/download/search/JESD 8 -13. pdf http: //cern. ch/proj-gbt Paulo. Moreira@cern. ch 6

GBTX-TO-FRONTEND: E-Link Modes § § § SEU tolerant § § http: //cern. ch/proj-gbt Paulo. Moreira@cern. ch GBT/Frontend interface: • • Electrical links (e-link) Serial Bidirectional Up to 40 links Programmable data rate: • • • Independently in five groups Independently for up/down links 80 Mb/s, 160 Mb/s and 320 Mb/s Lanes: • • To achieve > 320 Mb/s Two or more e-links can be grouped forming a “lane” Slow control channel: • 80 Mb/s E-Link: • • Three pairs: DOUT/DIN/CLK SLVS E-Links will be handled by E-ports: • • Electrically “Protocol” Package (preliminary): • • BGA: 361 – PINS 16 mm x 16 mm, 0. 8 mm pitch 7

The GBT Chipset § • • § § Radiation tolerant chipset: GBTIA: Transimpedance optical receiver GBLD: Laser driver GBTX: Data and Timing Transceiver GBT-SCA: Slow control ASIC Supports: • • • § Data readout TTC Slow control and monitoring links. Radiation tolerance: • • Bidirectional data transmission Bandwidth: • • The target applications are: Total dose Single Event Upsets Line rate: 4. 8 Gb/s Effective: 3. 36 Gb/s GBTIA Data<119: 0> Clock<7: 0> GBTX GBLD Frontend GBT-SCA Electronics Control<N: 0> http: //cern. ch/proj-gbt Paulo. Moreira@cern. ch 8

GBLD Main specs: • • Bit rate 5 Gb/s (min) Modulation: • • • Laser modulation current: 2 to 12 m. A Laser bias: 2 to 43 m. A “Equalization” • • • § current sink Single-ended/differential Pre-emphasis/de-emphasis Independently programmable for rising/falling edges Supply voltage: 2. 5 V Die size: 2 mm × 2 mm I 2 C programming interface Packaging: • Part of the versatile link project Engineers : • • • Gianni Mazza – INFN, Italy Angelo Rivetti – INFN, Italy Ken Wyllie – CERN, Switzerland Status: • • Chip fabricated and tested A re-spin is necessary Giovanni MAZZA et al. , “A 5 Gb/s Radiation Tolerant Laser Driver in 0. 13 um CMOS technology” http: //cern. ch/proj-gbt Paulo. Moreira@cern. ch 9

GBLD – Test Results § GBL tests: • • § The problem has now been identified: • • § Chip is “functional” however the bandwidth falls short of specifications! Fortunately the pre-emphasis circuit is working fine, allowing to partially recover the bandwidth! 2. 5 Gb/s Parasitic capacitance evaluation Layout symmetry The layout issues are being corrected: • Submission: 9 th of November 5 Gb/s http: //cern. ch/proj-gbt Paulo. Moreira@cern. ch 10

GBTIA Main specs: • Bit rate 5 Gb/s (min) • Sensitivity: 20 μA P-P (10 -12 BER) • Total jitter: < 40 ps P-P • Input overload: 1. 6 m. A (max) • Dark current: 0 to 1 m. A • Supply voltage: 2. 5 V • Power consumption: 250 m. W • Die size: 0. 75 mm × 1. 25 mm Packaging: • Part of the versatile link project Engineers : • Ping Gui – SMU, USA • Mohsine Menouni – CPPM, France Status: • Chip fabricated and tested • Chip fully meets specifications! • Radiation tolerance proven! • Work has started to encapsulate the GBTIA + PIN-diode in a TO Package • (Versatile link project) Mohsine MENOUNI et al. , “The GBTIA, a 5 Gbit/s radiation-hard optical receiver for the SLHC upgrades” http: //cern. ch/proj-gbt Paulo. Moreira@cern. ch 11

GBTIA – Test Results -6 d. Bm -18 d. Bm (specs: -17 d. Bm) http: //cern. ch/proj-gbt Paulo. Moreira@cern. ch 12

GBT-SERDES Engineers: Ozgur Cobanoglu - CERN, Switzerland Federico Faccio - CERN, Switzerland Rui Francisco – CERN, Switzerland Ping Gui – SMU, USA Alessandro Marchioro - CERN, Switzerland Paulo Moreira - CERN, Switzerland Christian Paillard - CERN, Switzerland Ken Wyllie - CERN, Switzerland http: //cern. ch/proj-gbt Paulo. Moreira@cern. ch 13

Forward Error Correction (FEC) High rates of Single Event Upsets (SEU) are expected for SLHC links: • • Experimental results confirmed that for Error -rates below 10 -12 Error Correction is mandatory (J. Troska et al. ) ! • § Upsets lasting for multiple bit periods have been observed Proposed code: • • Interleaved Reed-Solomon double error correction 4 -bit symbols (RS(15, 11)) Interleaving: 2 Error correction capability: • • • Code efficiency: 88/120 = 73% Line speed: 4. 80 Gb/s Coding/decoding latency: one 25 ns cycle • § BER § Particle “detection” by Photodiodes used in optical receivers. SEUs on PIN-receivers, Laser-drivers and SERDES circuits 2 Interleaving × 2 RS = 4 symbols 16 -bits GBT frame efficiency: 70% • • • A line code is always required for DC balance and synchronization For comparison, the Gigabit Ethernet frame efficiency is 80% (at the physical level) At a small penalty (10%, when compared with the Gigabit Ethernet) the GBT protocol will offer the benefits of Error Detection and Correction http: //cern. ch/proj-gbt BER § Paulo. Moreira@cern. ch 14

GBT-Serializer § Serializer: • • 4. 8 Gb/s 120 -bit shift register • • § Data path: • • § Divide by 120 f = 4. 8 GHz Triple voted for SEU robustness PLL: • § No SEU protection SEUs handled by the Reed-Solomon CODEC Clock divider: • • • § 3 × 40 -bit shift register (f=1. 6 GHz) 3 -to-1 fast multiplexer (f=4. 8 GHz) SEU hardened VCO Engeneers: • • • Ozgur Cobanoglu - CERN, Switzerland Federico Faccio - CERN, Switzerland Paulo Moreira - CERN, Switzerland Ozgur COBANOGLU et al, “A Radiation Tolerant 4. 8 Gb/s Serializer for the Giga-Bit Transceiver” http: //cern. ch/proj-gbt Paulo. Moreira@cern. ch 15

GBT-Serializer § Clock divider: • § TMR in the counter mandatory: • • • § A “missing” count on the clock divider unlocks the PLL resulting in a burst of errors lasting for “ms” A voter + a static DFF is far to slow! Even a voter + a dynamic DFF will not pass the corner simulations! A new “voted DFF” is proposed and used in the GBT-SERDES VCO: • • Depending on the charge injected, an SEU can stop the VCO oscillations for a few cycles! SEUs were modeled by current pulses injecting 300 f. C in 10 ps The VCO current is set so that an SEU event causes a phase shift which is a fraction of the bit period. Penalty: high power consumption. http: //cern. ch/proj-gbt Paulo. Moreira@cern. ch 16

GBT-CDR § Dual PLL CDR Loop: • • • § 1 st Loop: Frequency centering PLL 2 nd Loop: CDR Allows to reduce the CDR VCO gain for lower Jitter Half-Rate: • • Phase-detector Frequency-detector § Constant latency frame alignment circuit § As for the serializer: • • • § Unprotected data path TMR clock divider SEU hardened VCO Engineers: • • Ozgur Cobanoglu CERN, Switzerland Federico Faccio CERN, Switzerland Rui Francisco – CERN, Switzerland Paulo Moreira - CERN, Switzerland http: //cern. ch/proj-gbt Paulo. Moreira@cern. ch 17

Phase-Shifter X 1 X 8 (X 4 in the GBT-SERDES) Main features: • • 8 – channels (4 in the GBT-SERDES prototype) 1 PLL + Counter generates the three frequencies: 40 / 80 and 160 MHz 1 DLL per channel Mixed digital/analogue phase shifting technique: • Coarse deskewing – Digital • Fine deskewing – Analogue Power consumption: 5. 6 m. W/channel (simulated) Differential non-linearity: <6. 7% LSB Integral non-linearity: INL<6. 5% LSB Engineers : • • • Ping Gui – SMU, USA Tim Fedorov – SMU, USA Paul Hartin – SMU, USA Nataly Pico – SMU, USA Bryan Yu – SMU, USA http: //cern. ch/proj-gbt Paulo. Moreira@cern. ch 18

Slow Control & Monitoring ASIC: GBT - SCA GBT-SCA Main specs: • Dedicated to slow control functions • Interfaces with the GBTX using a dedicated E-link port • Communicates with the control room using a protocol carried (transparently) by the GBT • Implements multiple protocol busses and functions: • • I 2 C, JTAG, Single-wire, parallel-port, etc… Implements environment monitoring functions: • • Temperature sensing Multi-channel ADC Engineers: • • • Alessandro Gabrielli – INFN, Italy Kostas Kloukinas – CERN, Switzerland Alessandro Marchioro – CERN, Switzerland Antonio Ranieri – INFN, Italy Giuseppe De Robertis – INFN, Italy Filipe Sousa – CERN, Switzerland Status • Specification work undergoing: Alessandro GABRIELLI et al. , • 1 st Draft already available “The GBT-SCA, a radiation tolerant ASIC • • RTL design undergoing Tape-out: 2010 for detector control applications in SLHC http: //cern. ch/proj-gbt experiments” Paulo. Moreira@cern. ch 19

E-Links: e-port The FE interfaces with the GBTX through an e-port Specification work in progress The e-port handles: 7 B/8 B • • Two “tentative” protocols “proposed”: The physical interface; • Balanced The multiple data rates; • Fixed latency • Suitable for Trigger commands links • RTL code under development The lanes (for bandwidth > 320 Mb/s) Line coding: Clock recovery (if required) High-Level Data Link Control (HDLC) AC coupling (if required) The user application does not have to care about the frame formats in full detail: • • • It s done through a standard protocol: Atlantic interface proposed (http: //www. altera. com/products/ip/altera/t-altatlantic. html) • Packet oriented data • Bandwidth efficiency (~96%) • Non-fixed latency • Suitable for slow control and data links • RTL code ready To be specified Fixed latency needs to be considered! • Clock recovery and phase alignment • Lanes support An E-Link Port Adaptor (EPA) “macro” will be available for integration in the front-end ASICs Engineers: • • Sandro Bonacini – CERN, Switzerland Kostas Kloukinas – CERN, Switzerland http: //cern. ch/proj-gbt Paulo. Moreira@cern. ch 20

SLVS Tests § SLVS (Scalable Low Voltage Standard) § § JEDEC standard: JESD 8 -13 Tests based on the LM 4308 component from National Semiconductor: • • § Main features: § 2 m. A Differential max § Line impedance: 100 Ohm § § Signal: +- 200 m. V Various types of transmission media tested: • • • § Common mode ref voltage: 0. 2 V § 18 -bit CPU interface chip Complete SERDES, configurable as transmitter or receiver Kapton PCB Ethernet cable Test equipment • • Bidirectional link FPGAs perform pseudo-random data generation and checking Sandro BONACINI et al. , “e-link: A Radiation-Hard Low-Power Electrical Link for Chip-to-Chip Communication” http: //cern. ch/proj-gbt Paulo. Moreira@cern. ch 21

SLVS Transceiver § Receiver • • Electrical Specifications Power Supply: 1. 2 V to 1. 5 V Power Dissipation: • • § 300 u. W @ 320 Mbs <1 u. W @ power down Transmitter • • PR EL IM IN AR Y Electrical Specifications Power Supply: 1. 2 V to 1. 5 V Power Dissipation: • • 3. 0 m. W @ 320 Mbs <10 u. W @ power down Programmable Output Current § Engineer • Sandro Bonacini – CERN, Switzerland Status: • • Chip submitted MOSIS July MPW http: //cern. ch/proj-gbt Paulo. Moreira@cern. ch 22

GBT on FPGAs § GBT-SERDES successfully implemented in FPGAs: • Scrambler/ Descrambler + Encoder/ Decoder Xilinx - 4. 8 Gb/s + Serializer/CDR § FPGA Tested: • XILINX Virtex-4 FX • ALTERA Stratix. II GX § Ongoing work: • Optimization of use of resources • • § Firmware: • § Detailed implementation is device dependent Fixed and “deterministic” latency Reference designs will be available on both Xilinx and Altera devices Altera + opto TRx - 4. 8 Gb/s Engineers: • Sophie Baron – CERN, Switzerland • Jean-Pierre Cachemiche – CPPM, France • Frederic Marin– CPPM, France • Csaba Soos – CERN, Switzerland Frederic MARIN et al, “Implementing the GBT data transmission protocol in FPGAs” Csaba SOOS, “FPGA-based Bit-Error-Ratio Tester for SEUhardened Optical Links” http: //cern. ch/proj-gbt Paulo. Moreira@cern. ch 23

GBT Specifications Link specification group: § Formed in 2008 § Members: • Electronics coordinators of: • ALICE, ATLAS, CMS and LHCb • Five members of the Radiation Hard Optical Link (RHOL) project § “Mandate”: • Identify the “GBT” needs of each experiment for the SLHC upgrade • Discuss the specification documents (before they are distributed within the collaborations). § Meetings: • 1 st meeting (CERN – 2008/04/17) • ALICE, ATLAS, CMS and LHCb electronics coordinators presented outlooks of their requirements for SLHC. nd • 2 meeting (CERN – 2008/11/14) • GBT system proposal was presented to the electronics coordinators. rd • 3 meeting (CERN – 2009/05/05) • Link specification feedback Documents: § Share point web site created (2008): • http: //cern. ch/proj-gbt § Specification documents: • GBTX specifications (V 1. 0, January 2009) • GBTIA specifications (V 1. 7, May 2008) • GBLD specifications (V 2. 0, July 2008) • GBT-SCA specifications (V 1. 5, June 2008) • E-Port IP 7 B 8 B specifications (V 0. 1, December 2008) • E-Port IP Core specifications (V 0. 2, January 2009) • E-Port IP HDLC specs (V 0. 2, January 2009) http: //cern. ch/proj-gbt Paulo. Moreira@cern. ch 24

Project Schedule § 2008 • Design and prototyping of performance critical building blocks: • • First tests of optoelectronics components • • • § Proceed with the link specification meetings General link specification Design/prototype/test of basic serializer/de-serializer (GBT-SERDES) chip • • • GBT-SERDES (“Tape-out” 9 th of November) Design/prototype/test of optoelectronics packaging • GBTIA + PIN on TO CAN Detailed link specification document (4 th Quarter) 2010 • • § SEU tests on PIN receivers 2009 • § GBTIA, GBLD, Serializer, De-Serializer, Phase Shifter Prototype of “complete” GBTX chip Full prototype of optoelectronics packaging 2011 • • • Extensive test and qualification of full link prototypes System demonstrator(s) with use of full link Schedule of the final production version is strongly dependent on the evolution of the LHC upgrade schedule http: //cern. ch/proj-gbt Paulo. Moreira@cern. ch 25

Additional slides § Additional slides § Additional slides § Additional slides § Additional slides § Additional slides § Additional slides § Additional slides § Additional slides § Additional slides § Additional slides § Additional slides § Additional slides § Additional slides § Additional slides § Additional slides http: //cern. ch/proj-gbt Paulo. Moreira@cern. ch 26

Power consumption: GBTIA + GBLD + GBTX PR http: //cern. ch/proj-gbt Paulo. Moreira@cern. ch EL IM IN AR Y 27

GBT Link Bandwidth § § § § Bandwidth: • • User: 3. 36 Gb/s Line: 4. 8 Gb/s Dedicated channels: • • Link control: 80 Mb/s Slow control channel: 80 Mb/s Scrambler No bandwidth penalty • Efficiency: 73% To be compared with 8 B/10 B: 80% (no error correction capability) http: //cern. ch/proj-gbt Down-link highly flexible: (Will be clear later when discussing e-links) • Forward Error Correction and Frame Synchronization • • Link is symmetrical: • DC balance: • • Link is bidirectional § Can convey unique data to each frontend device that it is serving “Soft” architecture managed at the control room level Other schemes would require dedicated topologies that will be difficult to accommodate on a generic ASIC like the GBTX Now demonstrated to be compatible with FPGAs Paulo. Moreira@cern. ch 28

Single Event Upsets § An issue that needs to be carefully addressed for SLHC links is the expected high rates of Single Event Upsets (SEU). § Theses can be caused by: • • § Particle “detection” by Photodiodes used in optical receivers. SEUs on PIN-receivers, Laser-drivers and SERDES circuits First SEU Test results on High Speed Links (1. 5 - 2. 5 Gb/s) • Carried out first survey of results in different devices in Dec. 07 § § 12 device types tested • PINs without TIA • ROSAs with TIA Confirmed that for Error-rates below 10 -12 Error Correction is mandatory § Measured Upsets lasting multiple bit periods for the first time http: //cern. ch/proj-gbt Paulo. Moreira@cern. ch 29

Radiation Hard Optical Link Project Collaborators http: //cern. ch/proj-gbt Paulo. Moreira@cern. ch 30

R 61 -bit LSB S 0 rx. Input. P rx. Input. N 1: 2 MSB 61 rx. Even. Register<60: 0> rx. Odd. Register<60: 0> Sp F R 61 -bit LSB Phase Detector Half Rate 61 MSB 61 D Q SR 61 -bit LSB CP LPF + VCO MSB 1/2 rx. Clk 40 MHz 1/2 rx. Clk 80 MHz 1/5 rx. Clk 160 MHz 1/3 : 800 MHz F = 0º 2. 4 GHz 1/3 1/2 : 1. 2 GHz F = 90º rx. Skip. Cycle rx. Phase. Invert Coarse tuning LPF Replica control voltage PFD CP LPF VCO 1/60 Clock reference http: //cern. ch/proj-gbt Paulo. Moreira@cern. ch 31