CRU WORK SUMMARY 1 st July 2015 CRU

CRU WORK SUMMARY 1 st July, 2015 CRU WORKSHOP 1 st July, 2015 Presented by Jubin MITRA 1

Team Members Variable Energy Cyclotron Wigner Research Centre (VECC) for Physics Jubin MITRA Shuaib Ahmad KHAN Tapan Kumar NAYAK Erno DAVID Tivadar KISS CERN Filippo COSTA Peter CHOCHULA Alex KLUGE University of Jammu Anik GUPTA Bose Institute Sanjoy MUKHERJEE With active help and support from: CPPM University Of Calcutta Jean-Pierre CACHEMICHE and the group Rourab PAUL Amlan CHAKRABARTI 1 st July, 2015 Presented by Jubin MITRA 2

TABLE OF CONTENTS • • • What are we thinking of CRU Interfaces? CRU Functional Block Diagram What we have learned about PCIe 40 ? Participation in PCIe 40 board testing and debugging Firmware development status Questions for Development Team 1 st July, 2015 Presented by Jubin MITRA 3

WHAT ARE WE THINKING OF CRU INTERFACES? CRU has three interfaces. It is marked as 1, 2 and 3 respectively. 1 – GBT Link (radiation tolerant High speed Optical Link) 2 – To be decided (GBT/TTS/PCIe/10 Gigabit PON) 3 – DDL 3 link ( PCIe Gen 3 x 16) 1 st July, 2015 Presented by Jubin MITRA 4

CRU FUNCTIONAL BLOCK DIAGRAM As can be seen in the functional diagram, the problem is broadly being classified into 5 super modules, which can then be divided into multiple submodules. It also includes a supervising module, to act as a debugging tool for design engineers. The CRU firmware specific to each detector will use different percentage of these modules. Of the 5 functional modules only CIU module is visible to the CRU interfaces, rest of the 4 modules are internal functional ones and it will be like a black box to CRU users. Development of CIU is our current priority 1 st July, 2015 Presented by Jubin MITRA 5

A glimpse of GBT Test Setup at VECC on Stratix V board 1 st July, 2015 Presented by Jubin MITRA 6

First prototype of the PCIe 40 made by CPPM in Marseille 1 st July, 2015 Presented by Jubin MITRA 7

Features of PCIe 40 V 1 board • FPGA used Arria 10 (10 AX 115 S 4 F 45 I 3 SGES) • MAX V for Flash programming • 1 PLX 8747 chip for converting two x 8 PCIE into one x 16 PCIe • 2 Jitter cleaner: Si 5338, CDCE 62005 • 10 Gig PLL Clock: SI 5315 • 4 Tx Minipod and 4 Rx Minipod – Each supporting 12 links = 48 High Speed links available for data acquisition or distribution of timing, fast and slow control to front-ends • 1 SFP+ or PON device for Timing and Trigger reception With Arria 10 Altera supports vertical migration to Stratix 10 same density device 1 st July, 2015 Presented by Jubin MITRA 8

What we gain from Arria 10 with respect to Stratix V FPGA ? 1 st July, 2015 Stratix V Arria 10 5 SGXEA 7 N 2 F 45 C 3 10 AX 115 S 4 F 45 I 3 SGES Core voltage 0. 85 V 0. 95 V (For ES) else 0. 9 V ALMs 234720 427200 Total I/Os 1064 960 GPIOs 840 624 GXB Channel PMA and PCS/ HSSI channels 48 72 PCIe Hard IP Blocks 4 4 Memory Bits 52428800 55562240 DSP Blocks 256 1518 27 x 27 Multiplier 256 1518 Fractional PLL 28 32 DLLs 4 - I/O PLLs - 16 Global Clocks 16 32 HPS CPU Core - 0 Presented by Jubin MITRA 9

Participation in PCIe 40 testing and debugging • • We were largely involved in the debug of serial links. Most issues were corrected THE RESULT IS THAT LINKS ARE ALL UP AND RUNNING We still have some improvements to do (calibrate the channels after power up, or program few PLLs with LHC frequency). Showing one of the many tests that have been conducted Test setup : Transmit 5 Gbps data from PCIe 40 to AMC 40 and vice verse. With pattern checker and generator on either side to check uplink and downlink data communication 1 st July, 2015 Presented by Jubin MITRA 10

Participation in PCIe 40 testing and debugging Result: We are able to transmit data from PCIe 40 to AMC 40, and vice verse Eye Diagram of PCIe Tx side showing wide opening Table showing a comparison the eye height and width for AMC 40 Tx and PCIe 40 Tx Important point to note here: There is a need of polarity inversion in the Tx side of the PCIe 40 transmission. 1 st July, 2015 Presented by Jubin MITRA 11

Participation in PCIe 40 testing and debugging Showing BER of data received from AMC 40 to PCIe 40 Showing eye diagram in EYEQ of data received in AMC 40 from PCIe 40 at 5 Gbps 1 st July, 2015 Presented by Jubin MITRA 12

PCIe 40 firmware development status • GBT • Design migration from Stratix V to Arria 10 • Complete Simulation of the firmware • Implemented it in PCIe 40 • Using presently available jitter free clock of PCIe (100 MHz) to test 4 Gbps data rate • Latency Measurement • Making of GBT QSYS wrapper • TO DO: • Run using 120 MHz clock, with required 4. 8 Gbps • Do board testing of GBT protocol between Stratix V and Arria 10 • Do test with GBT chip • PCIe • Gen 2 x 8 +Gen 2 x 8 test for 16 lane connectivity • Basic read write test on BAR • 12 channel Transceiver tool kit design is made to test board to board communication between Stratix V and Arria 10 1 st July, 2015 Presented by Jubin MITRA 13

GBT : Design migration of version 3. 1. 1 from Stratix V to Arria 10 Stratix V The coding is modified keeping it in standard with the other GBT FPGA core. So, it is easier to maintain and upgrade in future. Good news: No Tx latency optimization problem in Arria 10 GBT design 1 st July, 2015 Presented by Jubin MITRA 14

GBT : Complete Simulation of the firmware in modelsim Direct launch of simulation N. B. : “tx clkout” and “rx clkout” initially generates lower frequency and after some delay generates the required frequency. So, care should be taken when driving PLL from this signal. 1 st July, 2015 Presented by Jubin MITRA 15

GBT : Showing the In System Sources and Probes screenshot 1 st July, 2015 Presented by Jubin MITRA 16

GBT : Showing Signal Tap II Screenshot 1 st July, 2015 Presented by Jubin MITRA 17

GBT : Showing the signal quality of GBT operating at 4 Gbps using 100 MHz PCIe clk Test Setup The bumps appearing on the left of the eye diagram are due to the deemphasis of the minipod receiver. This is not an unwanted distortion of the signal. We can disable the de-emphasis at 4. 8 Gigs because this feature is useful only at higher speeds (10 Gbits/s). 1 st July, 2015 Presented by Jubin MITRA 18

GBT : Showing the test setup for Latency Measurement ISSP 1 st July, 2015 Presented by Jubin MITRA 19

GBT : QSYS Wrapper for fast system integration 1 st July, 2015 Presented by Jubin MITRA 20

GBT : QSYS Wrapper in use 1 st July, 2015 Presented by Jubin MITRA 21

PCIe Testing • Testing of Example Design for Arria 10 with Avalon-MM DMA interface (ep_g 2 x 8_avmm) with two modules of PCIe Gen 2 x 8 lanes. Successfully detects 2 boards PCIe Gen 2 x 8. N. B. : Use 2 separate global ref clocks for two modules • PCI express example design Gen 3 x 8 lane in group of two is detected in PCIe 40 board, but on reading the device information in Linux we found that it is detected as GEN 1 (this is the issue with ES 1 of Arria 10 FPGA, where Gen 3 is falling back to Gen 1). So the design is compiled again with GEN 2 x 8 in group of two which is detected in Arria 10 and Read/write test on BAR is done 1 st July, 2015 Presented by Jubin MITRA 22

Screenshot of PCIe detection in mindshare software PLX 8747 Switch PCIe gen 2 x 8 from Arria 10 FPGA 1 st July, 2015 PCIe gen 2 x 8 from Arria 10 FPGA Presented by Jubin MITRA 23

12 channel Transceiver tool kit design for board to board communication between Stratix V and Arria 10 at 10. 312 Gbps PCIe 40 Tx side signal PCIe 40 Tx signal as received in AMC 40 1 st July, 2015 Presented by Jubin MITRA 24

Questions for Development Team • • • A realistic estimation of final firmware occupancy, speed and toggle rate necessary for optimizing the power supply tree of the final production card. Specification of environmental conditions (air flow, ambient temperature) Specification of mechanical space available Next slide shows a graphical representation of this issue N. B. : In Arria 10 Clock Calibration must be done before the firmware in loaded 1 st July, 2015 Presented by Jubin MITRA 25

More Logic Occupancy and Toggle rate There is a trade off More Power Required More Power IC required Power ICs More Heat Generated PCIe 40 Need More Mechanical Space More Cooling Required It would be fine to get as soon as possible a realistic firmware in order to simulate the FPGA power consumption. 1 st July, 2015 Presented by Jubin MITRA 26

THANKS TO CPPM Team (Marseille) 1 st July, 2015 Presented by Jubin MITRA 27