Beam Secondary Shower Acquisition System Igloo 2 GBT

Beam Secondary Shower Acquisition System: Igloo 2 GBT Implementation Status GBT on Igloo 2 Meeting Jose Luis Sirvent Ph. D. Student 06/05/2014 1

0. Introduction 1. 1 Wire Scanner principle • Invasive method for beam transverse profile measurement. • Carbon wire interaction (30 um) generates shower of secondary particles. • Transversal profile: – – • System compromises: – – – • Accelerator Type Quantity SPS Rotating 6 SPS Linear 4 Wire position (X axis) Secondary rain (Y axis) Wire blow-up (heat) Losses produced Mechanical stresses (Bellows) Calibration procedures Vibrations Types: – – – Rotating Fast Rotating Short/Long Linear • Total Scanners: 31 • Usage in a daily basis at CERN 2

0. Introduction 1. 2 BWS Prototype (Mechanical Aspects) • Scan at least as fast as the existing system (20 m/s needed to avoid wire damage) • Absolute accuracy of beam width determination of about 5 um (~5%) • Reduction of play in mechanical system • • • All elements mounted on same axis High accuracy angular position sensor Optical position sensor (Encoder) Overcome bellow limitations Locate all moveable parts in the vacuum Minimize fork and wire deformations: Acceleration profile optimized for low vibrations Mechanical design for minimum shaft and forks deformation • Unified design for integration in the different accelerators (PS, SPS & LHC) Stepped Vacuum Barrel X Axis: Optical position sensor Magnetic Stopper Motor Resolver Y Axis: Diamond Detector Carbon Wire 3

0. Introduction 1. 3 Complete test assembly for Proof of concept evaluation Power QIE 10 A QIE 10 B p. CVD Passive Splitting System SMA Coax Cable 50 ohm (1 -10 m) Connector Cividec Module QIE 10 Test Board Needed to be developed Monitoring PC VFC Board Or Arria V Dev. Kit RF Resistive network Igloo 2 Development board with GBTx firmware emulation Motivation: The final proof-of-concept can be evaluated by using this assembly. The Igloo 2 in this case could be configured to act purely as a GBTx asic, this way the system could be suitable to work with Igloo 2 or GBTx in case of change for final board. Tasks: 1. The system has to be configured to work in a complete assembly by using the knowledge from the previous tasks. 2. The set-up should be done in a way to make possible Igloo 2 -> GBTx migration. 3. Final tests and evaluation has to be done in with the assembly for system demonstration 4. At this point a decision could be done regarding the FPGA usage or the development of a compact board with GBTx. 4

0. Introduction 1. 4 Development of a compact board for BWS p. CVD diamond detector readout TTC SFP+ Ethernet NEW VFC VME FMC Carrier Board BOBR VME Board Cividec Module RF Resistive network SMA QIE 10 B Igloo 2 Or GBTx Compact Front-End Board Needed to be developed Fibre Optic TTC Data Control SFP+ Back plane VME 64 Connector QIE 10 A Back plane VME 64 Connector SMA Memory p. CVD Passive Splitting System SFP+ VTRx FPGA Arria V Memory Power FMC Connector SFP+ Clk_bunch Clk_Turn Clk_Events SFP+ VME FMC Carrier Board (VFC) Back-End VME Crate Motivation: Finally the last part of the project would be the development and testing of a very simple and compact board containing the Rad-Hard or Rad-Tol components. Tasks: 1. The QIE 10 development board previously done has to be extended house the FPGA (or GBTx) and VTRx in the same board. 2. Extensive testing has to be done in this board to guarantee that the performance reached in the development kits is repeated in this version. 3. Radiation studies are also important to characterize the complete radiation hardness of the board and identify failures. 4. This system could be installed in paralel with an operational BWS PMT & Scintillator system to crossvalidate performances. 5

1. GBT-FPGA Some information and resources GBT-Protocol implementation on FPGAS for GBTx communication & Emulation: • Public Share. Point: https: //espace. cern. ch/GBT-Project/GBT-FPGA/default. aspx • Public SVN Releases (old): https: //svnweb. cern. ch/cern/wsvn/gbt_fpga • Mailing List: gbt-fpga-users@cern. ch • Contacts for support: sophie. baron@cern. ch , manoel. barros. marin@cern. ch Last news: • PH-ESE Repository (last updates): https: //svn. cern. ch/reps/ph-ese/be/gbt_fpga • Indico Follow-up (GBT-FPGA): https: //indico. cern. ch/event/283075/ • Support & code available for (Dev. Kits): – Xilinx Virtex 5 / 6 / 7 & Kinex 7 … – Altera Cyclone V & Statrix V… – Microsemi: Smatrfusion 2, Igloo 2 ? ? ? At least not yet… but makes sense (CMS, MOPOS, us…) First Release 2011 Two Versions: First Release March 2014 Standard (STD) Data Readout (DAQ) Low and Deterministic latency (LATOP) FE control & Time, Trigger and control (TTC) 6

1. GBT-FPGA The Basic concept (Modular design) GBT-FPGA Firmware Starter Kit for Altera and Xilinx devices. Sophie Baron 2010. CERN/PH/ESE. GBT-FPTA Starter kit user guide V 0. 1 * Vendor Specific Modules (IP Cores) 7

1. GBT-FPGA The Basic concept (Clocking Scheme and TTC clock recovery) GBT-FPGA Comments on deterministic latency and recommendations to handle optimization schemes. Sophie Baron, PH/ESE. 2011 8

1. GBT-FPGA Current Status (Latency optimization) GBT-FPGA One unified core for multiple users. Manoel Barros Marin, PH/ESE/BE Students-Fellows seminar (05/02/2014). 9

2. Igloo 2 GBT-FPGA The code and the work I’m carrying out • For the moment I’ll migrate with the old STD version and only GBT frame (simplest), once done, tested and well understood, I’ll move to LATOP version. 10

2. Igloo 2 GBT-FPGA STD: Implementation status Substitute Xilinx IP’s by Microsemi IP’s (Some are straighforward) 11

2. Igloo 2 GBT-FPGA STD: Implementation status Substitute Xilinx IP’s by Microsemi IP’s (Others. . not) • Transceiver with EPCS @ 4. 8 GBPS: - A lot of configuration registers - Big amount of documentation - Different implemented protocols (not needed) - Power-Up Initialization needed (HPMS) - Synchronization issues - Needed standalone testing and verification 12

3. GBT-FPGA Overview in Igloo 2: (Clock Management as in Virtex 6) GBT_BANK SERDES_INIT_MASTER (Very simplified view) Data_In (83 bits) @40 Mhz TX_Word_CLK (240 MHz) TX_Frame_CLK (40 MHz) GBT_TX Scrambler Encoder Gearbox APB_BUS (PLL) Tx_Word (19 bits) @ 240 MHz Tx_CLK (240 MHz) GBT_MGT SERDES_0 Vendor Specific IP TX_Data_P (4. 8 Gbps) TX_Data_N (4. 8 Gbps) Ref. CLK 1_P (120 MHz) Ref. CLK 1_N (120 MHz) Rx_CLK (240 MHz) RX_Data_P (4. 8 Gbps) RX_Data_N (4. 8 Gbps) Rx_Word (19 bits) @ 240 MHz Data_Out (83 bits) @ 40 Mhz GBT_RX Gearbox Decoder Descrambler RX_Word_CLK (240 MHz) RX_Frame_CLK (40 MHz) RX_PLL TX_PLL 13

3. GBT-FPGA Overview in Igloo 2: 3. 1 The MGT block (Schematic view) Written from scratch According to Documentation: • When using SERDES the HPMS should drive the APB bus for Configuration. • 1 PLL Needed • No full control over SERDES • Manual configuration is possible • SERDES Registers should be properly initialized when using EPCS • When using EPCS the SERDES should be in PMA driven mode (manual bit lock steps). • Microsemi Recommends Default configurations with minor changes: • 1. 25 / 2. 5 GBPS • Lack of documentation for custom modes • Libero is constantly being updated and bugs repared… 14

3. GBT-FPGA Overview in Igloo 2: 3. 1 The MGT block (Schematic view) Written from scratch APB_BUS Recommendations by Microsemi: • Shadow_FIFOs with CLKint • Clock Constrains in TX & RX CLKs • Place FIFOS as close as possible from SERDES • Drive the 4 lanes reset with the same signal!! • Otherwise does not work!! CLK_TX_WORD CLK_RX_WORD 15

3. GBT-FPGA Overview in Igloo 2: 3. 1 The MGT block : SERDES Initialization & Control • Done through the Igloo 2_APB_MASTER: – Access to the SERDES registers for configuration and status check. Interface with the APB Bus protocol Very simple Read_Write Operations Decides which actions to execute, registers to configure and workflow 16

3. GBT-FPGA Overview in Igloo 2: 3. 1 The MGT block : SERDES Initialization & Control • Taking a look at Igloo 2_SERDES_APB_MANAGER: Registers Initialization SERDES_CLK = REF_CLK 1 EPCS Mode @ 4. 8 Gbps PMA Driven Mode (Manual bit lock) F =2 M =1 N = 20 Lanes Impedance 50 ohm IDLE RESET Register = Register + 1 Initialization All Registers initialized Init_Done RX_RESET CDR_FREQ_LOCK Maintaining its setup time CDR_PHASE_COARSE RX_RESET Refresh_Status Refresh Serdes Status READ SERDES_TEST_OUT Update Outputs CDR_PHASE_FINE 17

3. GBT-FPGA Overview in Igloo 2: 3. 1 The MGT block : SERDES Initialization & Control • Taking a look at Igloo 2_SERDES_APB_MANAGER: Registers Initialization SERDES_CLK = REF_CLK 1 EPCS Mode @ 4. 8 Gbps PMA Driven Mode (Manual bit lock) F =2 M =1 N = 20 Lanes Impedance 50 ohm IDLE RESET Register = Register + 1 Initialization : Caution All Registers initialized ream t S t ! i ! b e r e to th ing he k k r c o o l w t ’ e n es ontinu S do. Init_Done c E o D ation? t r R e d E p S e o e e Refresh t n h e I es nt t R e _ X m o R RX_RESET , Serdes Status m e Refresh_Status For the ation sequenc. RX_RESET liz • Initia CDR_FREQ_LOCK Maintaining its setup time CDR_PHASE_COARSE READ SERDES_TEST_OUT Update Outputs CDR_PHASE_FINE 18

3. GBT-FPGA Overview in Igloo 2: 3. 2 The GBT_TX & GBT_RX modifications done: • Changed the Xilinx IP by the Microsemi’s: – Core: Two-Port Large SRAM – Adapt code for ports names in: • TX: gbt_tx_gearbox_std_dpram. vhdl • RX: gbt_rx_gearbox_std_dpram. vhdl Igloo 2_txdpram_core Igloo 2_rxdpram_core 19

3. GBT-FPGA Overview in Igloo 2: 3. 3 Surrounding Infrastructure: From GBT-FPGA examples • Pattern Generator: Generates Fixed or counts • GBT Bank Reset manager: Resets all in order • Error Detector: To check transmission • Pattern Match: For delay determination From me • SERDES_INIT_REGISTERS • PLLs for 240 to 40 Mhz: Word to Frame CLK • UART communication: • For debugging on the PC • Check transmission status • Perform modules control : • Resets, Pattern, Serdes Lane… From GBT_Bank: • Configuration files • gbt_bank_package • gbt_bank_user_setup • vendor_specific_gbt_bank_package 20

3. GBT-FPGA Overview in Igloo 2: 3. 4 Communication with PC: Through Igloo 2 UART Core Microsemi Provides Drivers for PC to emulate Serial port through USB: Makes life easier when accessing to the USB when programming an application. Microsemi Provides Core for FPGA to communicate with the UART (the board has a UART to USB interface) Only needed to develop a “translator” module that interfaces the CORE with the Fabric signals you are interested to control 21

4. Tests done so far in Igloo 2: 4. 1 Testing GBT_TX & GBT_RX modified modules • • • It was needed to verify that the TX & RX modules work well with the modifications done. Dual Port Rams Xilinx IPs substituted by Microsemi Ips (GBT-FPGA STD Version) Design unconstrained up to now. (Improvements are specked) 84 Bits @ 40 MHz 20 Bits @ 240 MHz 84 Bits @ 40 MHz GBT Frame: Data visible to GBT-TX and from GBT-RX 84 bits @ 40 MHz 22

4. Tests done so far in Igloo 2: 4. 1 Testing GBT_TX & GBT_RX modified modules • • • It was needed to verify that the TX & RX modules work well with the modifications done. Dual Port Rams Xilinx IPs substituted by Microsemi Ips (GBT-FPGA STD Version) Design unconstrained up to now. (Improvements are spected) 84 Bits @ 40 MHz 20 Bits @ 240 MHz 84 Bits @ 40 MHz 23

4. Tests done so far in Igloo 2: 4. 1 Testing GBT_TX & GBT_RX modified modules • • TX & RX Frame CLK: Comes from the same FRAME_CLK (40 MHz) TX & RX Word CLK: Comes from the same WORD_CLK (240 MHz) Both clocks are artificially injected in the Test. Bench. The TX & RX frames are well recovered with a delay ~ 320 ns. 24

4. Tests done so far in Igloo 2: 4. 2 Let’s connect all together GBT_TX, GBT_MGT & GBT_RX 84 Bits @ 40 MHz 20 Bits @ 240 MHz 1 Bit @ 4. 8 GHz 20 Bits @ 240 MHz 84 Bits @ 40 MHz Simulation with static Frame: 0 x 0000 BABEAC 1 DADCDCFFFF Static frame well recovered!! 25

4. Tests done so far in Igloo 2: 4. 2 Let’s connect all together GBT_TX, GBT_MGT & GBT_RX 84 Bits @ 40 MHz 20 Bits @ 240 MHz 1 Bit @ 4. 8 GHz 20 Bits @ 240 MHz 84 Bits @ 40 MHz Simulation with dynamic Frame: Segmented counter Delay ~ 314 ns (Non deterministic, STD Version) Dynamic frame well detected!! We see the TX & RX Flags for delay determination 26

4. Tests done so far in Igloo 2: 4. 3 Making our own delay measurements By using: • • • Segmented count for transmission Pattern detected flags for reference Serdes Lane 1 (looped on the board) Delay : • • 320 ns (Fits the simulations) Non deterministic (Each start-up is different) Test Conditions: • • • Design Unconstrained Synplify parameters as default Only SERDES manual initialization 27

4. Tests done so far in Igloo 2: 4. 3 Comparing data with specked results from LATOP version 130. 3 ns Slide from Manoel Barros Marin 28

4. Tests done so far in Igloo 2: 4. 4 Interfacing and controlling transmission with PC Observations and status: • • For the moment when 1 FPGA is looped works well with Lane 1 as well as with Lane 2 (SMA cables) Not yet working with 2 FPGA (Synchronization issues? ) The bit lock is not yet reached in any of the configurations. Needed to study carefully synthesis constraints and optimizations: • The GBT_RX part seems to have some issues here… 29

5. Summary: • • GBT Implementation Status about to be ready on Igloo 2. This is not official & non supported modified GBT-FPGA code, based on STD version. We aim as well to recover the TTC clock from the optical lines with the SERDES CDR circuit. We have to specify the clocking scheme of our Front-End – Local Oscillator VS Dedicated clock line @ 40 Mhz Beam synchronous. • • • Still many things to understand/improve Any contribution/collaboration is more than welcome. The code needs to be cleaned and structured properly. 30

31

Manual APB Master & ROM For SERDES Configuration and Initialization: It works!! (Post-synthesis sim) Tx & RX @ 4. 8 Gbps T-Bank !! ble for GB ila va a ’s LL P 0 1 0 G 2 M o Now we have the Tw

Igloo 2 SERDES Testing (Tx part) Different Speeds & Configurations Scope not for eye diagram determination BW 1 Ghz, used just for reference 2* Signal Freq = Bit rate Transmission pattern “ 1010101010” EPCS @ 1. 25 Gbps Pre-Configured **EPCS : External Physical Coding Sublayer EPCS @ 2. 5 Gbps Pre-Configured EPCS @ 4. 8 Gbps!! Custom Parameters

Igloo 2 SERDES Testing (EPCS-4. 8 Gbps) Looping the lines, TX & RX simulation and start-up sequence

Igloo 2 SERDES Testing (TX & RX @ EPCS 4. 8 GBPS) Working with the Dev. Board and the means it provides Parallel Input Data (20 bits): “ 1111 0000 1111 XXXX” Rx Data. Tx_0 Data. Tx_1 … Data. Tx_19 Frame. Pos Recovered Parallel Received Data (20 bits): “ 11110 00011 11111 1 XXXX” SERDES Tx Frame. Pos 3 2 1 0 Data. Rx_1 … Data. Rx_19 3 2 1 0 11111 111 XX XX 111 10000