RPC Link System Status and Plan RPC Workshop

  • Slides: 42
Download presentation
RPC Link System Status and Plan (RPC Workshop) Vahid Amoozegar, Mohammad Ebrahimi, Roghayeh Ghasemi,

RPC Link System Status and Plan (RPC Workshop) Vahid Amoozegar, Mohammad Ebrahimi, Roghayeh Ghasemi, Elham Zareiyan Behzad Boghrati, Mojtaba Mohammadi Najaf Abadi Institute for Research in Fundamental Science (IPM) on behalf of CMS Muon Group 31 Aug. – 1 Sep 2020 Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 1

Link System Phase-II Upgrade Schedule Today Current phase Today Behzad Boghrati, RPC Link System

Link System Phase-II Upgrade Schedule Today Current phase Today Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 2

RPC Link system upgrade schedule Today Project Status: P O SH 0 K R

RPC Link system upgrade schedule Today Project Status: P O SH 0 K R 02 O W h, 2 C P 26 t R ST 25 A L un J 1. Current Phase: Initial Validation of Final Prototype in IPM 2. All Basic tests are passed successfully. 3. Our goals regard to the HM 2 are achieved. (HM 2: Initial Validation of Final Prototype, 2020/06/17 ) 4. We are in time with respect to schedule Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 3

Where we are now and What’s Next? ü HM-1 P O SH 0 K

Where we are now and What’s Next? ü HM-1 P O SH 0 K R 02 O W h, 2 C P 26 t R ST 25 A L un J Today. Current phase is Completed ü HM-2 Next Step Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 LB x 2 (2020 -07 -06) 4

Where we are now and What’s Next? ü HM-1 Today. Current phase is Completed

Where we are now and What’s Next? ü HM-1 Today. Current phase is Completed ü HM-2 ü LB x 2 (2020 -07 -06) We are here (Sep. 1 st) Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 5

Where we are now and What’s Next? ü For the time being prototype of

Where we are now and What’s Next? ü For the time being prototype of the Link board is completely functional ü In middle of July, in total we will have 3 LBs ü Internal Milestone (1 st Oct. ) 1. Integrate Firmware into a unique and concrete core 2. Controller Layer 3. Test the slow command List 4. Fixed Latency data Link 5. Data alignment and Clock Distribution ü BX Alignment ü Clock distribution and stability (short/long term stability) ü CLOCK Phase correction CB-Emulator Slow Controller / BEE Master Clock Ethernet DB Fiber 5 ~120 m LB Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 PRC FEB 6

Link System GUI (C++) - Main Page LB Timing Window Ethernet Setting Fast Command

Link System GUI (C++) - Main Page LB Timing Window Ethernet Setting Fast Command Diagnostic RPC FEB Radiation Mitigation Engine Status Window Link Board Control Board Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 7

Link System GUI (C++) - RPC Strip data logging LB Timing Window Ethernet Setting

Link System GUI (C++) - RPC Strip data logging LB Timing Window Ethernet Setting Fast Command Diagnostic RPC FEB Radiation Mitigation Engine Status Window Link Board Control Board Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 8

Link System GUI (C++) - Diagnostic LB Timing Window Ethernet Setting Fast Command Diagnostic

Link System GUI (C++) - Diagnostic LB Timing Window Ethernet Setting Fast Command Diagnostic RPC FEB Radiation Mitigation Engine Status Window Link Board Control Board Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 9

Summary • Three Link Boards have been produced and Tested. • Preliminary tests and

Summary • Three Link Boards have been produced and Tested. • Preliminary tests and validation are achieved. • FPGA Firmware integration is ongoing. • Parallel developing of the FPGA control layer and the PC GUI interface is ongoing. We expected these part completed around Oct. -Nov. 2020. • Data taking from the RPC chamber will be started around Nov. -Dec. 2020 and Radiation Tests. • Study and Preparation for the radiation tests is ongoing and will be reported on the other talk, today. Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 10

 • Thank you! Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop,

• Thank you! Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 11

Backup Slides Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug

Backup Slides Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 12

Outline Link System Upgrade Motivation New Link System Overview Radiation Consideration Data Rates at

Outline Link System Upgrade Motivation New Link System Overview Radiation Consideration Data Rates at barrel and Endcap Region RPC Layer-1 trigger Architecture and Link system Hits Data Frame Format • Control, Diagnostic and Synchronization of New Link System using New Slow Controller • Summary • • • Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 13

Link System Upgrade Motivation Present Link Board System Control Board LVDS cables Link Board

Link System Upgrade Motivation Present Link Board System Control Board LVDS cables Link Board Control & diagnostic (CCU chain) 40 MHz Synchronization Unit & LMUX 1376 Link Boards in 108 Boxes, Steered by 216 Control Boards FEB FEB Optic Links 90 m @ 1. 6 GHz 492 fibers (before splitting) • RPC signals synchronization, timing resolution is 25 ns RPC LB Upgrade • Data transmission speed is about 1. 6 Gbps • Control, diagnostic and monitoring of the Link system has been designed based on CCU ring (combination of copper cable and fiber optic), very susceptible to electromagnetic interference • CCU ring is not very fast, the bandwidth (40 MHz) share between 12 control boards • Most radiation hard electronic components are obsoleted • Electronic aging, presently the Link system at the end of LS 2 is already 13 years old Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 14

Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1

Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 15

New Link System Overview New Link system Features : 1. 2. 3. 4. FPGAs

New Link System Overview New Link system Features : 1. 2. 3. 4. FPGAs are KINTEX-7, XC 7 K 160 T – Industrial Version Muon hit time, TDC timing Resolution : 1. 56 ns Master Link board output data rate : 2. 56/5. 12/10. 24 Gbps Control Board communication with slow controller at 2. 56/5. 12/10. 24 Gbps 5. Embedded internal buffer (DDR 3) : 4 GByte 6. Radiation Mitigation: TMR + Internal Scrubbing • Scrub Rate of entire FPGA (Real time SEU detection and Correction) : 13 ms (31, 770 times faster than the rate of SEU at the Balcony) SEU at the Balcony : Every 413000 ms • • Over & Under Voltage Protection FPGA Over Temperature Protection Transient Voltage Suppressor ESD Protection (15 k. V) • 7. Safety Systems: Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 16

New Link Board Hardware Architecture • The design is based on the TDR baseline

New Link Board Hardware Architecture • The design is based on the TDR baseline and on following specifications and components: 1. 2. 3. 4. 5. 6. 7. 8. 9. FPGAs are KINTEX 7, XC 7 K 160 T-2 FFG 676 I. Fully support 10. 3 Gbps data transmission. FPGA Chip is equipped with heat sink. High resolution TDCs (at the steps of 1. 56 ns) implemented into the Kintex 7 FPGA Two Redundant optical data transmission line at the data rate of 10. 24 Gbps Ethernet link for the debugging and onboard JTAG programmer 256 M x 40 bits of SDRAM-DDR 3 for data buffering and debugging Data transmission with adjacent slave link boards through the SAMTEC back-plane type connector with bandwidth of 16 GHz and special front panel PCB board (Roger type). Radiation Mitigation is based on Triple Modular Redundancy (TMR) techniques and Soft Error Mitigation (SEM) IP core from Xilinx. Voltage Regulators selected from low dropout linear regulator families which has been tested already under radiation. Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 17

Hardware Prototype of New Link Board External SPI FLASH External ADC LV Linear Regulator

Hardware Prototype of New Link Board External SPI FLASH External ADC LV Linear Regulator Units Power-up Sequencing RPC INPUT SIGNALS X 96 LVDS Front Panel LED Indicators USB JTAG/UART Ethernet 1 G SFP+ (10 G) GTX BUS Low speed BUS Jitter Cleaner Clock Distribution Unit 256 M x 40 bits DDR 3 KINTEX-7 LVDS(XC 7 K 160 T-2 I) LVCMOS LV Protection Behzad Boghrati, RPC Link System Status and. Converters Plan, RPC Workshop, 31 Aug - 1 Sep 2020 18

New Link System Firmware List • New Link system Firmware TDC Calibration test setup

New Link System Firmware List • New Link system Firmware TDC Calibration test setup Fixed Latency GTX @ 10. 24 Gbps 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. Control Software TTC Generator TX - Lp. GBT Protocol (Xilinx) FEB Controller TDC (1. 56 ns) TTC Decoder Frame Generator TTC Clock Recovery & Phase Shift Multi Boot Remote Programming Control/Diagnostic/ Histogramming (LB) Ethernet LAN DDR 3 interface Controller Jitter Cleaner (State Machine Controller) Fixed Latency data transmission Control/Diagnostic (CB) Soft Error Mitigation Engine (SEM) Triple Modular Redundancy (TMR) TDC Performance measurement FEB Communication and Control Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 19

Time to Digital Converter Performance TDC 1. 56 ns @ 160 MHz / 640

Time to Digital Converter Performance TDC 1. 56 ns @ 160 MHz / 640 MSPS TDC Calibration test setup TDC Performance measurement • • ISERDES Based TDC FREQ. 160 MHz Sampling Rare 640 MSPS TDC Resolution 1. 56 ns Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 • DNL < ± 0. 006 LSB • INL max < 0. 01 LSB 20

High Speed Data Transmission – Optical Link Fixed Latency GTX @ 10. 24 Gbps

High Speed Data Transmission – Optical Link Fixed Latency GTX @ 10. 24 Gbps • Fiber Optic Length : 120 m Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep • OM 2 (Multi-Mode) 2020 • Maximum achievable Opening eye for the Ideal Case (Loop back inside the FPGA 1 ) is 55. 56% • Opening eye for the Real Case in which the data transmission will take by connecting two FPGAs through the 120 meters optical channel is 50~55%. • Test Period is 48 hours and number of errors is Zero 21

Radiation Consideration • The Link system will be installed on the Balcony of CMS,

Radiation Consideration • The Link system will be installed on the Balcony of CMS, where the rates are even lower than what we have at the periphery of the detector. • Total Irradiation Dose is 0. 001 -10 Gy @ 3000 fb-1 • Neutron Flux at the CMS Balcony is 1 x 104 cm-2 s-1 @5 x 1034 cm-2 s-1 • Neutron Fluence for 10 HL-LHC years is 1 x 1012 cm-2 • The new Link board components has been chosen from COTS which are validated for radiation at the level of 300 Gy • The FPGA TID KINTEX-7 (XC 7 K 160 T) is 3400 -4500 Gy • Scrub Time of entire FPGA (Real time SEU detection and Correction) : 13 ms • The Single Event upset (SEU) rate on configuration memory is 1 SEU every 413 sec. and 1 SEU every 1695 sec. at Block RAM • TMR and Configuration Scrubbing will mitigate the SEUs • Irradiation test planned for Summer 2020 Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 22

Radiation Consideration Estimation of SEU on the KINTEX-7 XC 7 K 160 T Configuration

Radiation Consideration Estimation of SEU on the KINTEX-7 XC 7 K 160 T Configuration and BRAM Memories • Number of Errors = ϬCRAM × Flux × Tirrd × NCRAM • ϬCRAM = Cross section of the Single Event upset of each bit at the configuration Memory (cm 2 bit-1) • Flux = Number of Neutron pre squire centimeter per second • Tirrd = Irradiation Time • NCRAM = number of bits at the Configuration Memory SEU EVENT Integrated Errors due to SEU at the Configuration Memory of KINTEX-7 BLOCK RAM Memory of KINTEX-7 XC 7 K 160 T ϬCRAM = 4. 52 x 10 -15 cm 2 bit-1 Flux = 1 x 104 cm-2 s-1 Time of Exposure = 1 sec ϬBRAM = 5. 07 x 10 -15 cm 2 bit-1 Flux = 1 x 104 cm-2 s-1 Time of Exposure = 1 sec NCRAM = 53, 540, 576 bits NBRAM = 11, 700, 000 bits ECRAM = 0. 00242 at 1 second ECRAM = 0. 1452 at 1 minute ECRAM = 8. 712 at 1 hour EBRAM = 0. 00059 at 1 second EBRAM = 0. 036 at 1 minute EBRAM = 2. 135 at 1 hour 1 SEU every 413 seconds 1 SEU every 1694 seconds SEU Time Interval (413 Second) SEU Time Interval >> Scrubbing time Time (Second) Scrubbing time (13 ms) Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 23

MLB Single Hit Rate wrt Max expected Background Rate at HL-LHC in Barrel •

MLB Single Hit Rate wrt Max expected Background Rate at HL-LHC in Barrel • Expected Max hit rate in Barrel in HL-LHC , safety factor 3: 600 cm-2. s-1 • Biggest Strip surface area : 120 X 3 cm 2 • Average cluster size: • Number of Strip per chamber : 96 • Max Single hit in one LB per bx : 600 x (120 x 3) x 96 x 25 X 10 -9= 0. 52 hit /LB x bx • One Master Link Board collects hits of two adjacent Slave Link boards • Max Single hit in one MLB per bx : 0. 52 x 3 = 1. 56 hit /LB x bx • Size of hit information Package : 15 bits (Yellow box) • MLB Max. Payload: 15 bits x 1. 56 hits x 40 MHz = 0. 936 Gbps • We foresee to send 232 bits/BX (for physics, already removed EOD and checksums) between MLB and RPC backend. • Maximum limit of single hits in one MLB = (one frame – overheads ) / size of one hit = (256 – 46) / 15 = 14 hits Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 24

MLB Single Hit Rate wrt Max Expected Background Rate at HL-LHC in Endcap •

MLB Single Hit Rate wrt Max Expected Background Rate at HL-LHC in Endcap • Expected Max hit rate in Endcap in HL-LHC , safety factor 3: 700 cm-2. s-1 • Biggest Strip surface area : 66 X 3. 125 cm 2 • Average cluster size: • Number of Strip per chamber : 96 • Max Single hit in one LB per bx : 700 x (66 x 3. 125) x 96 x 25 X 109 = 0. 35 hit /LB x bx • One Master Link Board collects hits of two adjacent Slave Link boards • Max Single hit in one MLB per bx : 0. 35 x 3 = 1. 05 hit /LB x bx • Size of event : 15 bits (Yellow box) • MLB Max. Payload: 15 bits x 1. 05 hits x 40 MHz = 0. 63 Gbps • We foresee to send 232 bits/BX (for physics, already removed EOD and checksums) between CB and RPC backend. • Maximum limit of single hits in one CB = (one frame – overheads ) / size of one hit = (256 – 46) / 15 = 14 hits Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 25

RPC Layer-1 Trigger Architecture and New Link system Hits Data Frame Format Item Header

RPC Layer-1 Trigger Architecture and New Link system Hits Data Frame Format Item Header + FEC Number of Hits (1. . 14) No. Strip (1. . 96) Sub-bx LB No. … BCN BC 0 Partition delay End of Data Bit 24 4 7 4 4 … 12 1 3 2 • Each Frame has 256 bit per each BX = 10. 24 Gbps • Data Frame format consists of transceiver overheads, Number of hits, hit information package, timing information, and a flag which shows status of transmission of current bunch cross in the sense that it is finished or not. • Transceiver overheads : Header + FEC + End of data • Number of hits shows the number of hit information package which exist inside the current frame information • Hit information Package includes of the detail information of a hit such as fired strip number ( 1 to 96), timing information of a hit (sub-bx) which is a four bit number and shows the exact hitting time of fired strip with resolution of 1. 56 ns, and the number of link board as a location of fired strip corresponding to the RPC Chamber • Timing information : using for synchronization and timing alignment of data transmission and contains a bunch cross number (12 bits) and bunch cross zero (BC 0). • Partition delay: In the rare physical events, it could be possible the number of hits will not fit inside one frame. In such cases, the reminder of hits information will be send on the consequent frames. The partition delay shows the number of consequent frames. In such frames, the number of bunch cross counter (BNC) is the same as last frame. Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 26

Summary • Present Link system has been working well for more than 14 years

Summary • Present Link system has been working well for more than 14 years (since 2006 -2007) • New Link system improves the muon hit time to 1. 56 ns and using the high bandwidth of data transmission (10. 24 Gbps) will increase the speed of data taking of RPC chambers • In new link system, radiation mitigation is achieved by using the firmware TMR and internal scrubbing (SEM) • The data transmission bandwidth of the new link system will cover the HL-LHC muon rates (with safety factor of 3) • Muon hit date will send from link system through high speed data transmission optical link to the RPC Endcap back-end electronic and Barrel backend layer-1. • Synchronization, control and diagnostic of the New link system will be done by using the new Slow Controller Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 27

 • Thank you! Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop,

• Thank you! Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 28

Control, Diagnostic and Synchronization of New Link System using New Slow Controller RPC Online

Control, Diagnostic and Synchronization of New Link System using New Slow Controller RPC Online Software RPC Endcap Back-end / Barrel Muon Trigger Layer-1 Link board Box (LBB) New Slow Controller Slow Control Command Histogram Diagnostic Muon Rates New Control board New Link board Hits Downlink Uplink RX_CLK TTC Clock @40. 078 MHz Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 29

New Slow Controller Functions • Downlink, data direction from new Slow Controller to Control

New Slow Controller Functions • Downlink, data direction from new Slow Controller to Control boards: • Number Links : 216 Downlinks (96 links Endcap CBs + 120 links Barrel CBs) • Link Protocol: Lp. GBT Protocol • Data Rate: 10. 24/5. 12/2. 56 Gbps (All of this speeds are acceptable) • Fast trigger, Broadcast commands : • BC 0, EC 0, L 1 A, Hard Reset • TTC clocks (40. 078 MHz) • Slow control Commands 1. 2. 3. 4. FEB Threshold setting (received from RPC online software) TTC fine skew adjustment (received from RPC online software) Open/Closed windows setting (received from RPC online software) FPGA configuration file (Remote Programming) (received from RPC online software) • Uplink, data direction from Control boards to the new Slow Controller: • RPC strip hit Histogram • Counting Hits on every RPC strip • Timing Histograms • Counting Hits on all RPC strips in each Bunch Cross • Link System Status • Voltages, current, Temperature • Faults, SEU • Firmware Version Control Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 30

High speed data transmission Performance @ 10. 24 Gbps • Conceptually, we want the

High speed data transmission Performance @ 10. 24 Gbps • Conceptually, we want the eye to be as “open” as possible, as a larger eye opening implies that we have more margin to the voltage and timing requirements. The eye must be wide enough to provide adequate time to satisfy the setup and hold requirement of the receiver, and have sufficient height to ensure that the voltage levels meet vih and vil requirements in a system that may possess multiple sources of noise. This allows the receiver to resolve the input signals successfully into digital values. • Maximum achievable Opening eye for the Ideal Case (Loop back inside the FPGA 1 ) is 55. 56% and the Opening eye for the Real Case in which the data transmission will take by connecting two FPGAs through the 120 meters optical channel is 50~55%. • The Test Period is 48 hours and number of errors is Zero Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 Opening eye – Ideal case Opening eye – Real case 31

New Link system Hits Data Frame Format Item Header + FEC Number of Hits

New Link system Hits Data Frame Format Item Header + FEC Number of Hits (1. . 14) No. Strip (1. . 96) Sub-bx LB No. … BCN BC 0 Partition delay End of Data Bit 24 4 7 4 4 … 12 1 3 2 • Each Frame has 256 bit per each BX = 10. 24 Gbps • Data Frame format consists of transceiver overheads, Number of hits, hit information package, timing information, and a flag which shows status of transmission of current bunch cross in the sense that it is finished or not. • Transceiver overheads : Header + FEC + End of data • Number of hits shows the number of hit information package which exist inside the current frame information • Hit information Package includes of the detail information of a hit such as fired strip number ( 1 to 96), timing information of a hit (sub-bx) which is a four bit number and shows the exact hitting time of fired strip with resolution of 1. 56 ns, and the number of link board as a location of fired strip corresponding to the RPC Chamber • Timing information : using for synchronization and timing alignment of data transmission and contains a bunch cross number (12 bits) and bunch cross zero (BC 0). Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 32

New Link system Hits Data Frame Format • Item Header + FEC Number of

New Link system Hits Data Frame Format • Item Header + FEC Number of Hits (1. . 14) No. Strip (1. . 96) Sub-bx LB No. … BCN BC 0 Partition delay End of Data Bit 24 4 7 4 4 … 12 1 3 2 Partition delay: In the rare physical events, it could be possible the number of hits will not fit inside one frame. In such cases, the reminder of hits information will be send on the consequent frames. The partition delay shows the number of consequent frames. In such frames, the number of bunch cross counter (BNC) is the same as last frame. Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 33

The RPC Architecture and L 1 T Link 1 - CB, Barrel 120 Fiber

The RPC Architecture and L 1 T Link 1 - CB, Barrel 120 Fiber Type of Data Max. Link Data Rate (Gbps) Max. Payload per Link (Gbps) Number of Links per sector Max. Payload per Sector (Gbps) 240 TTC, SC, Diag, Mon 2. 56/ 5. 12/ 10. 24 - - - - Max. Payload (Gbps) 2 - CB, Endcap 96 192 TTC, SC, Diag, Mon 3 - MLB, Barrel 300 Hit 10. 24 1 5 5 300 4 - MLB Endcap 192 Hit 10. 24 0. 63 2 1. 26 121 Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 34

Slow Control, Downlink (BEE to CB) • Downlink, data direction from Back-end Electronics to

Slow Control, Downlink (BEE to CB) • Downlink, data direction from Back-end Electronics to Control boards: • 216 Advanced links (96 links Endcap CBs + 120 links Barrel CBs) • Lp. GBT Protocol @ 10. 24 Gbps Item Header + FEC [0. . 23] L 1 A [24] BC 0 [25] EC 0 [26] Hard Reset [27] Command Type [28. . 30] Command Value A [31. . 38] Bit 24 1 1 3 8 Command End of Value B Value C Data [39. . 54] [55. . 253] [254, 255] 16 199 2 Command Types: 1. 2. 3. 4. 5. 6. TYPE 0 : Operating Mode (Calibration, Link Board Setting, Control Board setting, Physics RUN, Standby ) TYPE 1 : FEB Parameter Setting / Verification TYPE 2 : TTC fine skew adjustment TYPE 3 : Coarse/Fine windows setting TYPE 4 : FPGA Remote Programming TYPE 5. . TYPE 7: Reserved Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 35

Slow Control, Uplink (CB to BEE) • Uplink, data direction from Control boards to

Slow Control, Uplink (CB to BEE) • Uplink, data direction from Control boards to Back-end Electronics : • • • 216 Uplinks (96 links Endcap CBs + 120 links Barrel CBs) Lp. GBT Protocol Downlink Speed: 10. 24/5. 12/2. 56 Gbps (All of this speeds are acceptable) FEC 5 At any moment, a copy of received command from BEE and corresponding data or acknowledge from CB will send back to the BEE Item Header + FEC [0. . 23] L 1 A [24] BC 0 [25] EC 0 [26] Hard Reset [27] Command Type [28. . 30] Command Value A [31. . 38] Bit 24 1 1 3 8 Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 Command End of Value B Value C Data [39. . 54] [55. . 253] [254, 255] 16 199 2 36

RPC strip hit Histogram • Counting Hits on every RPC strip • • On

RPC strip hit Histogram • Counting Hits on every RPC strip • • On each LBs two multichannel counters have been implemented Adjustable window and Full window The buffer size of each Multichannel counter is 96 x 32 bits In total at each CB more than 9 (LBs)x 2 (MCC) X 96 X 32 bits • Total Buffer Size = 178176 bits ~ 180 kbits • Reading from these buffers are not time critical Hit Rate Histogram produced by present Link boards Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 37

Timing Histograms • Counting Hits on all RPC strips in each Bunch Cross •

Timing Histograms • Counting Hits on all RPC strips in each Bunch Cross • • • On each LBs one 32 bit counter has been implemented At each BX, the content of last BX will store at the FIFO The size of FIFO is 128 x 32 bits In total at each CB timing histogram buffer 9 (LBs)x 1 (Timing Histogram) X 128 X 32 bits • Total Buffer Size = 10368 bits ~ 10. 4 kbits • Reading from these buffers are not time critical Hit Rate Histogram produced by present Link boards Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 38

Slow Control, Downlink (BEE to CB) • Downlink, data direction from Back-end Electronics to

Slow Control, Downlink (BEE to CB) • Downlink, data direction from Back-end Electronics to Control boards: Item Header + FEC [0. . 23] L 1 A [24] BC 0 [25] EC 0 [26] Hard Reset [27] Command Type [28. . 30] Command Value A [31. . 38] Bit 24 1 1 3 8 Command Types: Command End of Value B Value C Data [39. . 54] [55. . 253] [254, 255] 16 199 2 Command Value A_H [35. . 38] Command Value: TYPE 0 : Operating Mode 1. Value A_H [35. . 38] = 0; Calibration mode 2. Value A_H [35. . 38] = 1; Link Board Parameter Setting Command Value A_L [31. . 34] (Target board) 3. Value A_H [35. . 38] = 2; Control Board Parameter setting Value A_L [31. . 34] = 0; CB 4. Value A_H [35. . 38] = 3; Physics RUN Value A_L [31. . 34] = 1; LB 1 5. Value A_H [35. . 38] = 4; Standby Value A_L [31. . 34] = 2; LB 2 6. Value A_H [35. . 38] = 5; Histogram … 7. Value A_H [35. . 38] = 6; Diagnostic Value A_L [31. . 34] = 9; LB 9 8. Value A_H [35. . 38] = 7; Loopback 9. Value A_H [35. . 38] = 8; FEB Test 10. Value A_H Behzad Boghrati, RPC Link System Status and Plan, RPC[35. . 38] = 9. . 16 ; Are Reserved 39 Workshop, 31 Aug - 1 Sep 2020

Example-1 • Let assume that at BX = 23, three strips are fired as

Example-1 • Let assume that at BX = 23, three strips are fired as follow: • Hit-1 : Stripe No. 40, sub-bx = 1 , LB No. = 2 • Hit-2 : Stripe No. 63, sub-bx = 16 , LB No. = 5 • Hit-3 : Stripe No. 90, sub-bx = 7 , LB No. = 8 Item Header + FEC 24 bits Number of Hits (1. . 14) 4 bits Strip No. (1. . 9 6) 7 bits Sub-bx 4 bits data x 3 40 1 Total Number of Hits Hit-1 LB No. 4 bits No. Strip (1. . 96) 7 bits Sub-bx 4 bits 2 63 16 LB No. 4 bits No. Strip (1. . 96) 7 bits Sub-bx 4 bits LB No. 4 bits BCN 12 bits 5 90 7 8 23 Hit-2 Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 BC 0 1 bit Partitio n delay 3 bits End of Data 2 bits 0 0 x Hit-3 40

Example-2 • Let assume that at BX = 23, twenty strips are fired as

Example-2 • Let assume that at BX = 23, twenty strips are fired as follow: • Hit-1 : Stripe No. 40, sub-bx = 1 , LB No. = 2 • …. • Hit-20 : Stripe No. 90, sub-bx = 7 , LB No. = 8 Item Header + FEC 24 bits Number of Hits (1. . 14) 4 bits Strip No. (1. . 9 6) 7 bits Sub-bx 4 bits LB No. 4 bits data x 14 40 1 2 Item Header + FEC 24 bits Number of Hits (1. . 14) 4 bits Strip No. (1. . 96) 7 bits Sub-bx 4 bits LB No. 4 bits data x 6 x x x … No. Strip (1. . 96) 7 bits Sub-bx 4 bits LB No. 4 bits BCN 12 bits … x x x 23 … No. Strip (1. . 96) 7 bits Sub-bx 4 bits LB No. 4 bits BCN 12 bits … 90 7 8 23 Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 BC 0 1 bit Partition delay 3 bits End of Data 2 bits 0 0 x BC 0 1 bit Partition delay 3 bits End of Data 2 bits 0 1 x 41

CMS Phase-2 TCDS 2: Synchronization and Clock distribution baseline ATCA Crate Link System Serenity

CMS Phase-2 TCDS 2: Synchronization and Clock distribution baseline ATCA Crate Link System Serenity RPC-Online LHC Clock The link is stablished long time ago, and we are still working on the Link at the IPBUS-Ethernet Fixed Latency spect Link System Emulate BEE functions Downlink Kintex-7 EVB Uplink Behzad Boghrati, RPC Link System Status and Plan, RPC Workshop, 31 Aug - 1 Sep 2020 42