New Common Project on Precision Timing High Precision

New Common Project on Precision Timing High Precision Timing Distribution for HL-LHC S. Baron, E. Mendes, HPTD Interest Group – ACES 2018

Outline • The High Precision Timing Distribution Project and Interest Group • A concrete example on potential outcome • Fixed phase transmission via FPGA-embedded TRx • Conclusion on practical aspects 24 April 2018 sophie. baron@cern. ch - ACES 2018 - CERN 2

Part 1: the HPTD project 24 April 2018 sophie. baron@cern. ch - ACES 2018 - CERN 3

New playing rules of the HL-LHC • High Pile-up expected for HL-LHC • From 20 (LHC nominal) to ~200 (HL-LHC) 24 April 2018 sophie. baron@cern. ch - ACES 2018 - CERN 4

Impact on luminous region Pile-up density for Baseline Pile-up density without Crab cavities Rogelio’s talk this morning https: //indico. cern. ch/event/681247/contributions/2926566/attachments/1638356/2614922/SLIDESlogo. pdf Pile-up density > 2 events/mm (for baseline) Challenge for vertices reconstruction based on 3 D 24 April 2018 sophie. baron@cern. ch - ACES 2018 - CERN 5

Going 4 D. . Josh Bendavid (CERN/LPC), Oct 2017 CMS Detector performance for HL-LHC • With <µ>=200, vertices reconstructed with 3 D fit might be merged and could be potentially separated using 4 D reconstruction • With enough time resolution, a 3 -5 x reduction in effective pileup density for track-primary vertex association can be expected (ie back to a pile-up of ~50) 24 April 2018 sophie. baron@cern. ch - ACES 2018 - CERN 6

Going 4 D… Adding specific Timing detectors for charged particles (see Lindsey’s talk) • MTD for CMS • HGTD for ATLAS A very challenging task by itself, but implicitly assuming an excellent clock distribution scheme… • High stability in time of a single link • High detector-wide stability • Precise tracking of clock propagation time 24 April 2018 sophie. baron@cern. ch - ACES 2018 - CERN 7

How excellent? • Extracted values from simulations implementing a Gaussian distribution of time • CMS/MIP key values are • ~30 ps rms timing capability on charged tracks • ≲ 15 ps rms on individual link • ATLAS/HGTD key values • ~30 ps rms timing capability on charged tracks • ≲ 10 ps rms on individual link So far, same order of magnitude from detector simulations …. now we should connect these detector simulations to timing hardware studies 24 April 2018 sophie. baron@cern. ch - ACES 2018 - CERN 8

Many questions about timing • Which level of jitter is acceptable? Where? • Which jitter types are we talking about? • Which equipment should we use to assess it? • How ‘fixed’ does a ‘fixed phase’ need to be? • Which architecture for Front-End clock distribution? • Which devices are the most performing ones for low jitter and fixed phase? • How can we compare the performance of different systems or devices? • … All these questions are COMMON to ATLAS and CMS (and ALICE, and LHCb) 24 April 2018 sophie. baron@cern. ch - ACES 2018 - CERN 9

To answer them properly … • …we need: • • • Close interaction with detectors’ designers and users Expertise in timing distribution and jitter analysis Expertise in FPGA & optoelectronics Access to all timing distribution systems Access to high-end equipment Resources to perform the investigations and report their results to the community Thanks to the opening of a HL-LHC position, CERN EP/ESE group launched a new project called ‘High Precision Timing Distribution’ (HPTD) – Eduardo Brandao De Souza Mendes 24 April 2018 sophie. baron@cern. ch - ACES 2018 - CERN 10

HPTD Mandate • 3 parallel activities & services proposed by the HPTD team Building up a Common Culture & Know-how 24 April 2018 Performing Targeted Studies to assist experiments in their technical choices sophie. baron@cern. ch - ACES 2018 - CERN Providing Guidance for systems characterization and performance assessment 11

24 April 2018 sophie. baron@cern. ch - ACES 2018 - CERN 12

Building up a Common Culture • Providing tools to share knowledge: • A public web page: https: //espace. cern. ch/High. Precision. Timing • Terminology, Figures of Merit, Methodologies, Literature • Reports, results • Links to existing solutions (see Jeroen’s talk) • An Interest Group: HPTD-interest-group@cern. ch • Open to any interested person from the community • Sharing experience (forum, newsletters) • feel free to subscribe via https: //e-groups. cern. ch • A Git. Lab group: https: //gitlab. cern. ch/HPTD • Useful S/W, F/W, H/W sources & references • Taking care of the common timing infrastructure: • Characterizing known common systems • Legacy TTC, TTC-PON, GBTx, Lp. GBT • Upgrading the RF-TTC common backbone system • From RF source to all experiments • Collaboration with the White Rabbit team and BE/RF 24 April 2018 sophie. baron@cern. ch - ACES 2018 - CERN 13

Performing Targeted Studies to assist experiments in their technical choices 24 April 2018 sophie. baron@cern. ch - ACES 2018 - CERN 14

Performing Targeted Studies • Very topical studies on subjects of common interest [Questions that we never have time to investigate …but which would be crucial to do the right choices] • • • How fixed can be the phase within FPGA-embedded TRx? (see second part of this presentation) How precisely can we monitor this phase? How is the phase relationship/stability across all TRx within one FPGA? And between different FPGAs on single motherboard fed by same reference clock? Can we relate jitter on high-speed serial link to jitter on recovered clock? How do optical link margins affect timing margins? Can we define a jitter-budget for the elements of a clock distribution chain? Skew jitter between end-node clocks on point-to-point versus point-to-multipoint distribution systems PLLs, FPGAs, Opto-devices selection guide How to optimize a specific PLL implementation for high performance? …. • Studies can be performed by the HPTD team…or by members of the interest group • The topics, priority list and agenda will be proposed and tuned using the inputs of the interest group 24 April 2018 sophie. baron@cern. ch - ACES 2018 - CERN 15

Providing Guidance for systems characterization and performance assessment 24 April 2018 sophie. baron@cern. ch - ACES 2018 - CERN 16

Providing Guidance to users • HPTD team welcomes users in the Timing Lab to characterize their hardware • High End instrumentation • • • >10 GHz Oscilloscopes >12 GHz Optical probes Clock generators with jitter injection capability Phase Noise Analyser Climatic Chamber • Experts guidance • Equipment used on a daily basis by ESE teams • In return, as a good practice, results or reports shall be made public on the sharepoint/interest group page. 24 April 2018 sophie. baron@cern. ch - ACES 2018 - CERN 17

Part 2: A first example of a targeted study How fixed can be the phase within FPGA-embedded TRx? How can we monitor it? (Study and report: Eduardo Brandao De Souza Mendes) 24 April 2018 sophie. baron@cern. ch - ACES 2018 - CERN 18

Clock Distribution over serial links Bunch Clock Tx Master Clock PHY Rx Node Clock Tx Node Clock PHY Rx End-Node Clock End-node BC signal Fixed and deterministic phase required The phase of a transceiver changes with reset • What can we achieve nowadays with the techniques learnt from the experience of developing ‘fixed-phase’ FPGA transceiver cores (e. g. GBT-FPGA, TTC-PON)? • How can we monitor it in real life? 24 April 2018 sophie. baron@cern. ch - ACES 2018 - CERN 19

Fixed and deterministic phase? • Topic raised by CMS to guide the architectural choice of their clock distribution system • Excellent starting point for a targeted study • The results presented here are details of a more global study ongoing on FPGA Tx and Rx stability, Phase drift monitoring techniques Global feedback monitoring, and general question about relevance of such a request, especially regarding physics phase monitoring based on collision monitoring. • See slides here • • 24 April 2018 sophie. baron@cern. ch - ACES 2018 - CERN 20

How Fixed is the serial stream wrt the ref clock? PLL Si 5344 refclk 320 MHz FPGA eval. board Tx@10. 24 Gb/s 320 MHz clock … 00001111… scope DSA 91204 A Phase (Delta time): Ref. Clk vs Tx ~600 k samples single shot - 2 ms acquisition window parameters: - standard deviation (TIE rms) - average 24 April 2018 sophie. baron@cern. ch - ACES 2018 - CERN 21

Transmitter path (no reset) • Average: no reset - stable measurement ~1. 2 ps 24 April 2018 sophie. baron@cern. ch - ACES 2018 - CERN 22

Transmitter path (reset at every acquisition) • Buffer-Bypass: a. k. a. fixed latency (technique used for GBT-FPGA, TTC-PON) • Average: reset at every acquisition ~25. 1 ps 24 April 2018 sophie. baron@cern. ch - ACES 2018 - CERN 23

Transmitter path – (solution? ) • Solution based on advanced proprietary features of Ultrascale+ GTH/GTY transceivers: source: https: //indico. cern. ch/event/598467/attachments/1475212/2284761/CERN_June_13_v_1_3. pdf 24 April 2018 sophie. baron@cern. ch - ACES 2018 - CERN 24

Transmitter path – (reset at every acquisition) • Results for first implementation (UI compensation): • Promising results which will be further verified for more channels/devices • Average: reset at every acquisition ~1. 3 ps 24 April 2018 sophie. baron@cern. ch - ACES 2018 - CERN 25

Implementing the phase monitoring • Obviously, We can’t monitor the phase directly on the serial stream 24 April 2018 sophie. baron@cern. ch - ACES 2018 - CERN 26

Implementing the phase monitoring • Ser. Des phase variations monitoring based on a clock-mirroring ie: refclk Delay Path ie: serial stream Phase Measurement DDMTD concept example Tx path → on going investigation → mirror paths are micro-paths inside the FPGA which will impact the measurement → DDMTD: https: //www. ohwr. org/projects/white-rabbit 24 April 2018 sophie. baron@cern. ch - ACES 2018 - CERN 27

Coming back to the HPTD project & the interest group… 24 April 2018 sophie. baron@cern. ch - ACES 2018 - CERN 28

Roles of HPTD team and Interest group HPTD TEAM INTEREST GROUP Common Culture Define framework Provide Reference documents Perform studies on common systems Animate group Prepare Newsletters Organise meetings Targeted Studies Propose Subjects Perform topical studies Publish reports Steer priorities Propose Subjects Perform & share topical studies Timing Lab Usage Welcome users in HPTD lab Provide Equipment and Expert Guidance Preliminary agreement on coming dates Publicly share studies results Provide Feedback Share experience, studies and designs MOTO: SHARING EXPERIENCE 24 April 2018 sophie. baron@cern. ch - ACES 2018 - CERN 29

Organisation • 4 meetings per year • • • March, June, September, December (Vidyo available) Share main studies/development results (from HPTD or other contributors) Tune priorities of forthcomming ones First one (Pre-meeting) was yesterday Indico category: https: //indico. cern. ch/category/10229/ • Public – see full study from Eduardo • Informal communication & exchange via the interest group mailing list • HPTD-interest-group@cern. ch • A newsletter sent twice a month to the interest group (starting May 15) • Concise and very topical • Open to any suggestion via email or on sharepoint • Sharepoint & Gitlab • Accessible to all • Members of the interest group are welcome to contribute • • 24 April 2018 https: //gitlab. cern. ch/HPTD https: //espace. cern. ch/High. Precision. Timing sophie. baron@cern. ch - ACES 2018 - CERN 30

QUESTIONS ? Thank you! https: //espace. cern. ch/High. Precision. Timing

Spare slides 24 April 2018 sophie. baron@cern. ch - ACES 2018 - CERN 32

Clock Distribution over serial links Bunch Clock Tx Master Clock Rx Feedback Master Clock 24 April 2018 PHY Rx Node Clock Tx Feedback Node Clock Rx Feedback Node Clock sophie. baron@cern. ch - ACES 2018 - CERN PHY Rx End-Node Clock End-node BC signal Tx Feedback End Node Clock 33

Receiver path • Which recovered clock in GTH/GTY transceivers? FPGA xcvr banks rx serial fabric FF FF … XCVR FF FF ODDR rxoutclk rxrecclk • Does it matter if it is cleaned by a PLL with 1 k. Hz bandwidth? • Repeatability in a programmable device? • On-going study for fixed-phase 24 April 2018 ref clock tx tx data (320 MHz) rxrecclk rxoutclk Integration Band 3. 4 ps 3. 5 ps 3. 6 ps 3. 9 ps 1 Hz - 1 k. Hz (rms rnd) 0. 6 ps 0. 8 ps 1. 2 ps 9. 1 ps 1 k. Hz – 10 MHz (rms rnd) sophie. baron@cern. ch - ACES 2018 - CERN