Shore station brainstorm Peter Jansweijer Nikhef Amsterdam Electronics

Shore station brainstorm Peter Jansweijer Nikhef Amsterdam Electronics. Technology September 11 -12, 2013 KM 3 Ne. T, CLBv 2 Workshop Valencia 1

Shore Station Broadcast brainstorm Broadcast Referenc e. Clock Start PTP Buffer t 4 Stop 1 t 4 Stop 2 t 4 Stop 3 t 4 Stop 4 Shore Station interface 4. Port-1 SFP Port-2 SFP Time Stamp t 4 j: DDMTD Rx Port-3 SFP DOM A Optical Network DOM B Rx Port-4 SFP DOM C Rx Port-5 SFP DOM D Rx Main Electrical Optical Cable Switch Routing Table (software? ) needs to be adjusted. PTP timestamps t 1 for all DOMs are equal and reside in outgoing port (needs firmware- or softwarechange or both) MAC Control-Level multicast MAC addresses; such as “pause frames” for flow control need to be handled correctly. Example: 1. Peter Jansweijer 2. Nikhef 3. Amsterdam Electronics. Technology Tx Time Stamp t 1 ◦ ◦ “DOM-B” request “Pause” = Okay (request over point to point link), but… “Shore station” request “Pause” may be problematic (request over broadcast link, all ports are stalled). Address single DOM? Other surprises? September 11 -12, 2013 KM 3 Ne. T, CLBv 2 Workshop Valencia 2

Other surprises: PTP t 1, t 4 timestamps over multiple switches One t 1 Many t 4 PTP t 1 Broadca st PTP t 4 DOM inputs Redistribution of PTP t 1 not only within one switch… 2. Routing PTP frames is an issue (layer-2; no IPv 4!)… Note that t 4 needs to be routed to the appropriate DOM via the broadcast channel. …over multiple switches (depending on detector broadcast network segmentation) This needs brainpower! 1. Peter Jansweijer Nikhef Amsterdam Electronics. Technology September 11 -12, 2013 KM 3 Ne. T, CLBv 2 Workshop Valencia 3

Brainstorm-1 10 Gbps COTS 1 Gbps WR PTP t 1 Broadca st PTP t 4 DOM inputs Peter Jansweijer Nikhef Amsterdam Electronics. Technology 1 Gbps WR 1. When layer-2 is not routable then distribute the timestamps using your custom protocol. 2. … kind of awkward! And it may consume switch ports September 11 -12, 2013 KM 3 Ne. T, CLBv 2 Workshop Valencia 4

Brainstorm-2 10 Gbps COTS 1 Gbps WR PTP t 1 Broadca st PTP t 4 DOM inputs 1. Peter Jansweijer Nikhef Amsterdam Electronics. Technology 2. 3. 1 Gbps WR Layer-2 is not routable, then… Wrap PTP t 1 in a layer-3 message and route this to the rest of the switches. Each timing input calculates timing servo action and passes this to the broadcasts port. September 11 -12, 2013 KM 3 Ne. T, CLBv 2 Workshop Valencia 5

Tempting thoughts (that shouldn’t be thought at this stage in the design cycle…) Peter Jansweijer Nikhef Amsterdam Electronics. Technology September 11 -12, 2013 KM 3 Ne. T, CLBv 2 Workshop Valencia 6

Current major troubles in the design Optical Peter Jansweijer Nikhef Amsterdam Electronics. Technology network with 80 channels @ 50 GHz is very tempting… (will it be possible? ) Broadcast networking is difficult to implement. Broadcast network increases “asymmetry” =>this complicates timing calibration. Each DOM has a different wavelength laser. This is not disastrous but requires more careful tracking of DOMs. September 11 -12, 2013 KM 3 Ne. T, CLBv 2 Workshop Valencia 7

Tempting idea: a WR-Switch in the DU (Paul Kooijman) 1 Gbps WR, 10 Gbps data 1 Gbps WR Shore Sub-Sea A Peter Jansweijer Nikhef Amsterdam Electronics. Technology tempting idea to put a WR switch in each DU! But it solves lots of problems! September 11 -12, 2013 KM 3 Ne. T, CLBv 2 Workshop Valencia 8

Tempting idea: a WR-Switch in the DU (Paul Kooijman) + Better bandwidth utilization per wavelength => + Less wavelengths needed => possibility for bidirectional network + Even with bidirectional network the optical network easier! + cost effective! Inexpensive SFPs in the string and a limited amount of expensive long haul SFPs (no needed EDFA’s 6 x plus! anymore) + No broadcast anymore! All point to point. Timing is distributed Only 2 minus as it is meant to be => “Normal network” + All DOMs can be the same (colorless) - More complexity sub-sea. 1 general - Shore station switch must handle 10 Gbps (WR worry! only available on 1 Gbps) September 11 -12, 2013 KM 3 Ne. T, CLBv 2 Workshop Valencia 9 Peter Jansweijer Nikhef Amsterdam Electronics. Technology One WR switch per DU (1 x 16 or 2 x 8 DOMs, later redesign 1 x 18 or 2 x 9 DOMs).

Tempting idea: distributed WRSwitch(es) in the CLB (Henk Peek) Peter Jansweijer Nikhef Amsterdam Electronics. Technology September 11 -12, 2013 KM 3 Ne. T, CLBv 2 Workshop Valencia 10

Tempting idea: distributed WRSwitch(es) in the CLB (Henk Peek) Peter Jansweijer Nikhef Amsterdam Electronics. Technology + Redundancy + Better bandwidth utilization per wavelength => + Less wavelengths needed => possibility for bidirectional network + Even with bidirectional network the optical network easier! + cost effective! Inexpensive SFPs in the string and a limited amount of expensive long haul SFPs (no needed EDFA’s anymore) + No broadcast anymore! All point to point. Timing is distributed as it is meant to be => “Normal network” 7 x plus! + All DOMs can be the same (colorless) - More complexity sub-sea. 3 x minus - Needs further CLB development (implement simple static switch functionality) - Shore station switch must handle 10 Gbps (WR only available on 1 Gbps) 1 general worry! - SFP reliability not proven! September 11 -12, 2013 KM 3 Ne. T, CLBv 2 Workshop Valencia 11

Conclusion on tempting thoughts These Peter Jansweijer Nikhef Amsterdam Electronics. Technology thoughts shouldn’t be thought at this stage in the design cycle… But please let us re-think before proceeding! We are designing an awkward system right now. If we take a step back now we may have an operational KM 3 Ne. T detector faster! ◦I know… This involves all ◦disciplines: Hardware Firmware ◦ ◦ Software (+ Networking) ◦ Mechanics ◦ September 11 -12, 2013 Optical network Power KM 3 Ne. T, CLBv 2 Workshop Valencia 12

Backup slides Peter Jansweijer Nikhef Amsterdam Electronics. Technology September 11 -12, 2013 KM 3 Ne. T, CLBv 2 Workshop Valencia 13

Bandwidth 7500 Peter Jansweijer Nikhef Amsterdam Electronics. Technology Hz hitrate/ PMT 48 bit/hit => 7500 * 48= 360. 000 bps/PMT 31 PMTs / DOM 360. 000 * 31 ~ 12 Mbps / DOM 18 DOMs / string 12 Mbps * 18 = 216 Mbps / string = 21% of a 1 Gbps channel September 11 -12, 2013 KM 3 Ne. T, CLBv 2 Workshop Valencia 14

WR I 2 C buses KC 705 FPGA clb_xwr_core Soft PLL FMC xwc_periph SFP_SDA SCL SDA H 30 C 10 SFP+MOD_DEF 1 B 28 C 18 SFP_MOD_DEF 2 K 21 L 21 IIC_SCL_MAIN IIC_SDA_MAIN U 49 PCA 954 8 Ch 1 (I 2 C mux) Peter Jansweijer Nikhef Amsterdam Electronics. Technology GPIO Set/Readback Register Bit 2: SCL Bit 3: SDA Bit 8: SFP_SCL Bit 9: SFP_SDA September 11 -12, 2013 C 30 SCL C 31 SDA FMC-J 22 SYSCON Regs SFP_SCL SFP SDA SCL EPRO M SDA KM 3 Ne. T, CLBv 2 Workshop Valencia 15