White Rabbit SubNanosecond timing over Ethernet Oh dear
White Rabbit Sub-Nanosecond timing over Ethernet “Oh dear! I shall be too late!” 1) 1) H. Z. Peek Nikhef Amsterdam Electronics. Technology Alice in Wonderland, Lewis Carroll (1865) H. Z. Peek. . . on behalf of the White Rabbit collaboration VLVn. T 11 Erlangen 12 -14, October 2011 1
Outline What is White Rabbit Network Building blocks Measurement results Applications Conclusion H. Z. Peek Nikhef Amsterdam Electronics. Technology VLVn. T 11 Erlangen 12 -14, October 2011 2
What is White Rabbit Networking and timing Based on well-known technologies / standards Open Hardware and Open Software International collaboration Main features: Transparent, H. Z. Peek Nikhef Amsterdam Electronics. Technology high-accuracy time distribution, Low-latency, deterministic data delivery, Designed for high reliability, Plug & play. VLVn. T 11 Erlangen 12 -14, October 2011 3
Timing over Ethernet History 1985 NTP (RFC 1305) 2002 PTP (IEEE 1588) 2012 White Rabbit (? ) sub milli-second sub micro-second sub nano-second H. Z. Peek Nikhef Amsterdam Electronics. Technology VLVn. T 11 Erlangen 12 -14, October 2011 4
White Rabbit timing distribution Synchronization with sub-ns accuracy over fiber A combination of: Synchronous Ethernet (Sync. E) syntonization Digital Dual-Mixer Time Difference (DDMTD) phase detection (j) Precision Timing Protocol (PTP) synchronization H. Z. Peek Nikhef Amsterdam Electronics. Technology VLVn. T 11 Erlangen 12 -14, October 2011 5
Timing distribution Reference Clock Tx Master SFP Rx j Ordinary serial data communication High Precision Low jitter channels! channels Tx SFP j Averaging • H. Z. Peek Nikhef Amsterdam Electronics. Technology Slave Rx SFP Tx Slave High Precision Low jitter Rx SFP Rx Tx all clock edges “mean phase” Syntonization • “The adjustment of two electronic circuits or devices in terms of frequency” VLVn. T 11 Erlangen 12 -14, October 2011 6
PTP (IEEE 1588) t 1 Tx SFP t 4 Ref Clk Master Cnt Slave t 1 t 2 Rx t 4 SFP Rx Tx j t 1 t 3 j Cnt =0 t 2 t 4 t 3 toffset t 1 Time Stamp Data (Packets) Clock Follow_Up Message (t 1) t 4 H. Z. Peek Nikhef Amsterdam Electronics. Technology Sync message (= Cnt + Master. Bit. Slide + j) 1. t 2 –t 1=offset + MSdelay 2. t 4 –t 3=-offset + SMdelay 3. MSdelay=SMdelay Delay_Req Message Delay_Response Message (t 4) Master Clock Time Slave Clock Time t 2 (= Cnt + Slave. Bit. Slide) t 1 t 2 t 3 t 1 t 2 t 4 t 3 t 1 t 2 VLVn. T 11 Erlangen 12 -14, October 2011 7
White Rabbit network H. Z. Peek Nikhef Amsterdam Electronics. Technology VLVn. T 11 Erlangen 12 -14, October 2011 8
White Rabbit Switch V 2 Central H. Z. Peek Nikhef Amsterdam Electronics. Technology element of White Rabbit network Fully custom design, designed from scratch 10 SFP ports (1000 Base-LX) Capable of driving long distance Single Mode fiber VLVn. T 11 Erlangen 12 -14, October 2011 9
White Rabbit Node Simple PCIe FMC Carrier (SPEC) Currently available H. Z. Peek Nikhef Amsterdam Electronics. Technology http: //www. ohwr. org/projects/spec/wiki VLVn. T 11 Erlangen 12 -14, October 2011 10
White Rabbit Switch V 3 2 H. Z. Peek Nikhef Amsterdam Electronics. Technology uplink ports, 16 downlink ports Hardware just assembled. Hardware & Software currently being tested. Production expected Q 1 -2012 http: //www. ohwr. org/attachments/741/White_Rabbit_Technical_Spec_05. pdf VLVn. T 11 Erlangen 12 -14, October 2011 11
Measurement test setup H. Z. Peek Nikhef Amsterdam Electronics. Technology Hot-air gun demo: http: //www. youtube. com/watch? v=ZSRQEExb dq 8 VLVn. T 11 Erlangen 12 -14, October 2011 12
Measurement results H. Z. Peek Nikhef Amsterdam Electronics. Technology VLVn. T 11 Erlangen 12 -14, October 2011 13
Applications Digital to Time Converter (DTC) = Alarm Clock Time to Digital Converter (TDC) = Time stamping H. Z. Peek Nikhef Amsterdam Electronics. Technology VLVn. T 11 Erlangen 12 -14, October 2011 14
Conclusions White H. Z. Peek Nikhef Amsterdam Electronics. Technology Rabbit enables measurement and control applications which are using distributed system technologies. Such applications may be spread over large distances. Data transmission delay changes are continuously measured and compensated. System timing of White Rabbit nodes are synchronized with high precision. VLVn. T 11 Erlangen 12 -14, October 2011 15
White Rabbit a multi-laboratory, multi-company effort H. Z. Peek Nikhef Amsterdam Electronics. Technology Thank you Coming Soon! VLVn. T 11 Erlangen 12 -14, October 2011 16
H. Z. Peek Nikhef Amsterdam Electronics. Technology VLVn. T 11 Erlangen 12 -14, October 2011 17
How do we know the time offset between Master and Slave? Toffset = Total delay(Master>Slave>Master) / 2 Can we measure propagation delay using existing serial communication channels? Serial Communication Coding Properties: 1 Clock & Data coded into one H. Z. Peek Nikhef Amsterdam Electronics. Technology stream DC-Balance 3 Special code-groups / Word Alignment 2 VLVn. T 11 Erlangen 12 -14, October 2011 18
Measure propagation delay using FPGA Ser. Des technology 1 3 Regenerate system clock at the receive Using a barrel shifter for word alignmen delay known with bit clock resolution Transmitter Lattice x 20 System Clock Receiver Xilinx LFSCM 25 Virtex-5 Ser. Des SFP H. Z. Peek Nikhef Amsterdam Electronics. Technology System Clock Lattice SC PCI Expressx 1 Evaluation Board Start Xilinx ML 507 Board SFP 100 Km Fiber Ser. Des LEDs Stop VLVn. T 11 Erlangen 12 -14, October 2011 19
Time offset measurement test setup 10 Km fiber Stop Loopback the recovered clock with a Voltage Controled Xtal Oscillator Clock Loopbac k (DPLL) DAC Start Master H. Z. Peek Nikhef Amsterdam Electronics. Technology Slave VCXO VLVn. T 11 Erlangen 12 -14, October 2011 20
Resynchronization + Barrel shifting action 0111010110000010101101110101 3 10 Rx. Rec. Clk Bit. Slide(4: 0) 0 H. Z. Peek Nikhef Amsterdam Electronics. Technology 0001 = 3 0000 0011 1 0 Unit Interval (UI) Start/Stop delay Algorithm: Propagation Delay = “Start-Stop” Delay + “LED Value” * UI Details : VLVn. T 09, October 15, 2009 in Athens • Presentation: http: //www. nestor. noa. gr/vlvnt 09/pres/Jansweijer_Meauring. Propagation. Delay. ppt • Paper: http: //dx. doi. org/10. 1016/j. nima. 2010. 04. 126 VLVn. T 11 Erlangen 12 -14, October 2011 21
Measure time offset Bidirectional + Loopback the recovered Stop Clk Start clock 1. 25 Gbps (IEEE 802. 3 1000 BASE-X = Gigabit Ethernet) Master Reference Time offset is. Tx. Usr. Clk determined by: Slave Rx Rx. Usr. Clk Tx Count H. Z. Peek Nikhef Amsterdam Electronics. Technology 1. 2. 3. Rx. Usr. Clk j SFP Rx Fine time Coarse time Tx toffset # of system clocks # bit clocks (i. e. barrel shifts) Phase between Master node Tx and Rx clock Tx. Usr. Clk 1. 25 Gpbs 20 x #bit clocks 800 ps 10. . 100 ps For details please see Technical Report “ETR 2010 -01”: http: //www. nikhef. nl/pub/services/biblio/technicalreports/ETR 2010 -01. pdf VLVn. T 11 Erlangen 12 -14, October 2011 22
Time offset and fibre dispersion Master time Slave time toffset l 1 dm. Tx ds. Rx tpdl 1 l 2 dm. Rx H. Z. Peek Nikhef Amsterdam Electronics. Technology tdisp tpdl 1 ds. Tx (tdisp = 15150 ps over 10. 7 km => D(l) = l 416 ps/km) VLVn. T 11 Erlangen 12 -14, October 2011 23
- Slides: 23