White Rabbit SubNanosecond timing over Ethernet Oh dear
White Rabbit Sub-Nanosecond timing over Ethernet “Oh dear! I shall be too late!” 1) 1) H. Z. Peek Nikhef Amsterdam Electronics. Technology Alice in Wonderland, Lewis Carroll (1865) H. Z. Peek. . . on behalf of the White Rabbit collaboration KM 3 Ne. T General Assembly Meeting 1
Outline What is White Rabbit Network Building blocks Measurement results Applications Conclusion H. Z. Peek Nikhef Amsterdam Electronics. Technology KM 3 Ne. T General Assembly Meeting 2
What is White Rabbit Networking and timing Based on well-known technologies / standards Open Hardware and Open Software International collaboration Main features: Transparent, H. Z. Peek Nikhef Amsterdam Electronics. Technology high-accuracy time distribution, Low-latency, deterministic data delivery, Designed for high reliability, Plug & play. KM 3 Ne. T General Assembly Meeting 3
Timing over Ethernet History 1985 NTP (RFC 1305) 2002 PTP (IEEE 1588) 2012 White Rabbit (? ) sub milli-second sub micro-second sub nano-second H. Z. Peek Nikhef Amsterdam Electronics. Technology KM 3 Ne. T General Assembly Meeting 4
White Rabbit timing distribution Synchronization with sub-ns accuracy over fiber A combination of: Synchronous Ethernet (Sync. E) syntonization Digital Dual-Mixer Time Difference (DDMTD) phase detection (j) Precision Timing Protocol (PTP) synchronization H. Z. Peek Nikhef Amsterdam Electronics. Technology KM 3 Ne. T General Assembly Meeting 5
Timing distribution Reference Clock Tx Master SFP Rx j Ordinary serial data communication High Precision Low jitter channels! channels Tx SFP j Averaging • H. Z. Peek Nikhef Amsterdam Electronics. Technology Slave Rx SFP Tx Slave High Precision Low jitter Rx SFP Rx Tx all clock edges “mean phase” Syntonization • “The adjustment of two electronic circuits or devices in terms of frequency” KM 3 Ne. T General Assembly Meeting 6
PTP (IEEE 1588) t 1 Tx SFP t 4 Ref Clk Master Cnt Slave t 1 t 2 Rx t 4 SFP Rx Tx j t 1 t 3 j Cnt =0 t 2 t 4 t 3 toffset t 1 Time Stamp Data (Packets) Clock Follow_Up Message (t 1) t 4 H. Z. Peek Nikhef Amsterdam Electronics. Technology Sync message (= Cnt + Master. Bit. Slide + j) 1. t 2 –t 1=offset + MSdelay 2. t 4 –t 3=-offset + SMdelay 3. MSdelay=SMdelay Delay_Req Message Delay_Response Message (t 4) Master Clock Time Slave Clock Time t 2 (= Cnt + Slave. Bit. Slide) t 1 t 2 t 3 t 1 t 2 t 4 t 3 t 1 t 2 KM 3 Ne. T General Assembly Meeting 7
White Rabbit network H. Z. Peek Nikhef Amsterdam Electronics. Technology KM 3 Ne. T General Assembly Meeting 8
White Rabbit Switch V 2 Central H. Z. Peek Nikhef Amsterdam Electronics. Technology element of White Rabbit network Fully custom design, designed from scratch 10 SFP ports (1000 Base-LX) Capable of driving long distance Single Mode fiber KM 3 Ne. T General Assembly Meeting 9
White Rabbit Node Simple PCIe FMC Carrier (SPEC) Currently available H. Z. Peek Nikhef Amsterdam Electronics. Technology http: //www. ohwr. org/projects/spec/wiki KM 3 Ne. T General Assembly Meeting 10
White Rabbit Switch V 3 2 H. Z. Peek Nikhef Amsterdam Electronics. Technology uplink ports, 16 downlink ports Hardware just assembled. Hardware & Software currently being tested. Production expected Q 1 -2012 http: //www. ohwr. org/attachments/741/White_Rabbit_Technical_Spec_05. pdf KM 3 Ne. T General Assembly Meeting 11
Measurement test setup H. Z. Peek Nikhef Amsterdam Electronics. Technology Hot-air gun demo: http: //www. youtube. com/watch? v=ZSRQEExb dq 8 KM 3 Ne. T General Assembly Meeting 12
Measurement results H. Z. Peek Nikhef Amsterdam Electronics. Technology KM 3 Ne. T General Assembly Meeting 13
Applications Digital to Time Converter (DTC) = Alarm Clock Time to Digital Converter (TDC) = Time stamping H. Z. Peek Nikhef Amsterdam Electronics. Technology KM 3 Ne. T General Assembly Meeting 14
Applications-II H. Z. Peek Nikhef Amsterdam Electronics. Technology KM 3 Ne. T General Assembly Meeting 15
Conclusions White H. Z. Peek Nikhef Amsterdam Electronics. Technology Rabbit enables measurement and control applications which are using distributed system technologies. Such applications may be spread over large distances. Data transmission delay changes are continuously measured and compensated. System timing of White Rabbit nodes are synchronized with high precision. KM 3 Ne. T General Assembly Meeting 16
Conclusions-II White Rabbit = Networking and timing Is ideally suited for Prototype H. Z. Peek Nikhef Amsterdam Electronics. Technology hard- soft-ware becomes available this year. Open hard- soft-ware sources available which enables tendering. Standardization effort (IEEE and ITU) is foreseen. KM 3 Ne. T General Assembly Meeting 17
White Rabbit a multi-laboratory, multi-company effort H. Z. Peek Nikhef Amsterdam Electronics. Technology Thank you Coming Soon! KM 3 Ne. T General Assembly Meeting 18
H. Z. Peek Nikhef Amsterdam Electronics. Technology KM 3 Ne. T General Assembly Meeting 19
How do we know the time offset between Master and Slave? Toffset = Total delay(Master>Slave>Master) / 2 Can we measure propagation delay using existing serial communication channels? Serial Communication Coding Properties: 1 Clock & Data coded into one H. Z. Peek Nikhef Amsterdam Electronics. Technology stream DC-Balance 3 Special code-groups / Word Alignment 2 KM 3 Ne. T General Assembly Meeting 20
Measure propagation delay using FPGA Ser. Des technology 1 3 Regenerate system clock at the receive Using a barrel shifter for word alignmen delay known with bit clock resolution Transmitter Lattice x 20 System Clock Receiver Xilinx LFSCM 25 Virtex-5 Ser. Des SFP H. Z. Peek Nikhef Amsterdam Electronics. Technology System Clock Lattice SC PCI Expressx 1 Evaluation Board Start Xilinx ML 507 Board SFP 100 Km Fiber Ser. Des LEDs Stop KM 3 Ne. T General Assembly Meeting 21
Time offset measurement test setup 10 Km fiber Stop Loopback the recovered clock with a Voltage Controled Xtal Oscillator Clock Loopbac k (DPLL) DAC Start Master H. Z. Peek Nikhef Amsterdam Electronics. Technology Slave VCXO KM 3 Ne. T General Assembly Meeting 22
Resynchronization + Barrel shifting action 0111010110000010101101110101 3 10 Rx. Rec. Clk Bit. Slide(4: 0) 0 H. Z. Peek Nikhef Amsterdam Electronics. Technology 0001 = 3 0000 0011 1 0 Unit Interval (UI) Start/Stop delay Algorithm: Propagation Delay = “Start-Stop” Delay + “LED Value” * UI Details : VLVn. T 09, October 15, 2009 in Athens • Presentation: http: //www. nestor. noa. gr/vlvnt 09/pres/Jansweijer_Meauring. Propagation. Delay. ppt • Paper: http: //dx. doi. org/10. 1016/j. nima. 2010. 04. 126 KM 3 Ne. T General Assembly Meeting 23
Measure time offset Bidirectional + Loopback the recovered Stop Clk Start clock 1. 25 Gbps (IEEE 802. 3 1000 BASE-X = Gigabit Ethernet) Master Reference Time offset is. Tx. Usr. Clk determined by: Slave Rx Rx. Usr. Clk Tx Count H. Z. Peek Nikhef Amsterdam Electronics. Technology 1. 2. 3. Rx. Usr. Clk j SFP Rx Fine time Coarse time Tx toffset # of system clocks # bit clocks (i. e. barrel shifts) Phase between Master node Tx and Rx clock Tx. Usr. Clk 1. 25 Gpbs 20 x #bit clocks 800 ps 10. . 100 ps For details please see Technical Report “ETR 2010 -01”: http: //www. nikhef. nl/pub/services/biblio/technicalreports/ETR 2010 -01. pdf KM 3 Ne. T General Assembly Meeting 24
Time offset and fibre dispersion Master time Slave time toffset l 1 dm. Tx ds. Rx tpdl 1 l 2 dm. Rx H. Z. Peek Nikhef Amsterdam Electronics. Technology tdisp tpdl 1 ds. Tx (tdisp = 15150 ps over 10. 7 km => D(l) = l 416 ps/km) KM 3 Ne. T General Assembly Meeting 25
- Slides: 25