UDT UDP based Data Transfer Yunhong Gu Robert

UDT: UDP based Data Transfer Yunhong Gu & Robert Grossman Laboratory for Advanced Computing University of Illinois at Chicago conference XX

Outline n n n Background UDT Protocol UDT Congestion Control Implementation/Simulation Results Summary 5/18/2021 conference XX 2

Background n Distributed data intensive applications over wide area optical networks: q n Transport protocol support: q n Grid computing, access of bulk scientific data, data mining, high resolution video, etc. Efficient and fair bandwidth utilization TCP does not work! 5/18/2021 conference XX 3

Trans-Atlantic TCP Performance n Chicago -> Amsterdam, 1 Gbps link capacity, 110 ms RTT, 100 seconds run time q q q n TCP: 5 Mbps @ default setting (64 KB buffer) TCP: 100 Mbps @ 12 MB buffer (=1 Gbps*110 ms) Parallel TCP: 800 Mbps @ 64 TCP concurrent flows, with each having 1 MB buffer Two concurrent TCP flows, 1 from Chicago to Amsterdam, 1 within Chicago local networks: q 5/18/2021 2 Mps vs. 940 Mbps! conference XX 4

Why TCP Fails n Discover/recover slow on high BDP links q n n n Increase 1 byte per RTT Drastic decrease in sending rate Fairness bias on longer RTT links More prone to link error in high BDP links B: throughout in packets per second, p: loss rate 5/18/2021 conference XX 5

Requirements to the New Protocol n FAST q n FAIR q n High utilization of the abundant bandwidth either with single or multiplexed connections Intra-protocol fairness, independent of RTT FRIENDLY q 5/18/2021 TCP compatibility conference XX 6

Use Scenarios n n Small number of sources shares abundant bandwidth Bulk data transfer q q 5/18/2021 Most of the packets can be packed in maximum segment size (MSS) in a UDT session MSS can be set up by applications and the optimal value is the path MTU conference XX 7

What’s UDT? n UDT: UDP based Data Transfer q q n Reliable, application level, duplex, transport protocol, over UDP with congestion control Implementation: Open source C++ library Two orthogonal parts q q 5/18/2021 The UDT protocol framework that can be implemented above UDP, with any suitable congestion control algorithms The UDT congestion control algorithm, which can be implemented in any transport protocols such as TCP conference XX 8

Packet Structure n Data Packet: q n Control Packet: q q n Header: 1 bit flag + 31 bit sequence number Header: 1 bit flag + 3 bit type + 12 bit reserved + 16 bit ACK seq. no. + (0 - 32 n)bit control info Type: ACK, ACK 2, NAK, Handshake, Keep-alive, and Shutdown Actual size of a UDT packet can be ascertained from UDP header 5/18/2021 conference XX 9

Data Packet 0 Packet Sequence Number User Data Payload § Flag Bit: 0 § UDT uses 31 -bit packet based sequence number, ranging from 0 to (231 - 1) § Sequence number may be wrapped if it exceeds the maximum available number 5/18/2021 conference XX 10

Control Packet 1 type reserved ACK Seq. No. Control Information Field § Flag Bit: 1 § type: 3 -bit § handshake (000), shutdown (101), keep-alive (001) § ACK (010), ACK 2 (110), NAK (011) § UDT uses sub-sequencing: each ACK and related ACK 2 are assigned a 16 -bit unique ACK sequence number 5/18/2021 conference XX 11

Acknowledgements n Selective acknowledgement (ACK) q q q n Generated at every constant interval to send back largest continuously received sequence number of data packets. The sender sends back an ACK 2 to the receiver for each ACK (sub-sequencing). Also carries RTT, packet arrival speed, and estimated link capacity. Explicit negative acknowledgement (NAK) q q q 5/18/2021 Generated as soon as loss is detected. Loss information may be resent if receiver has not received the retransmission after an increasing interval. Loss information is compressed in NAK. conference XX 12

Timing n UDT uses 4 timer: rate control, ACK, NAK, and retransmission timers; their period are RCTP, ATP, NTP, and RTP, respectively. q n Rate Control Timer: trigger rate control q n These timers use system time as origin, which is queried after each time bounded UDP receiving to check if any timer expires. RCTP = 0. 01 seconds ACK Timer: trigger acknowledgement q 5/18/2021 ATP = RCTP conference XX 13

Timing n NAK Timer: trigger negative acknowledgement q n NTP = RTT Retransmission Timer: trigger retransmission based on time-out and maintain connection status q 5/18/2021 RTP = (exp-count + 1) * RTT + ATP where exp-count is the number of continuous time -out conference XX 14

Timing n Packet sending period (STP) q q q 5/18/2021 It is a period used to schedule packet sending: no more than 1 packet can be sent in each period. Tuned by rate control High precision implementation using CPU clock cycles conference XX 15

UDT Architecture Pkt. Scheduling Timer Sender DATA ACK Recver ACK 2 ACK Timer NAK Timer Recver Retransmission Timer Rate Control Timer 5/18/2021 conference XX 16

Congestion Control n Rate based congestion control (Rate Control) q q q n RC tunes the packet sending period. RC is triggered periodically at the sender side. RC period is constant of 0. 01 seconds. Window based flow control (Flow Control) q q 5/18/2021 FC limits the number of unacknowledged packets. FC is triggered on each received ACK at the sender side. conference XX 17

Congestion Control Framework Sender buffer FC Window Last ACK New packet sending period Rate control tunes packet sending period FC window resizes and moves after an ACK is received 5/18/2021 conference XX 18

Rate Control n n AIMD: Increase parameter is related to link capacity and current sending rate; Decrease factor is 1/9, but not decrease for all loss events. Link capacity is probed by packet pair, which is sampled UDT data packets. q Every 16 th data packet and it successor packet are sent back to form a packet pair. … q 5/18/2021 … The receiver uses a median filter on the interval between the arrival times of each packet pair to estimate link capacity. conference XX 19

Rate Control 1. If loss rate is greater than 1%, do not increase; 2. Number of packets to be increased in next RCTP time is: where B is estimated link capacity, C is current sending rate. Both are in packets per second. MSS is the packet size in bytes. β = 1. 5 * 10 -6. 3. Recalculate packet sending period (STP). 5/18/2021 conference XX 20

Rate Control B = 10 Gbps, MSS = 1500 bytes C (Mbps) B - C (Mbps) Increase Param. (Pkts) [0, 9000) (1000, 10000] 10 [9000, 9900) (100, 1000] 1 [9900, 9990) (10, 100] 0. 1 [9990, 9999) (1, 10] 0. 01 [9999, 9999. 9) (0. 1, 1] 0. 001 9999. 9+ <0. 1 0. 00067 5/18/2021 conference XX 21

Rate Control Decrease sending rate by 1/9, (or equivalently, increase packet sending period by 1. 125), only if n Received an NAK, whose largest lost sequence number is greater than the largest sequence number when last decrease occurred; or The number of loss events since last decrease has exceeded a threshold, which increases exponentially and is reset when condition 1 is satisfied. 1. 2. No data will be sent out for the next RCTP time if a decrease occurs. n q 5/18/2021 Help to clear congestion. conference XX 22

Flow Control n n Similar to BDP W = W*0. 875 + AS*(RTT+ATP)*0. 125 AS is the packets arrival speed at receiver side. q q 5/18/2021 The receiver records the packet arrival intervals. AS is calculated from the average of latest 16 intervals after a median filter. It is carried back within ACK. conference XX 23

Slow Start n n n Flow window starts at 2 and increases to the number of acknowledged packets, until the sender receives an NAK or reaches the maximum window size, when slow start ends. Packet sending period is 0 during slow start phase and set to the packet arrival interval at the end of the phase. Slow start only occurs at the beginning of a UDT session. 5/18/2021 conference XX 24

Implementation: Performance 5/18/2021 conference XX 25

Implementation: Intra-protocol Fairness 5/18/2021 conference XX 26

Implementation: TCP Friendliness 5/18/2021 conference XX 27

Implementation: TCP Friendliness (cont. ) 5/18/2021 conference XX 28

Implementation: File Transfer 1 Gbps/15. 9 ms Canarie Disk R: 800 Mbps W: 550 Mbps 1 Gbps/110 ms Star. Light Disk R: 800 Mbps W: 500 Mbps SARA Disk R: 1300 Mbps W: 900 Mbps To Star. Light Canarie SARA From Star. Light 460 505 560 Canarie 440 502 - SARA 441 - 660 5/18/2021 conference XX 29

Simulation: UDT Throughput at Different Bandwidth and RTT 5/18/2021 conference XX 30

Simulation: Performance of Concurrent UDT Flows 5/18/2021 conference XX 31

Simulation: Intra-protocol Fairness 5/18/2021 conference XX 32

Simulation: RTT Independence 5/18/2021 conference XX 33

Simulation: TCP Friendliness 5/18/2021 conference XX 34

Simulation: Convergence/Stability 5/18/2021 conference XX 35

Simulation: Complex Scenario 100 10 100 50 Link capacity Mbps Node Drop. Tail Flow and its ID Flow ID Throughput (Mbps) 5/18/2021 1 2 89. 3 90. 0 3 4 5. 18 41. 7 conference XX 5 6 50. 8 4. 78 36

Simulation: Multi-bottleneck B A 200 x C X 0. 1 1 10 20 40 60 AB 198. 8 189. 2 180. 1 170. 9 152. 5 137. 6 AC 0. 098 0. 979 9. 955 19. 88 39. 46 57. 70 X 80 100 120 140 160 180 AB 108. 4 104. 6 100. 8 101. 3 100. 7 100. 3 AC 73. 49 92. 42 98. 47 98. 04 98. 65 99. 00 5/18/2021 conference XX 37

Summary n UDT Protocol q q n Application level upon UDP Selective acknowledgement / explicit negative acknowledgement UDT Congestion Control q Rate Control n n n q Flow Control n 5/18/2021 Bandwidth estimation for fast probing available bandwidth and fast recovery AIMD for fairness Constant rate control interval Dynamic flow window according to packet receiving speed conference XX 38

UDT Characters n n n Good use of available bandwidth Application level - no changes in router and operating system No manual tuning Fair and Friendly: intra-protocol fairness, TCP friendliness, and RTT independence. Open source 5/18/2021 conference XX 39

Thank You! LAC: www. lac. uic. edu UDT: sourceforge. net/projects/dataspace Internet Draft: draft-gg-udt-01. txt conference XX