New troubleshooting and diagnostic tools in Tungsten Fabric
New troubleshooting and diagnostic tools in Tungsten Fabric dataplane - Kiran K N < kirankn@juniper. net> 1
DPDKInfo tool ● This is a troubleshooting and diagnostic tool for DPDK enabled v. Router in R 2008 ● Dpdkinfo tool reads the internal data structures and data from v. Router and DPDK library and displays them ● Available in contrail-tools container ● Usage: dpdkinfo [--help] --version|-v Show DPDK Version --bond|-b Show Master/Slave bond information --lacp|-l <all/conf> Show LACP information from DPDK --mempool|-m <all/<mempool-name>> Show Mempool information --stats|-n <eth> Show Stats information --xstats|-x < =all/ =0(Master)/ =1(Slave(0))/ =2(Slave(1))> Show Extended Stats information --lcore|-c Show Lcore information --app|-a Show App information Optional: --buffsz <value> Send output buffer size 2
DPDKInfo tool Bond information in linux style (cat /proc/net/bonding/bond 0) $ dpdkinfo -b No. of bond slaves: 1 Bonding Mode: 802. 3 AD Dynamic Link Aggregation Transmit Hash Policy: Layer 2+3 (Ethernet MAC + IP Addresses) transmit load balancing MII status: UP MII Link Speed: 2000 Mbps MII Polling Interval (ms): 10 Up Delay (ms): 0 Down Delay (ms): 0 Driver: net_bonding 802. 3 ad info : LACP Rate: slow Aggregator selection policy (ad_select): Stable System priority: 65535 System MAC address: 52: 54: 00: 46: 55: 22 Active Aggregator Info: Aggregator ID: 0 Number of ports: 1 Actor Key: 17 Partner Key: 14 Partner Mac Address: a 4: bf: 01: 6 b: 16: 20 Slave Interface(0): 0000: 02: 01. 0 Slave Interface Driver: net_e 1000_em MII status: UP MII Link Speed: 1000 Mbps Permanent HW addr: 52: 54: 00: 46: 55: 22 Aggregator ID: 0 Duplex: full Bond MAC addr: 52: 54: 00: 46: 55: 22 Details actor lacp pdu: system priority: 65535 system mac address: 52: 54: 00: 46: 55: 22 port key: 17 port priority: 255 port number: 1 port state: 61 (ACT AGG SYNC COL DIST ) Details partner lacp pdu: system priority: 65534 system mac address: a 4: bf: 01: 6 b: 16: 20 port key: 14 port priority: 65535 port number: 15 port state: 61 (ACT AGG SYNC COL DIST ) 3
DPDKInfo tool LACP configuration and statistics $ dpdkinfo --lacp all LACP Rate: slow Fast periodic (ms): 900 Slow periodic (ms): 29000 Short timeout (ms): 3000 Long timeout (ms): 90000 Aggregate wait timeout (ms): 2000 Tx period (ms): 500 Update timeout (ms): 100 Rx marker period (ms): 2000 Slave Interface(1): 0000: 02. 0 Details actor lacp pdu: port state: 61 (ACT AGG SYNC COL DIST ) Details partner lacp pdu: port state: 61 (ACT AGG SYNC COL DIST ) LACP Packet Statistics: Tx 0000: 02: 01. 0 4807 0000: 02. 0 4807 Rx 4644 Slave Interface(0): 0000: 02: 01. 0 Details actor lacp pdu: port state: 61 (ACT AGG SYNC COL DIST ) Details partner lacp pdu: port state: 61 (ACT AGG SYNC COL DIST ) 4
DPDKInfo tool Lcore to <Interface, queue> mapping $ dpdkinfo -c No. of forwarding lcores: 8 No. of interfaces: 5 Lcore 0: Interface: bond 0. 101 Interface: vhost 0 Lcore 4: Queue ID: 0 Queue ID: 1 Queue ID: 0 Queue ID: 4 Queue ID: 3 Queue ID: 2 Interface: bond 0. 101 Interface: tap 41 aceabd-87 Queue ID: 5 Queue ID: 3 Interface: bond 0. 101 Queue ID: 6 Interface: bond 0. 101 Queue ID: 7 Lcore 5: Lcore 1: Interface: bond 0. 101 Interface: tapa 7 af 118 d-a 1 Interface: tap 41 aceabd-87 Lcore 6: Lcore 2: Interface: bond 0. 101 Interface: tapa 7 af 118 d-a 1 Interface: tap 41 aceabd-87 Queue ID: 2 Queue ID: 1 Queue ID: 0 Interface: bond 0. 101 Interface: tapa 7 af 118 d-a 1 Interface: tap 41 aceabd-87 Queue ID: 3 Queue ID: 2 Queue ID: 1 Lcore 7: Lcore 3: 5
DPDKInfo tool Mempool Information $ dpdkinfo -mall -------------------------Name Size -------------------------rss_mempool 16384 3421 frag_direct_mempool frag_indirect_mempool 4096 slave_port 0_pool 8193 slave_port 1_pool 8193 packet_mbuf_pool 8192 Used Available 12963 4096 0 202 389 123 0 4096 7991 7804 8069 4096 $ dpdkinfo -m rss_mempool flags = 10 nb_mem_chunks = 77 size = 16384 populated_size = 16384 header_size = 64 elt_size = 9520 trailer_size = 80 total_obj_size = 9664 private_data_size = 64 avg bytes/object = 9664. 882812 Internal cache infos: cache_size=256 cache_count[0]=281 cache_count[8]=332 cache_count[9]=190 cache_count[10]=57 cache_count[11]=382 cache_count[12]=170 cache_count[13]=40 cache_count[14]=21 total_cache_count=2303 common_pool_count=10648 6
DPDKInfo tool Physical interface Statistics $ dpdkinfo -neth Master Info: RX Device Packets: 353631, Bytes: 68248463, Errors: 0, Nombufs: 0 Dropped RX Packets: 0 TX Device Packets: 499158, Bytes: 308085695, Errors: 0 Queue Rx: [0]353631 Tx: [0]499158 Rx Bytes: [0]68248463 Tx Bytes: [0]307520831 Errors: ----------------------------------- Slave Info(0000: 02: 00. 1): RX Device Packets: 97421, Bytes: 48536766, Errors: 0, Nombufs: 0 Dropped RX Packets: 0 TX Device Packets: 144098, Bytes: 56801274, Errors: 0 Queue Rx: [0]97421 Tx: [0]144098 Rx Bytes: [0]48536766 Tx Bytes: [0]56801272 Errors: ----------------------------------- Slave Info(0000: 02: 00. 0): RX Device Packets: 256210, Bytes: 19711697, Errors: 0, Nombufs: 0 Dropped RX Packets: 0 TX Device Packets: 355060, Bytes: 251284421, Errors: 0 Queue Rx: [0]256210 Tx: [0]355060 Rx Bytes: [0]19711697 Tx Bytes: [0]250719559 Errors: ----------------------------------- 7
DPDKInfo tool Physical Interface Registers (Extended statistics) $ dpdkinfo -x 1 Slave Info(0): 0000: 02: 00. 0 Rx Packets: rx_good_packets: 256611 rx_q 0 packets: 256611 rx_size_64_packets: 67362 rx_size_65_to_127_packets: 181451 rx_size_128_to_255_packets: 2174 rx_size_256_to_511_packets: 2876 rx_size_512_to_1023_packets: 2136 rx_size_1024_to_max_packets: 612 rx_broadcast_packets: 66705 rx_multicast_packets: 69450 rx_total_packets: 256611 Tx Packets: tx_good_packets: 355670 tx_q 0 packets: 355670 tx_total_packets: 355670 tx_size_64_packets: 40412 tx_size_65_to_127_packets: 74250 tx_size_128_to_255_packets: 79485 tx_size_256_to_511_packets: 2786 tx_size_512_to_1023_packets: 8051 tx_size_1024_to_max_packets: 150686 tx_multicast_packets: 72173 tx_broadcast_packets: 33356 Rx Bytes: rx_good_bytes: 19738985 rx_q 0 bytes: 19738985 rx_total_bytes: 19738985 Tx Bytes: tx_good_bytes: 251760114 tx_q 0 bytes: 251194370 Errors: Others: out_pkts_untagged: 355670 ----------------------------------- 8
DPDKInfo tool Global/Application Information $ dpdkinfo -a No. of lcores: 18 No. of forwarding lcores: 8 Fabric interface: bond 0. 101 Slave interface(0): enp 2 s 0 f 0 Slave interface(1): enp 2 s 0 f 1 Vlan name: vfw 0 Vlan tag: 101 Vlan vif: vfw 0 Ethdev (Master): Max rx queues: 128 Max tx queues: 64 Ethdev nb rx queues: 8 Ethdev nb tx queues: 64 Ethdev nb rss queues: 8 Ethdev reta size: 128 Ethdev port id: 2 Ethdev nb slaves: 2 Ethdev slaves: 0 1 0 0 Ethdev (Slave 0): 0000: 02: 00. 0 Nb rx queues: 8 Nb tx queues: 64 Ethdev reta size: 128 Ethdev (Slave 1): 0000: 02: 00. 1 Nb rx queues: 8 Nb tx queues: 64 Ethdev reta size: 128 Tapdev: fd: 580 vif name: vfw 0 fd: 590 vif name: vhost 0 fd: 621 vif name: mon 4 Monitorings: tap 41 aceabd-87 9
Sandump tool ● This is a troubleshooting and diagnostic tool for DPDK enabled v. Router in R 2008 ● Sniffs the messages transferred ● Available in between the Agent and v. Router on contrail-tools the netlink channel container v. Router Agent Sandump tool pkt 0 netlink DPDK v. Router Wireshark Application 10
Sandump tool Usage Method 1 - Live view Method 2 - Saving a pcap and view in wireshark application 11
pkt 0 interface packets analyser Analyzer for packets transferred between v. Router and Agent ● First packet of a flow ● Exception packets like ARP, DHCP, DNS, BFD etc. v. Router Agent pkt 0 analyzer pkt 0 netlink DPDK v. Router tcpdump -i pkt 0 vifdump 2 Wireshark Application 12
pkt 0 interface packets analyzer Packet Format Proprietary Agent Header ARP/DHCP/DNS packet OR First packet of flow 13
Clear vif counters New options to vif command to clear counters 14
References DPDKInfo tool https: //www. juniper. net/documentation/en_US/contrail 20/topics/task/co nfiguration/vrouter-cli-utilities-vnc. html#id-dpdkinfo-command Sandump tool https: //www. juniper. net/documentation/en_US/contrail 20/topics/task/ins tallation/sandump-tool. html pkt 0 interface packets analyzer https: //www. juniper. net/documentation/en_US/contrail 20/topics/task/co nfiguration/adding-agent-header-using-wireshark-plugin. html 15
Q&A THANK YOU!! Team ● Yashika B ● Haji M ● Anand ● Kiran K N 16
- Slides: 16