x Vhost Zerocopy Best Practice MARVININTEL NPG Agenda

x Vhost Zero-copy Best Practice MARVIN-INTEL NPG

Agenda • Vhost Dequeue zero-copy introduction • VM 2 NIC Best Practice • VM 2 VM Best Practice • NUMA Best Practice 2

VM 2 NIC VIRTIO GUEST AVAIL DESC HOST KVM VHOST AVAIL DESC event. FD MBUF COPY RXQ TXQ NIC 3

VM 2 NIC Dequeue zero-copy VHOST USED AVAIL DESC MBUF HOLD ZMBUF NIC RXQ TXQ 4

VM 2 NIC Best Practice • Resource competition • Limitation: zero copy not work when bonding to vfio-pci with iommu mode 5

VM 2 NIC Best Practice Tips Millions VM 2 NIC Performance 10 9 8 7 • VIRTIO queue size = NIC queue size • Free threshold = TXD – burst size 6 5 4 3 2 1 0 32 64 128 256 512 600 700 800 NIC Free Threshold 896 928 960 992 6

VM 2 NIC Best Practice Line rate DPDK PVP Performance • Turning point - 512 100 90 80 70 60 50 40 30 20 10 0 64 128 256 Zero-copy 512 1024 1280 1518 Default 7

VM 2 VM De-queue zero-copy GUEST 0 GUEST 1 AVAIL USED DESC IRQFD VHOST AVAIL USED DESC KVM MBUF event. FD HOLD GUEST 1 RXQ GUEST 0 TXQ COPY ZMBUF 8

VM 2 VM Best Practice VM 2 VM IPerf TCP 45 40 Gbits/Sec 35 30 25 20 15 10 5 0 Zero-copy Default 9

VM 2 VM Best Practice VM 2 VM when touch packet header 10 9. 5 8 7. 5 7 6. 5 Packet Size Zero-copy 10 24 10 88 11 52 12 16 12 80 13 44 14 08 14 72 96 0 89 6 2 83 76 8 70 4 64 0 57 6 51 2 44 8 38 4 32 0 25 6 19 2 12 8 6 64 Millions 9 Default 10

VM 2 VM Best Practice • Enable VIRTIO_F_ANY_LAYOUT for better performance • DPDK 18. 02 • Perf improvement With ANY_LAYOUT Without ANY_LAYOUT 13. 9 Mpps 12. 9 Mpps 11

NUMA Best Practice CPU and Memory elements QPI CPU VHOST CPU VIRTIO • • VQ DESC • VQ AVAIL USED DESC AVAIL VHOST & VIRTIO CPU VHOST VQ and MP VHOST DESC/AVAIL/USE Ring VIRTIO VQ VIRTIO DESC/AVAIL/USE Ring USED MP 12

NUMA Best Practice • • CPU node has the dominate effect on performance Zero copy will be better when CPUs on different nodes CPU same node Memory same node Zero copy CPU same node Memory cross QPI Zero copy CPU cross QPI Memory same node Default CPU cross QPI Memory same node Zero copy CPU cross QPI Memory cross QPI Zero copy Throughput 9. 0 Mpps 8. 9 Mpps 3. 1 Mpps 5. 0 Mpps 4. 9 Mpps VHOST cache miss/Sec 2 K 4 K 21 M 7 M 8 M 13

NUMA Best Practice • NIC cross QPI better than CPU cross QPI CPU same node NIC cross QPI Zero copy CPU cross QPI NIC same node Zero copy NIC Throughput 11. 3 Mpps 10. 1 Mpps VHOST cache miss/sec 350 K 15 M 14

Thanks