Zerocopy Receive Path in Virtio Kalman Meth Mike
- Slides: 7
Zero-copy Receive Path in Virtio Kalman Meth, Mike Rapoport, Joel Nider {meth, rapoport, joeln}@il. ibm. com This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 645402.
Virtio Architecture Host VM Guest …………. user space kernel space p Host kernel buffer DMA KVM Hypervisor MAC 4 MAC 3 MAC 2 network MAC 1 Ethernet adapter NIC User buffer Guest kernel buffer Socket interface Guest kernel buffer virtio/vnet Host kernel buffer Copy data between guest buffers and host buffers Ethernet ring buffer(s)
Problem • In the KVM hypervisor, incoming packets from the network must pass through several objects in the Linux kernel before being delivered to the guest VM. • To keep the implementation simple, both the hypervisor and the guest keep their own sets of buffers on the receive path. • For large packets, the overall processing time is dominated by the copying of data from hypervisor buffers to guest buffers.
Solution: Zero-copy receive • Maintain separate ring buffer for each guest. • Guest allocates buffers and posts to appropriate host adapter ring buffer. • When data arrives, the NIC performs DMA directly into guestallocated buffers.
Zero-Copy Rx Architecture Host VM Guest user space …………. kernel space p MAC 4 MAC 3 MAC 2 MAC 1 Ethernet adapter network Guest kernel buffer macvtap macvlan NIC Guest kernel buffer Socket interface virtio/vnet DMA KVM Hypervisor User buffer Pass the buffers down through the kernel layers Per-MAC ring buffer
Technical Issues • Need to match guest interface with adapter device that consumes its buffers. • Need support in ethernet adapter to assign a particular queue to a specified MAC address. • What if not enough user-space buffers have been posted? • Drop packets? • Ethernet adapter driver expects 4 K page aligned buffers (not what virtio-net currently provides). • Add flag in virtio/vhost to negotiate and use 4 K aligned pages • Need 2 values returned for each buffer: • For each buffer filled by Ethernet adapter, need to match it back to index in virtio-net ring. • Need to provide virtio with number of bytes actually written into buffer. • Special handling for broadcast and multicast packets? • Do we need to turn off GRO, and have the guest do all the re-assembly? • Allow different virtio/vhost devices to use different schemes (4 K vs meargeable vs big buffers, etc), depending on capabilities of underlying physical device.
Proposed Interfaces • int (*ndo_set_zero_copy_rx)(struct net_device *dev, struct net_device *base_dev); • Pass base_dev down the stack to the ethernet adapter so it knows which MAC address is using zero-copy rx. • int (*ndo_post_rx_buffer)(struct net_device *dev, struct sk_buff *skb); • Passes a single (4 K aligned) buffer to the ethernet adapter • skb contains pointer to upper level device, so ethernet adapter knows to which MAC address the buffer is associated. • Ethernet adapter should save the skb to return the buffer up the stack (via the sock interface). Can use the skb auxiliary fields to hold the index of the buffer in the virtio-net ring. • May need a callback to handle some cases; perhaps similar to zerocopy transmit mechanism. Need to provide both number of bytes and index into virtio-net ring buffer.
