HYPERV IOMMU ARCHITECTURE AGENDA Brief Overview of DMA

HYPER-V IOMMU ARCHITECTURE

AGENDA [Brief] Overview of DMA and Interrupts Hyper-V Current I/O Architecture Device Assignment I/O architecture Security Issues IOMMU Overview Hypervisor IOMMU Architecture Additional Use Cases References Questions

TERMINOLOGY System Physical Address (SPA) – An address that refers to the physical memory in the system Guest Physical Address (GPA) – An address that refers to the guest view of physical memory Logical Processor (LP) - Physical CPU which can be a thread (SMT) or core Virtual Processor (VP) – Virtual CPU for the partitions Root partition – A partition that manages the child partitions

DMA AND INTERRUPTS

DIRECT MEMORY ACCESS (DMA) Allows devices to read/write data to/from memory Uses physical addressing Can be coherent or non coherent Allows processor to work in parallel while data is accessed by the device

DMA (CONT. ) Device A requests DMA at 0 x 21000 DMA Request Device A Address: 0 x 21000 DMA Request Physical Address: 0 x 21000

INTERRUPTS Interrupts are events that indicate that a condition exists somewhere in the system, the processor or within the currently executing program that requires the attention of a processor (Intel Manual) Interrupts can be generated either by the processor, devices or the currently executing program On all modern PC systems, Interrupts are processed through the Local APIC unit on each processor

PROCESSOR INTERRUPT HANDLING Each processor has one Local APIC unit Each Local APIC unit has a unique id in the system Local APIC receives interrupt requests from internal and external sources and sends them to processor core for handling Local APIC can send interrupt requests to other logical processors in the system. This is called Inter Processor Interrupt (or IPI) Interrupt requests are sent by writing to the ICR register on the APIC page

INTERRUPT HANDLING (CONT…) Interrupts are identified from various fields in the interrupt request shown below

DEVICE INTERRUPTS Two ways for a device to generate an interrupt request: � I/O APIC � Message Signaled Interrupt (MSI)

DEVICE INTERRUPTS I/O APIC � This is used by devices that are connected to I/O APIC in the system � The interrupt requests comes to the I/O APIC which sends the interrupt request to Local APIC � Multiple devices can be connected to same I/O APIC input, thus sharing the interrupt

DEVICE INTERRUPTS MSI � Devices generate an MSI interrupt by writing MSI Data to a special address called MSI Address � Each device can generate its own interrupts and there is no need to share interrupts between multiple devices � This is much more flexible method and is required on the PCIe (PCI Express) devices

HYPER-V CURRENT I/O ARCHITECTURE

HYPER-V CURRENT I/O ARCHITECTURE Root Partition Child Partition Networking Stack VSP NIC Driver VSC VMBUS Hypervisor NIC

LIMITATIONS Significant cross-partition communication overhead Hardware features are not utilized (such as RSS) Extra memory copy overhead Root partition bottleneck

DEVICE ASSIGNMENT I/O ARCHITECTURE Root Partition Child Partition Networking Stack NIC Driver Hypervisor NIC

SECURITY ISSUES The child partition is untrusted The device assigned to the child partition has ability: � to do DMA � to generate interrupt requests This means the child partition can program the device to: � Read/Write the hypervisor, root partition or other partition’s memory � Spoof interrupts from other devices � Do. S attack by causing interrupt storm � Effectively take over the whole system

WHAT IS NEEDED FOR SAFE DEVICE ASSIGNMENT Ability to restrict devices to access its assigned child partition’s memory only Ability to prevent interrupt spoofing or storm attack from such devices The answer: IOMMU

IOMMU OVERVIEW

WHAT IS AN IOMMU? Hardware that allows DMA requests to be programmed in terms of GPA and verified using I/O page tables before accessing memory – DMA remapping Hardware that allows interrupt requests to be verified using interrupt remapping tables and targeted to the specified processors – Interrupt remapping Intel VT-d 2 and AMD IOMMU

DMA REMAPPING Provides translation of GPA to SPA for DMA requests � Uses I/O page tables for translation Supports Domains � Domain is a container for one or more devices that share one I/O page table � I/O page tables are used to ensure that a device can only access memory that belongs to its assigned domain � I/O page table can only be accessed by the privileged software thus preventing any tampering by untrusted software

DMA REMAPPING (CONT. ) Device A requests DMA at 0 x 21000 DMA Request Device A Address: 0 x 21000 IOMMU GPA SPA 0 x 21000 0 x. CF 000 0 x 22000 0 x. D 0000 0 x 23000 0 x. D 1000 DMA Request Address: 0 x. CF 000

DMA REMAPPING (CONT. ) Device B requests DMA (uses different translation tables) DMA Request Device B Address: 0 x 21000 IOMMU GPA 0 x 21000 0 x 22000 0 x 23000 SPA 0 x. EA 000 0 x. CF 000 0 x. EB 000 0 x. D 0000 0 x. EC 000 0 x. D 1000 DMA Request Address: 0 x. EA 000

DMA REMAPPING (CONT. ) Device C requests DMA Request Device C Address: 0 x 21000 IOMMU GPA SPA 0 x 21000 - 0 x 22000 - 0 x 23000 -

DMA REMAPPING Enables DMAs to be programmed in terms of GPA Allows for restricting DMA access to a subset of physical memory

INTERRUPT REMAPPING Allows software to setup an interrupt remapping table (IRT) Interrupt requests from devices are intercepted and routed based on the IRT Restricts the interrupt requests from a device to the assigned vector on assigned processor

INTERRUPT REMAPPING (CONT. ) Interrupt Request Index* (2) Device: BDF-A IOMMU Interrupt Request LP 2, Vector 0 x 82 Logical Processor 2 Device ID Interrupt Index Target BDF-A 1 LP 1, Vector 0 x 80 BDF-A 2 LP 2, Vector 0 x 82 BDF-B 3 LP 1, Vector 0 x 81 BDF-C 4 LP 2, Vector 0 x 91 *Interrupt Index is calculated from MSI Data/Address pair or IOAPIC RTE.

Hyper-V IOMMU Architecture

HYPERVISOR IOMMU ARCHITECTURE Introduces two new concepts – Device Domain and Device Interrupt Provides support for DMA remapping and interrupt remapping to multiple partitions Abstracts hardware differences between AMD IOMMU and Intel VT-d 2 implementation

DEVICE DOMAIN Device domain is a container for one or more devices Contains one I/O page table Associated with a partition Devices can be attached to (and detached from) a domain All DMA requests from the attached device goes through the domain I/O page table Uses PCI Segment/Bus/Device/Function as the device identifier One identity mapped domain per partition

DEVICE DOMAIN Root Partition Child Partition Hypervisor Device (Assigned to child partition) IOMMU I/O Page Table (For Child Partition setup by the Hypervisor)

DEVICE INTERRUPTS Hardware Interrupt Remapping � Hardware interrupt remapping ensures device interrupt come on a specific physical processor and interrupt vector � Ensures that a device can’t spoof an interrupt request Software Interrupt Remapping � Software Interrupt Remapping maps a specific hardware interrupt from an LP to a VP Routing an interrupt from a device to specific VP, vector pair requires both software and hardware interrupt remapping

DEVICE INTERRUPTS Child Partition (1) Child Partition (2) Virtual Processor Root Partition Virtual Processor Hypervisor Software Interrupt Remapping Table Device (Assigned to child partition 1) IOMMU Hardware Interrupt Remapping Table Logical Processor 0 Logical Processor 1

REFERENCES AMD IOMMU specifications � http: //www. amd. com/us- en/assets/content_type/white_papers_and_tech_docs/3 4434. pdf Intel VT-d 2 specifications � ftp: //download. intel. com/technology/computing/vptech/In tel(r)_VT_for_Direct_IO. pdf