DPDK Prevention Detection of DPCP memory corruption 9

DPDK: Prevention & Detection of DP/CP memory corruption 9 MARCH 2018 AMOL PATEL, SIVAPRASAD TUMALLA

Background • • Most of the custom packet-processing hardware i. e. Network Processors provides mechanisms to, • Separate Dataplane and Controlplane memory • Protect and detect the Dataplane memory corruption/configuration issues We are Proposing something similar for the DPDK applications running on the GPP (General Purpose Processor)

Background • • The DPDK applications are running on the multi-threaded model DPDK primary process memory map Main process spawns worker-threads for the Dataplane processing This is a flat memory architecture wherein Control-thread and DP workerthread’s memory is part of the same address-space, it would be difficult to isolate the issue in case of Dataplane/Controlplane data structure corruption We need ways to prevent/detect the Controlplane/Dataplane memory corruption. Huge Page Segment 1 Mem-zone area 1 Huge Page Segment 2 Huge Page Segment 3 Mem-zone area 2 P r i m a r y P r o c e s s M e m o r y Shared Libraries Hugepage mmap Stack-n Thread - n Stack-2 Thread - 2 Stack-1 Thread - 1 Heap text

Ways to Prevent and Detect DP/CP Memory corruption • • Prevention • Control-plane memory (stack) protection • Dataplane memory protection • Controlplane and Dataplane memory access control Detection • Dataplane Memory Corruption Detection • DPDK Mem-zone Dump

Prevention • Controlplane stack protection • Dataplane memory protection • Controlplane and Dataplane memory access control

Controlplane memory Protection : Existing memory model Shared Libraries Huge Page Segment 1 Mem-zone area 1 Huge Page Segment 2 Huge Page Segment 3 Mem-zone area 2 P r i m a r y P r o c e s s M e m o r y Hugepage mmap • Worker-threads are the Linux threads • Stack memory for the workerthreads is allocated in primarythreads’ Address space • Thread level MMU protection does NOT exist • All the worker-threads of the primary-process has access to stack memory of all the other threads, one worker thread can corrupt the other worker thread’s stack Stack-n Thread - n Stack-2 Thread - 2 Stack-1 Thread - 1 Heap text

Worker thread Stack Protection: Proposed Memory Model Shared Libraries Huge Page Segment 1 Mem-zone area 1 Huge Page Segment 2 Huge Page Segment 3 Mem-zone area 2 P r i m a r y P r o c e s s • Hugepage mmap Stack-n Worker Thread-n Mechanism: • Stack of the worker-thread to be allocated on mem-zone • Assign the allocated mem-zone memory to the worker-thread stack address, • int thread_attr_setstack(pthread_attr_t *attr void *stackaddr, size_t stacksize); Stack-2 Worker. Thread - 2 Stack-1 Worker Thread- 1 Heap text • Advantages: • Stricter control on any possible out-of-bound memory access for the stack variables • Localizing the stack corruption effect to the thread • Ability to dump the thread’s stack contents by dumping the mem-zone

DP Memory Protection: Existing DPDK Dataplane Memory Layout Mem-zone 1 Huge Page Segment 1 DP table • • Mem-zone 2 Huge Page Segment 2 Mechanism: Mem-zone 2 RTE_CONFIG Huge Page Segment 3 Shared Libraries • Stack Thread - n Disadvantages • Memory allocated for the different Dataplane tables might be scattered in different memory -segments/huge pages • No direct way to check for out-of-bound write for the Dataplane tables • There is no way to dump the contents of all the Dataplane tables and counters in one go Stack Thread - 2 Huge Page Segment 4 Stack Thread - 1 malloc, calloc, zmalloc regions Heap - n Huge Page Segment 5 text DP tables DP caches DP counters Other tables Actual tables are allocated using the rte_lmp_create/rte_hash_create which internally invokes the rte_dpdk callocs() to allocate memory from the Hugepage.

DP Memory Protection: Proposed DPDK DP Memory Allocation using memory zones DP-Mem-zone Huge Page Segment 1 Huge Page Segment 2 DP table 1 • Guard-band DP-Mem-zone • DP table 2 Guard-band DP-Mem-zone DP counters • Huge Page Segment 3 Mechanism Shared Libraries Advantages • Dataplane memory separation • DPDK memory zone dump APIs supported to dump all the DP tables. • Ease the troubleshooting the memory and configuration issues • Guard-band allows checking of any out-of-bound access Stack Thread - n Stack Thread - 2 Huge Page Segment 4 Stack Thread - 1 rte_malloc, rte_calloc Heap - n Huge Page Segment 5 text One mem-zone reserved for all the dataplane tables, data-structures, all the DP tables are placed on this mem-zone by a user-defined wrappers Disadvantages • • Additional wrappers for allocating the Dataplane tables (LPM, Hash, counters) on the mem-zones

CP / DP Memory Access Control Worker thread to access the dataplane-table memzones in Readonly mode (mmap call supports the RO access, DPDK would have to add this support). • In the worker threads, all the dataplane pointers MUST be accessed with the “constant” primitive to avoid any accidental corruption • Using “restrict” qualifier for all Read. Only function argument • Dataplane code not to change any dataplane entries except the fastpath counters • Control plane tracking the #entries for the DP tables and returning error if it’s reached MAX (this way we can avoid any over-stepping to the next DP table) •

Detection • Detection (Debugging / troubleshooting) • Dataplane Memory Corruption Detection • DPDK Mem-zone Dump

Dataplane Memory Corruption Detection • Single DP memzone for the dataplane objects (forwarding tables, counters, structure etc. ), each DP reserves few extra-rows for every DP table (Max + N) which serves as the guard-band • Monitoring the guard-bands b/w the individual DP tables to keep check on any out of bound access • Use DPDK utilities to dump the memzones to dump the DP tables, counters for debugging • Application test-framework to check any out-of-bound access for DP tables by monitoring the memzone

4 Stage Memzone Dumping Process • DPDKProc. Info –m : Dump Memzone physical and Virtual address • cat /proc/<pid>/maps : Dump process’ virtual memory map Huge_page mapping • Indentify the Memzone Hugepage mapping by comparing the virtual-address • hexdump /mnt/huge/mntrte_N : Dump the Memzone content

Detection – DPDK Memzone Dump : dpdkprocinfo –m

cat /proc/<pid>/maps

Troubleshooting – DPDK Memzone Dump : hexdump /mnt/huge/rtemap_

Questions?