ERIM Secure Efficient inprocess Isolation with Memory Protection

ERIM: Secure, Efficient in-process Isolation with Memory Protection Keys Anjo Vahldiek-Oberwagner, Eslam Elnikety, Nuno O. Duarte, Michael Sammler, Peter Druschel, Deepak Garg

Applications in the Absence of Isolation • All state accessible at all times to • Bugs • Security vulnerabilities Application 2

Applications in the Absence of Isolation Heartbleed Bug ~70% of CVE assigned by Microsoft are memory safety issues. Microsoft Security Response Center: “A proactive approach to more secure code”, 2019 3

Example In-Process Isolation Use Cases Cryptographic Secrets Untrusted Application Trusted Crypto Library Managed runtimes from native libraries Native Library Managed Runtime 4

User-space Threat Model Untrusted Application Trusted Compartment Untrusted Trusted Operating System CPU Attacker’s Capabilities include, but not limited to • Control-flow hijacks • Memory corruption (i. e. , out-of-bounds accesses) Out of scope: • Side-channel, row hammer or microarchitectural attacks 5

State of In-Application Isolation Techniques Execution overhead OS/VMMbased 2 Lang. & RT 3 ERIM Untrusted Trusted Switch overhead Low Medium – High None Low OS/VMM Technique Application Sensitive Data Application OS + VMM 1 Lw. C, 2 SMVs, Shreds, Wedge, Nexen, Dune, Se. Cage, Trust. Visor SFI 6

State of In-Application Isolation Techniques Language and Runtime Techniques Trusted Switch overhead Low Medium Application None Low Execution overhead Untrusted OS/VMMLow based 2 Lang. & Medium – RT 3 High ERIM Low Sensitive Data Operating System 1 Lw. C, 2 SMVs, Shreds, Wedge, Nexen, Dune, Se. Cage, Trust. Visor SFI 7

State of In-Application Isolation Techniques Execution overhead OS/VMMbased 2 Lang. & RT 3 ERIM Untrusted Trusted Switch overhead Low Medium – High None Low ERIM Application ERIM Sensitive data Operating System 1 Lw. C, SMVs, Shreds, Wedge, Nexen, Dune, Se. Cage, Trust. Visor 2 SFI, Native Client, Memsentry-MPX 8

Memory Protection Keys (MPK) Address Space • Available in Skylake server CPUs • Tag memory pages with PKEY Page 3 Page 1 Page 2 Page Table Entry (PTE) PKEY … … 0 Page 1 … 9

Intel Memory Protection Keys (MPK) Address Space • Available in Skylake server CPUs • Tag memory pages with PKEY Page 3 Page 1 Page 2 Page Table Entry (PTE) PKEY … … 2 Page 1 … 10

Intel Memory Protection Keys (MPK) Address Space • Available in Skylake server CPUs • Tag memory pages with PKEY • Permission Register (PKRU) Page 3 Page 1 Page 2 CPU Core PKRU Register 15 W 0 15 R 0 … … 2 W 0 2 R 0 1 W 0 1 R 0 0 W 1 0 R 1 Page Table Entry (PTE) PKEY … … 2 Page 1 … 11

Intel Memory Protection Keys (MPK) Address Space • Available in Skylake server CPUs Page 3 • Tag memory pages with PKEY • Permission Register (PKRU) • Userspace instruction to update PKRU Page 1 Page 2 • Fast switch between 11 – 260 cycles/switch CPU Core PKRU Register 15 W 0 15 R 0 … … 2 W 1 2 R 1 1 W 0 1 R 0 0 W 1 0 R 1 Page Table Entry (PTE) PKEY … … 2 Page 1 … 12

Intel Memory Protection Keys (MPK) • • Address Space Available in Skylake server CPUs Tag memory pages with PKEY Permission Register (PKRU) By itself, Page 1 Userspace instruction to update PKRU MPK does not protect Page 2 • Fast switch at 50 cycles/switch Page 3 against malicious attacks. CPU Core PKRU Register 15 W 1 15 R 1 … … 2 W 1 2 R 1 1 W 1 1 R 1 0 W 1 0 R 1 Page Table Entry (PTE) PKEY … … 2 Page 1 … 13

Overview of ERIM • Prevent MPK exploitation Untrusted Application PKEY 0 • Safe call gates • Prevent execution of permission register updates outside of call gates Trusted Compartment PKEY 1 Code: 48 83 c 0 08 44 01 fa 83 fa 07 77 0 f 01 ef 83 ff 07 0 f 96 c 2 80 14

Overview of ERIM • Prevent MPK exploitation Untrusted Application PKEY 0 • Safe call gates • Prevent execution of permission register updates outside of call gates Trusted Compartment PKEY 1 Code: 48 83 c 0 08 44 01 fa 83 fa 07 77 0 f 01 ef 83 ff 07 0 f 96 c 2 80 15

Overview of ERIM • Prevent MPK exploitation Untrusted Application PKEY 0 • Safe call gates • Prevent execution of permission register updates outside of call gates Trusted Compartment PKEY 1 • Creating usable binaries • Inadvertent PKRU update instruction • Rewrite strategy Code: 48 83 c 0 08 44 01 fa 90 ef 01 ef 83 fa 07 77 0 f 01 83 ff 07 0 f 96 c 2 80 16

Overview of ERIM • Prevent MPK exploitation Untrusted Application PKEY 0 • Safe call gates • Prevent execution of permission register updates outside of call gates Trusted Compartment PKEY 1 • Creating usable binaries • Inadvertent PKRU update instruction • Rewrite strategy • Evaluation • Frequently-switching use cases • 10% higher throughput compared to best existing technique Code: 48 83 c 0 08 44 01 fa 83 fa 07 77 0 f 90 01 ef 83 ff 07 0 f 96 c 2 80 17

Updating the permission in PKRU register • WRPKRU • Write EAX into PKRU • XRSTOR • If bit 9 of EAX is set • Load PKRU register from specified memory address 18

Safe switching using call gates Trusted Compartment perm = TRUSTED WRPKRU (perm) goto trusted_entry(T) perm = UNTRUSTED WRPKRU (perm) perm = TRUSTED Untrusted Application 19

Safe switching using call gates Trusted Compartment perm = TRUSTED WRPKRU (perm) goto trusted_entry(T) Untrusted Application perm = UNTRUSTED WRPKRU (perm) if (perm != UNTRUSTED) exit; 20

Prevent execution of WRPKRU/XRSTOR outside of call gates Trusted Compartment New Memory (No Execute) Untrusted Application System Calls ERIM Operating System Prevent execution of unvetted pages by 1) Monitoring system calls and removing the execute permission 2) ERIM’s fault handler scans memory pages and ensures: • WRPKRU is part of a call gate • XRSTOR is followed by if(eax | 0 x 100) exit(); 21

Overview of ERIM • Prevent MPK exploitation Untrusted Application PKEY 0 • Safe call gates • Prevent execution of permission register updates outside of call gates Trusted Compartment PKEY 1 • Creating usable binaries • Inadvertent PKRU update instruction • Rewrite strategy • Evaluation • Frequently-switching use cases • 10% higher throughput compared to best existing technique Code: 48 83 c 0 08 44 01 fa 83 fa 07 77 0 f 01 ef 83 ff 07 0 f 96 c 2 80 22

Creating usable binaries • ERIM halts executables with inadvertent WRPKRUs/XRSTORs Inter-Instruction WRPKRU Instruction 1 Instruction 2 … 0 F 01 EF… Intra-Instruction WRPKRU Instruction 1 010 F 01 EF 0000 Eliminate inadvertent WRPKRU/XRSTOR by binary rewriting at compile time, runtime prior to enabling execute permission, or via static binary rewriting for pre-compiled binaries 23

Rewriting inadvertent WRPKRUs/XRSTORs Devise rewrite rules for inadvertent WRPKRUs Inter-Instruction: Instruction 1 Instruction 2 … 0 F 01 EF… … 0 F 90 01 EF… Nop 24

Rewriting inadvertent WRPKRUs/XRSTORs Devise rewrite rules for inadvertent WRPKRUs Intra-instruction WRPKRU Simplified x 86 instruction format: Prefix Opcode Mod R/M SIB Displacement Immediate Required Optional

Rewriting inadvertent WRPKRUs/XRSTORs Devise rewrite rules for inadvertent WRPKRUs Example rewrite rule: add ecx, [ebx + 0 x 01 EF 0000] push eax; mov eax, ebx; add ecx, [eax + 0 x 01 EF 0000]; pop eax; Opcode 0 x 01 Mod R/M 0 x 0 F Displacement 0 x 01 EF 0000 Opcode 0 x 01 Mod R/M 0 x 07 Displacement 0 x 01 EF 0000 26

Overview of ERIM • Prevent MPK exploitation Untrusted Application PKEY 0 • Safe call gates • Prevent execution of permission register updates outside of call gates Trusted Compartment PKEY 1 • Creating usable binaries • Inadvertent PKRU update instruction • Rewrite strategy • Evaluation • Frequently-switching use cases • 10% higher throughput compared to best existing technique Code: 48 83 c 0 08 44 01 fa 83 fa 07 77 0 f 90 01 ef 83 ff 07 0 f 96 c 2 80 27

Prototype implementation • ERIM userspace library • Call gates • Memory allocator for trusted component overloading malloc-like functions • Memory inspection (exclude unsafe WRPKRU/XRSTOR) • Prevent execution on pages with unsafe WRPKRUs/XRSTOR a) P-Trace and seccomp BPF userspace monitor b) Linux Security Module • Remove inadvertent WRPKRUs/XRSTORs • Static binary rewrite tool based on Dyn. Inst 28

Evaluation How frequent are inadvertent WRPKRUs/XRSTORs? • • Inspected about 200, 000 executable files of 5 Linux distributions Found 1213 inadvertent WRPKRU/XRSTOR in binary code Dyn. Inst disassembled 1, 023 100% rewrite success What is ERIM’s overhead in frequently-switching use cases? • Isolating session keys in Nginx • Isolating a managed runtime (node. js) from native libraries • Isolating in-memory state of reference monitors (CPI/CPS) 29

Use case: Session Key Isolation Address Space NGINX Connection Management Content Open. SSL & Lib. Crypto AES Compartment HTTPS session Handshake protocol Cryptographic keys AES encrypt/decrypt AES key initialization 30

Nginx Throughput with protected session keys ERIM throughput within 5% of native. Normalized Throughput 1 Native 0, 8 ERIM 0, 6 0, 4 0, 2 0 0 1 2 4 8 16 File size in KB 32 64 128 31

Nginx Throughput with protected session keys 1. 3 million switches per second Normalized Throughput 1 Native 0, 8 ERIM 0, 6 0, 4 0, 2 0 0 1 2 4 8 16 File size in KB 32 64 128 32

Comparison to Prior Art 95. 4% ERIM 86. 4% VMFUNC 73. 2% Mem. Sentry-MPX Throughput 1 0, 8 0, 6 0, 4 0, 2 0 0 kb 1 kb Native 2 kb ERIM 4 kb VMFUNC 8 kb 16 kb Mem. Sentry-MPX 32 kb 64 kb 128 kb Light-weight Context 33

Summary • Prevent MPK exploitation Untrusted Application PKEY 0 • Safe call gates • Prevent execution of permission register updates outside of call gates Trusted Compartment PKEY 1 • Creating usable binaries • Inadvertent PKRU update instruction • Rewrite strategy • Evaluation • Frequently-switching use cases • 10% higher throughput compared to best existing technique Code: 48 83 c 0 08 44 01 fa 83 fa 07 77 0 f 90 01 ef 83 ff 07 0 f 96 c 2 80 34

Thank you! ERIM: Secure, Efficient in-process Isolation with Memory Protection Keys Anjo Vahldiek-Oberwagner, Eslam Elnikety, Nuno O. Duarte, Michael Sammler, Peter Druschel, Deepak Garg Code available at https: //gitlab. mpi-sws. org/vahldiek/erim 35

Backup 36

Intel Memory Protection Keys (MPK) • Tag memory pages with a memory domains (bits 62: 59 in page table) • Permission register (PKRU) enables R/W to a domain • Update accessible permissions from userspace • Fast switching, without context/PT switch • By itself, protects against bugs only PKRU register (32 bit, 2 bits per domain) Page Table Entry 11 … Do Do … in ma Domain (bits 62: 59) … Do 2 Do … … 3 2 1 0 37

State of the art: Isolating in-memory state ASLR-based Hiding Application Sensitive data OS/VMM-Based Application OS + VMM Language and Runtime Techniques Application ERIM: Memory Isolation using Intel MPK ASLR 1 Sensitive data ERIM Operating System Low OS/VMM Low -based 2 Lang. & Medium – RT 3 High ERIM Application Operating System Switch Untrusted Trusted overhead Sensitive data Operating System Sensitive data Execution overhead 1 Low None Low Medium None Low Threat model Application bugs only Any userspace ASLR-Guard, Near, Xn. R 2 Lw. C, SMVs, Shreds, Wedge, Nexen, Dune, Se. Cage, Trust. Visor 3 Mem. Sentry, SFI 38

Isolating sensitive state with Intel MPK Address Space TRUSTED Domain 1 Domain 0 UNTRUSTED Sensitive State Permission Register (PKRU) 11 00 11 Untrusted Application State D 0 D 1 Domain switch is a user-mode register write: efficient but vulnerable to attack. 39

Using ERIM to isolate memory Inlined switches Function overwriting Function overloading via LD_PRELOAD fct_A(…) { …. switch(Trusted) fct_A(…) { …. } access sensitive data BUILD_BRIDGE(fct_A); switch(Untrusted) … } fct_B(…) { … CALL_BRIDGE(fct_A, args); … } Shared library defines: fct_A(…) { f = dlsym(fct_A, …); switch(Trusted); ret = f(args); switch(Untrusted); return ret; } 40

Comparison to MPX 41

Comparison to VMFUNC EPT switch 42

Comparison to Lw. C 43

How frequent are inadvertent WRPKRUs/XRSTORs? Debian 8 56035 Ubuntu 14 58548 Ubuntu 16 69907 Gentoo 9940 Gentoo Gold 9940 665 603 720 73 34 4244 1147 2105 124 46 WPKRU/XRSTOR in code 481 276 384 41 31 Disassembled by Dyninst Inter-instruction Intra-instruction 420 30 390 215 29 186 332 44 288 32 5 27 24 5 19 Elf files with WRPKRU/XRSTOR Executable WRPKRU/XRSTOR 44

How frequent are inadvertent WRPKRUs? Elf files w/ WRPKRU/XRSTOR Executable WRPKRUXRSTOR WPKRU/XRSTOR in code Disassembled by Dyninst Inter-instruction Number Rewritable by NOP Intra-instruction Number Rewritable by rule 5 Rewritable by rule 4/6 Debian 8 56035 All 665 4244 481 420 30 30 390 199 191 Ubuntu 14 Ubuntu 16 Gentoo Gold 58548 69907 9940 WRPKRU XRSTOR All WRPKRU XRSTOR 174 541 603 215 435 720 189 580 73 22 59 34 17 20 288 3956 1147 442 705 235 1870 124 26 98 46 18 28 63 418 276 66 210 384 83 301 41 9 32 31 14 17 52 368 215 55 160 332 73 259 32 9 23 24 14 10 30 0 29 29 0 44 41 3 5 5 0 22 368 186 26 160 288 32 256 27 4 23 19 9 10 22 177 181 26 155 246 32 214 27 4 23 19 9 10 0 194 5 0 5 42 0 0 0 0 45

ERIM Related Work Hardware-based Isolation: • Trusted Execution Engines (TEE) [SGX, Trust. Zone] • Reducing TCB of TEE [Flicker] • Sandbox applications in TEE [Haven, Scone] Hypervisor/OS-based: • Reference monitors [Dune, Wedge, Lw. C] • Sandboxing Applications [Capsicum] • Privilege Separation [Priv. Trans] • Hiding secrets in execute-only code [Redactor, Near] 46

ERIM Related Work Software-fault isolation: • Compilation-based [Native. Client] • Emulation [Vx 32] • Just-in-time compiled languages [Native. Client++] Inlined Reference Monitoring: • Control-Flow Integrity [CPI] • Sandboxing annotated code [Shreds] • Intercepting Android framework [Aurasium] 47

Call Gates WRPKRU (RW_TRUSTED) // entry point to trusted WRPKRU (DIS_TRUSTED) cmp DIS_TRUSTED, EAX je continue exit continue: Elevate privileges and transfer to trusted entry point Remove privileges, check for reduced privileges and return from trusted component 48

Creating safe binaries Devise rewrite rules for WRPKRU in code segment Inter-instruction WRPKRU (0 x 0 F 01 EF) Example rewrite rule: Instruction 1 Instruction 2 … 0 F 01 EF… … 0 F Nop 01 EF… 49

Creating safe binaries Intra-instruction WRPKRU Simplified x 86 instruction format: Prefix Opcode Mod R/M Example rewrite rule: add ecx, [ebx + 0 x 01 EF 0000] SIB Displacement Immediate Opcode 0 x 01 Mod R/M 0 x 0 F Displacement 0 x 01 EF 0000 push eax; mov eax, ebx; add ecx, [eax + 0 x 01 EF 0000]; pop eax; Opcode 0 x 01 Mod R/M 0 x 07 Displacement 0 x 01 EF 0000 50

Creating safe binaries: Rewrite Rules 51

WRPKRU Occurrances 52

Nginx Throughput with protected session keys File size Native (req. /s) ERIM rel. (%) Switches/s 0 1 2 4 8 16 32 64 128 95, 761 87, 022 82, 137 76, 562 67, 855 45, 483 32, 381 17, 827 8, 937 95. 83 95. 18 95. 44 95. 25 95. 98 97. 10 97. 31 100. 0 99. 99 1, 342, 605 1, 220, 266 1, 151, 877 1, 073, 843 974, 780 812, 173 779, 141 679, 371 556, 152 CPU load 100 100 96. 7 86. 4 CPU bound Network bound 53

ERIMized C Program typedef struct secret { int compute(secret* s, int m) { int number; int ret = 0; } secret; ERIM_SWITCH_T; secret* init. Secret() { ret = f(s->number, m); ERIM_SWITCH_T; ERIM_SWITCH_U; secret * s = malloc(sizeof(secret)); return ret; s->number = random(); } ERIM_SWITCH_U; return s; } 54

SPEC 2006 with CPS/CPI 55

NGINX multiple worker 56