Memory Allocator Security Yves Younan Wouter Joosen Frank

Memory Allocator Security Yves Younan, Wouter Joosen, Frank Piessens and Hans Van den Eynden Distri. Net, Department of Computer Science Katholieke Universiteit Leuven Belgium Yves. Younan@cs. kuleuven. ac. be

Overview Ø Introduction Ø Attacks Ø Memory Allocators Ø A Safer Allocator Ø Related Work Ø Conclusion Yves Younan – Security of Memory Allocators for C and C++ 2

Introduction Ø Many allocators ignore security Ø Performance and waste limitation is more important Ø Many allocators can be abused to perform code injection attacks Ø More security is possible at a modest cost Yves Younan – Security of Memory Allocators for C and C++ 3

Overview Ø Introduction Ø Attacks Ø Heap-based buffer overflow Ø Off by One/Off by Five Ø Dangling Pointer References Ø Memory Allocators Ø A Safer Allocator Ø Related Work Ø Conclusion Yves Younan – Security of Memory Allocators for C and C++ 4

Heap-based buffer overflows Ø Heap memory is dynamically allocated at run-time Ø Can also be overflowed but no return address is available Ø Modify data pointers (IPO) or function pointers - not always available Ø Modify the memory management information associated with heap memory Yves Younan – Security of Memory Allocators for C and C++ 5

Off by one / Off by five Ø Special case of buffer overflow: limited space needed Ø Off by one: write one byte past the bounds Ø Off by five: don’t occur often but demonstrate low-space attacks Ø Usually only exploitable on little endian machines (LSB is stored before MSB) Yves Younan – Security of Memory Allocators for C and C++ 6

Dangling pointer references Ø Pointers to memory that is no longer allocated Ø Dereferencing is unchecked in C Ø Generally leads to crashes (SIGSEGV) Ø Can be used for code injection attacks when deallocated twice (double free) Ø A double free can be used to change memory management information allowing an overwrite of arbitrary memory locations Yves Younan – Security of Memory Allocators for C and C++ 7

Overview Ø Introduction Ø Attacks Ø Memory Allocators Ø Doug Lea’s malloc (Linux) Ø CSRI/Quickfit Ø Poul-Henning Kamp’s malloc (BSD) Ø Boehm’s garbage collector Ø A Safer Allocator Ø Related Work Ø Conclusion Yves Younan – Security of Memory Allocators for C and C++ 8

Doug Lea’s malloc Ø GNU lib C is based on this malloc Ø Every allocation is represented by a chunk Ø Management information stored before the chunk Ø Free chunks stored in doubly linked list of free chunks Ø Two bordering free chunks are coalesced into a larger free chunk Ø Description based on dlmalloc 2. 7. 2 Yves Younan – Security of Memory Allocators for C and C++ Ø Attack on it first described by Solar Designer 9

Doug Lea’s malloc Chunk 1 Size of prev chunk Size of chunk 1 Chunk 3 Size of chunk 2 Size of chunk 1 Forward Pointer User data Backward Pointer Old user data Chunk 2 Size of chunk 1 Size of chunk 2 Forward Pointer Backward Pointer Old user data High addr Yves Younan – Security of Memory Allocators for C and C++ Chunk 4 10

Heap Overflow (dlmalloc) Chunk 1 Size of prev chunk Size of chunk 1 Chunk 3 Size of chunk 2 Size of chunk 1 Forward Pointer Injected code User data Backward Pointer Old user data Chunk 2 Size of chunk 1 Size of chunk 2 Forward Pointer Backward Pointer Old user data Stack Return address Yves Younan – Security of Memory Allocators for C and C++ Chunk 4 Old user data 11

Off by one (dlmalloc) Ø Chunk sizes are multiples of 8 Ø Size contains two flags mmapped and prev_inuse Ø Two chunks must be next to each other (no padding) for off by one Ø Prev_size of next will be used for data Ø Overwrite 1 byte of the size and set prev_inuse to 0 and set a smaller size Ø Make a fake chunk, containing modified Yves Younan – Security of Memory Allocators for C and C++ pointers 12

Off by one (dlmalloc) Chunk 1 Size of prev chunk Size of chunk 1 Injected code Size of fake chunk User data Size of chunk 2 Forward Pointer Backward Pointer Chunk 2 Size of chunk 2 Forward Pointer Backward Pointer Old user data Yves Younan – Security of Memory Allocators for C and C++ Stack Return address 13

Dangling pointer references (dlmalloc) Chunk 1 Size of prev chunk Chunk 2 Chunk 3 Size of prev chunk Size of chunk Forward Pointer Backward Pointer Old user data User data Old user data Injected Code Stack Return address Yves Younan – Security of Memory Allocators for C and C++ 14

Doug Lea’s malloc conclusion Ø Vulnerable to: ØHeap overflow ØOff by one/five ØDouble free Ø Version 2. 8. x contains some mitigation techniques, see related work Yves Younan – Security of Memory Allocators for C and C++ 15

Overview Ø Introduction Ø Attacks Ø Memory Allocators Ø Doug Lea’s malloc (Linux) Ø CSRI/Quickfit Ø Phkmalloc (BSD) Ø Boehm’s garbage collector Ø A Safer Allocator Ø Related Work Ø Conclusion Yves Younan – Security of Memory Allocators for C and C++ 16

CSRI/Quickfit CSRI Chunk 1 Size of chunk 1 1 Inuse User data Chunk 2 Size of chunk 2 0 ‘125’ Size of chunk 1 1 Chunk 2 Old user data Previous Pointer Size of chunk 2 0 Forward Pointer Previous Pointer Old user data Next Pointer Size of chunk 2 0 High addr Yves Younan – Security of Memory Allocators for C and C++ 17

Heap Overflow (CSRI) Chunk 1 Size of chunk 1 1 Chunk 3 Size of chunk 2 Injected code User data Next Pointer Old user data Size of chunk 1 1 Chunk 2 Size of chunk 2 0 ‘125’ Old user data Previous Pointer Next Pointer Size of chunk 2 0 Previous Pointer Size of chunk 1 Chunk 4 Size of chunk 2 Forward Pointer Backward Pointer Stack Return address Yves Younan – Security of Memory Allocators for C and C++ Old user data 18

CSRI/Quickfit Conclusion Ø Very similar to dlmalloc Ø Vulnerable to heap overflow Ø No double free possible: inuse bit checked Ø No off by one/five possible: size at beginning compared with size at end Yves Younan – Security of Memory Allocators for C and C++ 19

Overview Ø Introduction Ø Attacks Ø Memory Allocators Ø Doug Lea’s malloc (Linux) Ø CSRI/Quickfit Ø Phkmalloc (BSD) Ø Boehm’s garbage collector Ø A Safer Allocator Ø Related Work Ø Conclusion Yves Younan – Security of Memory Allocators for C and C++ 20

Poul-Henning Kamp’s malloc Ø Standard Free. BSD allocator (used in other BSDs as well). Ø Takes advantage of the virtual memory system Ø Tries to minimize accessed pages Ø Makes sure that objects that are smaller or equal to a page do not span pages Ø Two layers: page layer, chunk layer Ø Attack on it first described by BBP Yves Younan – Security of Memory Allocators for C and C++ 21

Phkmalloc: Page layer Ø Manages allocation and freeing of pages Ø Information about pages allocated to the heap are stored in a page directory Ø First elements contain pointers to a linked list of page information for each possible size of chunks MALLOC_NOT_MIN Not allocated by phk Ø Other elements store information about particular E pages MALLOC_FREE Page is free MALLOC_FIRST Page is first in multipage object Yves Younan – Security of Memory Allocators for C and C++ MALLOC_FOLLOW Following page (multipage 22

Phkmalloc: page directory Page with small chunks Heap Free � Page Large multipage � � chunk Page directory Free � Page ����� � pginfo Pointer to pginfo list … Pointer to pginfo list MALLOC_FREE MALLOC_FIRST MALLOC_FOLLOW MALLOC_FREE Pointer to pginfo Yves Younan – Security of Memory Allocators for C and C++ 23

Phkmalloc: free pages Ø Free pages stored in a sorted (address) doubly linked list Ø Adjacent free pages are coalesced and only a pointer to the first page is kept Ø Free page info (pgfree) is stored using malloc (allows swapping pages out) Ø To increase performance a cache pointer is kept of a free pgfree Yves Younan – Security of Memory Allocators for C and C++ 24

Phkmalloc: free pages Page with small chunks Heap Free � Page freelist Free � Page Large multipage � � chunk pgfree next prev page end size Yves Younan – Security of Memory Allocators for C and C++ Free � Page ����� � 25

Phkmalloc: Chunk layer Ø Three different types of chunks: small, medium large Ø Large is larger than half a page Ø Large object allocation: search for contiguous free pages, set MALLOC_FIRST and MALLOC_FOLLOW Ø Small or medium chunks are rounded up to the next power of two Ø Pages only contain chunks of the same size Yves Younan – Security of Memory Allocators for C and C++ 26

Phkmalloc: pginfo Ø Pginfo is used to describe a small or medium page Ø Stored in the beginning of the page for small chunks Ø Allocated using malloc for medium chunks Ø The entry in the pagedir or small/medium pages points to pginfo Yves Younan – Security of Memory Allocators for C and C++ 27

Phkmalloc: pginfo Page with small chunks Heap � � Page directory Pointer to pginfo list … Pointer to pginfo list MALLOC_FREE MALLOC_FIRST MALLOC_FOLLOW MALLOC_FREE � � Pginfo next page size shift free total bits[] ����� � Pointer to pginfo Yves Younan – Security of Memory Allocators for C and C++ 28

Heap overflow (phkmalloc) Ø Pginfo (for medium chunks) and pgfree structs are stored together with normal chunks Ø Can be overflowed Ø We will demonstrate using pginfo Ø Make page-field point to target memory Ø Modify the bits[] array to make all chunks seem free Ø When a chunk is of that size is requested, the Yves Younan – Security Memory Allocators C and C++ 29 allocator will ofreturn thefor page-pointer as a valid

Heap overflow (phkmalloc) Chunk Lower addresses Stack Return address Page with medium chunks Pginfo next page size shift free total bits[] Yves Younan – Security of Memory Allocators for C and C++ 30

Off by one (phkmalloc) Chunk Lower addresses Fake Pginfo next page Pginfo size shift next free page total size shift bits[] free total bits[] Yves Younan – Security of Memory Allocators for C and C++ Stack Return address 31

Phkmalloc conclusion Ø Vulnerable to: Ø Heap overflow ØOff by one Ø Not vulnerable to double free: a check is done to see if the chunk is free or not Yves Younan – Security of Memory Allocators for C and C++ 32

Overview Ø Introduction Ø Attacks Ø Memory Allocators Ø Doug Lea’s malloc (Linux) Ø CSRI/Quickfit Ø Phkmalloc (BSD) Ø Boehm’s garbage collector Ø A Safer Allocator Ø Related Work Ø Conclusion Yves Younan – Security of Memory Allocators for C and C++ 33

Boehm’s Garbage Collector Ø Conservative garbage collector for C/C++ Ø Assumes values in memory are pointers if they look like pointers Ø Automatically releases memory to the system when no longer needed Ø No dangling pointer references are possible (unless programmer does explicit freeing) Ø Uses mark and sweep Yves Younan – Security of Memory Allocators for C and C++ 34

Boehm’s Garbage Collector Ø Makes a difference between small and large chunks Ø Large chunks are larger than half of page size (IA 32) and rounded up to page size Ø Small chunks are allocated in pages and the page is divided in chunks of same size Ø Allocated chunks only contain data Ø Free chunks contain pointer to the next free chunk Yves Younan – Security of Memory Allocators for C and C++ 35

Heap overflow (Boehm) Ø If attackers can overflow a chunk, they can overwrite the next pointer Ø An attacker can make the next pointer point to a target memory location Ø Eventually the allocator will return the pointer’s target as a valid chunk Ø Usually an off by four attack Yves Younan – Security of Memory Allocators for C and C++ 36

Double free (Boehm) Chunk 21 Next Yves Younan – Security of Memory Allocators for C and C++ Next 37

Boehm Conclusion Ø Vulnerable to: ØHeap overflow ØOff by one-four ØDouble free: only if the programmer explicitly frees memory (usually not necessary) Yves Younan – Security of Memory Allocators for C and C++ 38

Overview Ø Ø Ø Introduction Attacks Memory Allocators A Safer Allocator Related Work Conclusion Yves Younan – Security of Memory Allocators for C and C++ 39

Design Ø On most modern systems code and data are loaded into separate memory locations Ø We apply the same to chunk information and chunks Ø Chunk info is stored in separate contiguous memory Ø This area is protected by guard pages Ø A hashtable is used to associate chunks with chunkinfo Ø The hashtable contains pointers to a linked – Security information of Memory Allocators for C accessed and C++ 40 list. Yvesof. Younan chunk through the

Modified memory layout Text Data Bss Chunks Non-writable page Chunkinfo Hashtable Non-writable page Stack Yves Younan – Security of Memory Allocators for C and C++ 41

Dnmalloc: hashtable Ø Hashtable is stored before the stack in a mmaped area big enough to hold it Ø Each page is divided in 256 possible chunks (of 16 bytes, minimum chunk size) Ø These chunks are separated into 32 groups of 8 chunks Ø Each group has an entry in the hashtable, maximum 32 entries for every page Ø One element of the group is accessed through a linked list of max. 8 elements Yves Younan – Security of Memory Allocators for C and C++ 42

Dnmalloc: hashfunction Ø To find a chunk’s information from a chunk we do the following: Ø Substract the start of the heap from the chunk’s address Ø Shift the result 7 bits to the right: gives us the entry in the hashtable Hashtable Ø Go over the linked. Chunkinfo list till we have the. Chunkinfo correct Ptrchunk to chunkinfo Hashnext Ptr to chunkinfo Forward … Backward Size Ptr to chunkinfo Chunk Yves Younan – Security of Memory Allocators for C and C++ Chunk 43

Dnmalloc: Managing chunk information Ø A fixed area is mapped below the hashtable for chunkinfos Ø Free chunkinfos are stored in a linked list Ø When a new chunkinfo is needed the first element in the free list is used Ø If none are free a chunk is allocated from the map Ø If the map is empty we map extra memory for it (and move the guard page) Ø Chunk information is protected by guard Yves Younan – Security of Memory Allocators for C and C++ 44 pages

Dnmalloc performance overhead Spec CPU 2000 results for dlmalloc and dnmalloc (13 runs on 8 identical pcs (P 4 2. 8 ghz, 512 mb) = 104 runs) Progra m Dlmalloc runtime Dnmalloc runtime Overhead percentage gzip 253 +- 0 255. 98 +- 0. 01 1. 18% vpr 360. 93 +- 0. 16 360. 55 +- 0. 13 -0. 11% gcc 153. 93 +- 0. 05 154. 76 +- 0. 04 0. 54% mcf 287. 19 +- 0. 07 290. 09 +- 0. 07 1. 01% crafty 253 +- 0 254 +- 0 0. 40% parser 346. 95 +- 0. 02 346. 61 +- 0. 05 -0. 10% eon 771. 05 +- 0. 13 766. 55 +- 0. 11 -0. 58% perlbmk 243. 20 +- 0. 04 253. 51 +- 0. 05 4. 24% gap 184. 07 +- 0. 02 184 +- 0 -0. 04% vortex 250 +- 0 258. 79 +- 0. 04 3. 52% bzip 2 361. 64 +- 0. 05 363. 26 +- 0. 07 0. 45% 0. 43 for 513. 27 0. 41 Yves Younantwolf – Security of 522. 48 Memory+Allocators C and+C++ -1. 76% 45

Overview Ø Ø Ø Introduction Attacks Memory Allocators A Safer Allocator Related Work Ø Robertson et al. heap protector Ø Contrapolice Ø Glibc 2. 3. 5 integrity checks Ø Conclusion Yves Younan – Security of Memory Allocators for C and C++ 46

Robertson’s heap protector Ø Checksum stored in every chunk’s header Ø Checksum encrypted with a global readonly random value Ø Checksum added when allocated, checked when freed Ø Could be bypassed if memory leaks exist Ø Dlmalloc 2. 8. x implements a slightly modified version of this Yves Younan – Security of Memory Allocators for C and C++ 47

Overview Ø Ø Ø Introduction Attacks Memory Allocators A Safer Allocator Related Work Ø Robertson et al. heap protector Ø Contrapolice Ø Glibc 2. 3. 5 integrity checks Ø Conclusion Yves Younan – Security of Memory Allocators for C and C++ 48

Contrapolice Ø Protects chunks by placing canaries (random) before and after the chunk Ø Before exiting from a copy function, it checks if the canary before matches the canary after Ø Could be bypassed if the canary value is leaked Yves Younan – Security of Memory Allocators for C and C++ 49

Overview Ø Ø Ø Introduction Attacks Memory Allocators A Safer Allocator Related Work Ø Robertson et al. heap protector Ø Contrapolice Ø Glibc 2. 3. 5 integrity checks Ø Conclusion Yves Younan – Security of Memory Allocators for C and C++ 50

Glibc 2. 3. 5 integrity checks Ø Adds several integrity checks (unlink, free, realloc and other places): Ø Before unlink, check: p->fd->bk == p->bk->fd == p Ø Before free: pointer must be smaller than its negative size (and correctly aligned)� Ø Various other checks for chunktypes that are handled differently Ø Can be bypassed in some cases Ø Attack discovered by Phantasmal Phantasmagoria Ø Describes 5 techniques for bypassing glibc Yves Younan – Security of Memory Allocators for C and C++ integrity checks 51

Overview Ø Ø Ø Introduction Attacks Memory Allocators A Safer Allocator Related Work Conclusion Yves Younan – Security of Memory Allocators for C and C++ 52

Conclusion Ø Many allocators ignore security issues Ø Safer allocators are not necessarily much slower Ø Our approach still has an important limitation: only chunk information is protected, not what is in the chunk Ø This work is part of larger research where other important areas in memory are also separated from normal data (currently we’re finishing up on a stackbased countermeasure) Ø Which is part of my real research: a more methodical approach to designing countermeasures Yves Younan – Security of Memory Allocators for C and C++ 53

Conclusion Ø 2 papers associated with this talk: Ø Yves Younan, Wouter Joosen, Frank Piessens and Hans Van den Eynden. Security of Memory Allocators for C and C++. Technical Report CW 419, Departement Computerwetenschappen, Katholieke Universiteit Leuven, July 2005 Ø Yves Younan, Wouter Joosen and Frank Piessens. Applying machinemodel-aided countermeasure design to improve memory allocator security. 22 nd Chaos Communication Congress, Berlin, Germany, December 2005. Ø Dnmalloc implementation is available on http: //www. fortknox. org (also has other papers) Yves Younan – Security of Memory Allocators for C and C++ 54