Runtime Protection via Dataflow Flattening Bertrand Anckaert Ghent

Runtime Protection via Dataflow Flattening Bertrand Anckaert Ghent University/ Boston Consulting Group Mariusz H. Jakubowski Ramarathnam Venkatesan Chit Wei (Nick) Saw Microsoft Research Redmond, WA (USA) The Third International Conference on Emerging Security Information, Systems and Technologies SECURWARE 2009 June 18 -23, 2009 – Athens/Glyfada, Greece

Introduction • Software protection – Complicate reverse engineering and tampering. – Enforce execution as intended by developer. – DRM, licensing, anti-malware, OS security, etc. • Dataflow analysis – Track flow of data through program. – Locate and tamper “interesting” data. • Goals of our work: – Develop methods against malicious dataflow analysis. – Study dataflow flattening as an element of comprehensive protection frameworks. SECURWARE 2009 June 18 -23, 2009 2

Overview Protecting data operations • • • Introduction Background Dataflow flattening Implementation and experiments Applications Conclusion SECURWARE 2009 June 18 -23, 2009 3

Background • Oblivious RAMs [Goldreich and Ostrovsky ’ 96] – Randomized memory-access patterns – Each fetch/store replaced by many fetch/stores – Cannot infer program operation from memory accesses • Control-flow flattening – Program’s CFG converted to flat (two-level) graph – Cannot infer control-flow structure from execution on the flat graph SECURWARE 2009 June 18 -23, 2009 4

Overview Protecting data operations • • • Introduction Background Dataflow flattening Implementation and experiments Applications Conclusion SECURWARE 2009 June 18 -23, 2009 5

Dataflow Flattening • Two main aspects: – Making dataflow graph appear complete – Randomizing memory-access patterns • Informally: – “Every variable affects every other variable. ” – “Program accesses memory at random. ” SECURWARE 2009 June 18 -23, 2009 6

Flattening Dataflow Graphs • Basic idea: Make dataflow graph appear complete. – Every variable affects all other variables. – Cannot infer useful variable dependencies. A B X Y • Data-centric analog of control-flow flattening SECURWARE 2009 June 18 -23, 2009 7

Flattening Dataflow Graphs • Basic idea: Make dataflow graph appear complete. – Every variable affects all other variables. – Cannot infer useful variable dependencies. A B X Y • Data-centric analog of control-flow flattening SECURWARE 2009 June 18 -23, 2009 8

Dataflow Flattening via an MMU Heap Program Memory Management Unit

MMU • Software-based Memory Management Unit: – Periodic reordering of heap data – Migration of variables from stack to heap – Pointer masking • Variable references redirected through MMU SECURWARE 2009 June 18 -23, 2009 10

MMU Operation • Heap subdivided into encrypted pages (e. g, 4 KB). • Upon each access of a heap page: – Retrieve n extra pages with probability 1/p. – Randomly shuffle the (expected) 1+n/p pages. – Re-salt and re-encrypt each page. SECURWARE 2009 June 18 -23, 2009 11

Security Analysis • Security analyzed via practical metrics • Confusion factor C as a metric – Define C as the number of possible places for a page in memory. – Let N = total number of memory pages. – Oblivious RAMs: C = N after each memory access. – Our approach: C converges to N as accesses occur. SECURWARE 2009 June 18 -23, 2009 12

Practical Issues • Most program variables are stack-based. – Solution: Migrate variables from stack to heap. – Explicitly allocate and free heap memory when entering and exiting stack frames. • Pointers can reveal access patterns. – Solution: Scramble pointers. – Only MMU knows mapping between addresses and variables. SECURWARE 2009 June 18 -23, 2009 13

Overview Protecting data operations • • • Introduction Background Dataflow flattening Implementation and experiments Applications Conclusion SECURWARE 2009 June 18 -23, 2009 14

Implementation • Tool for transforming C programs – Based on Phoenix compiler infrastructure – Compiler backend plug-in – Instrumentation of Phoenix IR • Transformations – Interception and custom implementation of heap operations (malloc, free, etc. ) – Conversion of stack variables to heap variables – Pointer scrambling (encryption) SECURWARE 2009 June 18 -23, 2009 15

Experimental Results Performance impact (slowdown) Selected SPEC benchmarks (compression, optimization, QCD, chess, fluid dynamics) Simple algorithms (pseudorandom-number generation, summing a list of integers) SECURWARE 2009 June 18 -23, 2009 16

Overview Protecting data operations • • • Introduction Background Dataflow flattening Implementation and experiments Applications Conclusion SECURWARE 2009 June 18 -23, 2009 17

Applications • Software protection – Anti-malware systems – Licensing, DRM, product activation, etc. – Defenses against information-extraction and sidechannel attacks • More comprehensive tools – Element of broader protection strategies – Means of realizing “engineering assumptions” needed by some security models SECURWARE 2009 June 18 -23, 2009 18

Conclusion • Dataflow flattening – Makes dataflow graph appear complete. – Randomizes memory-access patterns. – Complicates inference of algorithms from their data operations. • Future directions – Dataflow flattening as part of more comprehensive systems – Security analysis via models and metrics SECURWARE 2009 June 18 -23, 2009 19