Software Vulnerabilities Definition Classification and Prevention Reverse Engineering

Software Vulnerabilities: Definition, Classification, and Prevention Reverse Engineering (Software Security) © SERG

Overview • What are software vulnerabilities? • Types of vulnerabilities – Buffer Overflows – Format String Vulnerabilities • How to find these vulnerabilities and prevent them? • Classes of software vulnerabilities Reverse Engineering (Software Security) © SERG

What Are Software Vulnerabilities? • A software vulnerability is an instance of a fault in the specification, development, or configuration of software such that its execution can violate the (implicit or explicit) security policy. Software Vulnerabilities © SERG

Types of Software Vulnerabilities • Buffer overflows – Smash the stack – Overflows in setuid regions • Heap overflows • Format string vulnerabilities Reverse Engineering (Software Security) © SERG

Buffer Overflows Definition, Examples, and Defenses Reverse Engineering (Software Security) © SERG

What is a Buffer? • Example: – A place on a form to fill in last name where each character has one box. • “Buffer” is used loosely to refer to any area of memory where more than on piece of data is stored. Reverse Engineering (Software Security) © SERG

Buffer Overflows [5] • The most common form of security vulnerability in the last 10 years – 1998: 2 out of 5 “remote to local” attacks in Lincoln Labs Intrusion Detection Evaluation were buffer overflows. – 1998: 9 out of 13 CERT advisories involved buffer overflows. – 1999: at least 50% of CERT advisories involved buffer overflows. Reverse Engineering (Software Security) © SERG

How Does a Buffer Overflow Happen? • Reading or writing past the end of the buffer overflow • As a result, any data that is allocated near the buffer can be read and potentially modified (overwritten) – A password flag can be modified to log in as someone else. – A return address can be overwritten so that it jumps to arbitrary code that the attacker injected (smash the stack) attacker can control the host. Reverse Engineering (Software Security) © SERG

Two Steps • Arrange for suitable code to be available in the program’s address space (buffer) – Inject the code – Use code that is already in the program • Overflow the buffer so that the program jumps to that code. Reverse Engineering (Software Security) © SERG

Inject the Code • Use a string as input to the program which is then stored in a buffer. • String contains bytes that are native CPU instructions for attacked platform. • Buffer can be located on the stack, heap, or in static data area. Reverse Engineering (Software Security) © SERG

Code Already in Program • Only need to parameterize the code and cause the program to jump to it. • Example: – Code in libc that executes “exec(arg)”, where arg is a string pointer argument, can be used to point to “/bin/sh” and jump to appropriate instructions in libc library. Reverse Engineering (Software Security) © SERG

Jump to Attack Code • Activation record – stack smashing attack • Function pointer • Longjpm(3) buffer Reverse Engineering (Software Security) © SERG

Memory Regions [4] Reverse Engineering (Software Security) © SERG

Code/text Segment • Static • Contains code (instructions) and read-only data • Corresponds to text section of executable file • If attempt to write to this region segmentation violation Reverse Engineering (Software Security) © SERG

Data Segment • Permanent data with statically known size • Both initiated and uninitiated variables • Corresponds to the data-bss sections of the executable file • brk(2) system call can change data segment size • Not enough available memory process is blocked and rescheduled with larger memory Reverse Engineering (Software Security) © SERG

Heap • Dynamic memory allocation • malloc() in C and new in C++ More flexibility • More stable data storage – memory allocated in the heap remains in existence for the duration of a program • Data with unknown lifetime – global (storage class external) and static variables Reverse Engineering (Software Security) © SERG

Stack – I [3] • Provides high-level abstraction – Allocates local variables when a function gets called (with known lifetime) – Passes parameters to functions – Returns values from functions • Push/Pop operations (LIFO) – implemented by CPU • Size – dynamically adjusted by kernel at runtime Reverse Engineering (Software Security) © SERG

Stack – II • Stack Pointer (SP) – TOP of stack (or next free available address) • Fixed address – BOTTOM of stack • Logical Stack Frame (SF) – contains parameters to functions, local variables, data to recover previous SF (e. g: instruction pointer at time of function call) • Frame Pointer (FP)/local Base Pointer (BP) – Beginning of Activation Record (AR), used for referencing local variables and parameters (accessed as offsets from BP) Reverse Engineering (Software Security) © SERG

Activation Record [5] • Contains all info local to a single invocation of a procedure – – – Return address Arguments Return value Local variables Temp data Other control info Reverse Engineering (Software Security) © SERG

Accessing an Activation Record (AR) • Base pointer: beginning of AR – Arguments are accessed as offsets from bp • Environment pointer: pointer to the most recent AR (usually a fixed offset from bp) • Stack pointer: top of AR stack – Temporaries are allocated on top on stack Reverse Engineering (Software Security) © SERG

When a Procedure is Called • Previous FP is saved • SP is copied into FP new FP • SP advances to reserve space for local variables • Upon procedure exit, the stack is cleaned up Reverse Engineering (Software Security) © SERG

Function Pointer • Find a buffer adjacent to function pointer in stack, heap or static data area • Overflow buffer to change the function pointer so it jumps to desired location • Example: attack against superprobe program - Linux Reverse Engineering (Software Security) © SERG

Longjpm Buffer • setjmp(buffer) to set a checkpoint • longjmp(buffer) to go back to checkpoint • Corrupt state of buffer so that longjmp(buffer) jumps to the attack code instead Reverse Engineering (Software Security) © SERG

Example pushl $3 void function(int a, int b, int c) pushl $2 { pushl $1 char buffer 1[5]; call function char buffer 2[10]; } void main() { function(1, 2, 3); } pushl %ebp movl %esp, %ebp subl $20, %esp Reverse Engineering (Software Security) © SERG

Buffer Overflow Example void main() { int x; x = 0; function(1, 2, 3); x = 1; printf("%dn", x); } void function(int a, int b, int c) { char buffer 1[5]; char buffer 2[10]; int *ret; ret = buffer 1 + 12; (*ret) += 8; } Reverse Engineering (Software Security) © SERG

Result of Program • Output: 0 • Return address has been modified and the flow of execution has been changed • All we need to do is place the code that we are trying to execute in the buffer we are overflowing, and modify the return address so it points back to buffer Reverse Engineering (Software Security) © SERG

Example [6] char shellcode[ ] = “xebx 1 fx 5 ex 89x 76x 08x 31xc 0x 88x 46x 07x 89x 46x 0 cxb 0x 0 b” “x 89xf 3x 8 dx 4 ex 08x 8 dx 56x 0 cxcdx 80x 31xdbx 89xd 8x 40xcd” “x 80xe 8xdcxffxff/bin/sh”; char large_string[128]; void main() { char buffer[96]; int i; long *long_ptr = (long *) large_string; /* long_ptr takes the address of large_string */ /* large_string’s first 32 bytes are filled with the address of buffer */ for (i = 0; i < 32; i++) *(long_ptr + i) = (int) buffer; /* copy the contents of shellcode into large_string */ for (i = 0; i < strlen(shellcode); i++) large_string[ i ] = shellcode[ i ]; /* buffer gets the shellcode and 32 pointers back to itself */ strcpy(buffer, large_string); } Reverse Engineering (Software Security) © SERG

Example Illustrated [6] Process Address Space argc user stack RA sfp long_ptr i buffer large_string[128] Shellcode[] heap bss Reverse Engineering (Software Security) © SERG

Buffer Overflows Defenses • Writing correct code (good programming practices) • Debugging Tools • Non-executable buffers • Array bounds checking • Code pointer integrity checking (e. g. , Stack. Guard) Reverse Engineering (Software Security) © SERG

Problems with C • Some C functions are problematic – Static size buffers – Do not have built-in bounds checking • While loops – Read one character at a time from user input until end of line or end of file – No explicit checks for overflows Reverse Engineering (Software Security) © SERG

Some Problematic C Functions [2] Function Severity Solution: Use gets Most Risky fgets(buf, size, stdin) strcpy, strcat Very Risky strncpy, strncat sprintf, vsprintf Very Risky snprintf, vsnprintf or precision specifiers scanf family Very Risky precision specifiers or do own parsing realpath, syslog Very Risky (depending on implementation) Maxpathlen and manual Very Risky (depending on implementation) Truncate string inputs to reasonable size getopt, getopt_long, getpass checks Reverse Engineering (Software Security) © SERG

Good Programming Practices – I DO NOT USE: Instead USE: void main( ) { char buf [40]; gets(buf); } void main( ) { char buf [40]; fgets(buf, 40, stdin); } Reverse Engineering (Software Security) © SERG

Good Programming Practices – II DO NOT USE: Instead USE: void main() { char buf[4]; char src[8] = "rrrrr"; strcpy(buf, src); } if (src_size >= buf_size) { cout<< "error"; return(1); } else { strcpy(buf, src); } OR strncpy(buf, src, buf_size - 1); buf[buf_size - 1] = ''; Reverse Engineering (Software Security) © SERG

Debugging Tools • More advanced debugging tools – Fault injection tools – inject deliberate buffer overflow faults at random to search for vulnerabilities – Static analysis tools – detect overflows • Can only minimize the number of overflow vulnerabilities but cannot provide total assurance Reverse Engineering (Software Security) © SERG

Non-executable Buffers [7, 8] • Make data segment of program’s address space non-executable attacker can’t execute code injected into input buffer (compromise between security and compatibility) Reverse Engineering (Software Security) © SERG

Non-executable Buffers [7, 8] • If code already in program, attacks can bypass this defense method • Kernel patches (Linux and Solaris) – make stack segment non-executable and preserve most program compatibility Reverse Engineering (Software Security) © SERG

Array Bounds Checking • Attempts to prevent overflow of code pointers • All reads and writes to arrays need to be checked to make sure they are within bounds (check most array references) – – Campaq C compiler Jones & Kelly array bound checking Purify memory access checking Type-safe languages (e. g. , Java) Reverse Engineering (Software Security) © SERG

Code Pointer Integrity Checking • Attempts to detect that a code pointer has been corrupted before it is de-referenced • Overflows that affect program state components other than code pointer will succeed • Offers advantages in performance, compatibility with existing code and implementation effort – Hand-coded stack introspection – Stack. Guard Point. Guard Reverse Engineering (Software Security) © SERG

Stack. Guard • Compiler technique that provides protection by checking the return address in AR • When detects an attack causes app to exit, rather than yielding control to attacker – Terminator canary – Random canary Reverse Engineering (Software Security) © SERG

Heap Overflows • Harder to exploit, yet still common • Need to know which variables are security critical • Cause a buffer overflow to overwrite the target variables (generally buffer needs to have lower address) Reverse Engineering (Software Security) © SERG

Example void main(int argc, char **argv) { char *super_user = (char *)malloc(sizeof(char)*9); char *str = (char *)malloc(sizeof(char)*4); char *tmp; super_user = super_user - 40; strcpy(super_user, "viega"); if (argc > 1) strcpy(str, argv[1]); else strcpy(str, "xyz"); tmp = str; while(tmp <= super_user + 12) { printf("%p: %c (0 x%x)n", tmp, isprint(*tmp) ? *tmp : '? ', (unsigned int)(*tmp)); tmp+=1; } } Reverse Engineering (Software Security) © SERG

Output Reverse Engineering (Software Security) © SERG

Output After Overflow Reverse Engineering (Software Security) © SERG

Format String Vulnerabilities I [9] • Caused by design misfeatures in the C standard library combined with problematic implementation (bad programming habits!) • printf family (syslog, printf, fprintf, sprintf, and snprintf): format string tells the function the type and sequence of arguments to pop and the format for output • Hostile input can be passed directly as the format string for calls to printf functions Reverse Engineering (Software Security) © SERG

Format String Vulnerabilities II • Attacker can write arbitrary values to (almost any) arbitrary addresses in memory – Overwrite a stored UID for a program that drops and elevates privileges – Overwrite an executed command – Overwrite a return address so it points to a buffer with shell code in it OWN the program Reverse Engineering (Software Security) © SERG

Format String – By Example fmtme. c int main(int argc, char **argv) { char buf[100]; int x; if(argc != 2) exit(1); x = 1; snprintf(buf, sizeof buf, argv[1]); buf[sizeof buf - 1] = 0; printf("buffer (%d): %sn", strlen(buf), buf); printf("x is %d/%#x (@ %p)n", x, x, &x); return 0; } Reverse Engineering (Software Security) © SERG

Stack Layout of Program • When snprintf function is called: Reverse Engineering (Software Security) © SERG

Exploiting the Program I • Test 1: %. /fmtme "hello world“ buffer (11): hello world x is 1/0 x 1 (@0 x 804745 c) Reverse Engineering (Software Security) © SERG

Exploiting the Program II • Test 2: print integers on stack above format string %. /fmtme "%x %x“ buffer (15): 1 f 31 1031 3133 x is 1/0 x 1 (@0 x 804745 c) • 4 values of output: the variable x and three 4 -byte integers taken from the uninitialized buf variable Reverse Engineering (Software Security) © SERG

Exploiting the Program III • Test 3: control the values stored in the buffer %. /fmtme "aaaa %x %x“ buffer (15): aaaa 1 6161 x is 1/0 x 1 (@0 x 804745 c) • The 4 'a' characters we provided were copied to the start of the buffer and interpreted by snprintf as an integer argument with the value 0 x 6161 ('a' is 0 x 61 in ASCII) Reverse Engineering (Software Security) © SERG

Exploiting the Program V Test 4: %. /fmtme "x 58x 74x 08%d%n" buffer (5): X 1 x is 5/x 05 (@0 x 8047458) Reverse Engineering (Software Security) © SERG

Exploiting the Program V (contd. ) • snprintf copies the first 4 bytes into buf, scans the "%d" format and prints out the value of x, and reaches the "%n" directive • Pulls the next value off the stack, which comes from the first 4 bytes of buf. These 4 bytes have just been filled with "x 58x 74x 08", or, 0 x 08047458 • Writes amount of bytes output so far, 5, into this address Reverse Engineering (Software Security) © SERG

Solutions • • Remove the %n feature Permit Only Static Format Strings Count the Arguments to printf Format. Guard Reverse Engineering (Software Security) © SERG

References [1] https: //www. cerias. Purdue. edu/techreportsssl/public/98 -09. pdf [2] Building Secure Software, Viega and Mc. Graw [3] Smashing the Stack for Fun and Profit, Aleph One [4] Stony Brook State University of New York (http: //www. cs. sunysb. edu/~cse 304) Reverse Engineering (Software Security) © SERG

References [5] Buffer Overflows: Attacks and Defenses for the Vulnerability of the Decade (http: //www. cse. ogi. edu/DISC/projects/immunix [6]www. cs. utexas. edu/users/deepakku/security/secur ity. ppt [7]http: //www. cse. ogi. edu/DISC/projects/immunix/S tack. Guard/usenixsc 98_html/node 21. html Reverse Engineering (Software Security) © SERG

References [8]http: //docs. sun. com/db/doc/8164883/6 mb 2 joasj? a=view [9] Format String Attacks, Tim Newsham, Guardent, Inc. 2000 Reverse Engineering (Software Security) © SERG

Other Useful Resources • "How to Write Buffer Overflows" by "Mudge“, 1997 • "Stack Smashing Vulnerabilities in the UNIX Operating System" by Nathan P. Smith, 1997 • "The Tao of Windows Buffer Overflows" by "Dil. Dog“, 1998 Reverse Engineering (Software Security) © SERG

Gemini: Using Program Transformation to Secure C Programs Against Buffer Overflows Reverse Engineering (Software Security) © SERG

Overview • • • Motivation TXL Transformation process Case study results Conclusion & future work Reverse Engineering (Software Security) © SERG

Motivation • Stack buffer overflows most common security vulnerability found in C code • The most popular e-mail, web and DNS software is open-source C code • Stack buffer overflows approx. account for 50% of security vulnerabilities since 1997 Reverse Engineering (Software Security) © SERG

Motivation • Stack Buffer Overflows: – Strings grow opposite of stack – Return address is above strings in memory – Overwriting a string can overwrite return address • Prevention – Transforming local arrays to “pointer-to-arrays” in a way that: • Relieves the need for programmer intervention • Relieves the need for specific compilers • Minimizes run-time overhead Reverse Engineering (Software Security) © SERG

Stack Buffer Overflows void func(char *str 1, char *str 2) { char buf[80] ; … strcpy(buf, str 1) ; } Reverse Engineering (Software Security) © SERG

In a Nutshell • Transform every array that occurs in a function to a pointer • Preserve the original semantics – Both good and bad (where possible) • Pointer arithmetic should still work • Returning de-allocated memory should still work Reverse Engineering (Software Security) © SERG

In a Nutshell Array Pointer-to-Array Reverse Engineering (Software Security) © SERG

TXL • Text transformation tool – Provide TXL with an EBNF grammar, a set of transformations for that grammar and input text • Generates scanner and parser at run-time • After applying transformations, outputs resulting text • Strongly typed – Guarantees Syntactic equivalence, but not semantic equivalence Reverse Engineering (Software Security) © SERG

TXL Reverse Engineering (Software Security) [2001 Cordy] ©J. SERG

TXL example • Objective – Replace occurrences of strcpy(s 1, s 2)by strncpy(s 1, s 2, sizeof(s 1)) • TXL Rule rule fix. Strcpy Declarations [repeat compound_statement_body] replace $ [postfix_expression] strcpy '( To. Buf [unary_expression] ', From. Buf [argument_expression] ') where not Declarations [check. Pointer To. Buf] by strncpy '( To. Buf ', From. Buf ', sizeof To. Buf ') end rule Reverse Engineering (Software Security) © SERG

A Parse Tree #include <stdio. h> #include <string. h> void main(int argc, char **argv) { char foo[5]; char *ptr = strcpy(foo, argv[0]); printf("%sn", ptr); } Reverse Engineering (Software Security) © SERG

TXL Transformation • Assumptions – Globals, pointers, static and extern variables are not vulnerable to stack buffer overflows – We will only be given compileable C code • • This is sometimes not the case with systems that use the autoconf package Transformation Steps: 1. 2. 3. 4. 5. 6. Declaration expansion typedef Flattening Declaration Transformation sizeof Alias Declarations Add free and Transform return and sizeof Initialization Functions Reverse Engineering (Software Security) © SERG

1. Declaration expansion ¨Makes transformation process easier Reverse Engineering (Software Security) © SERG

2. typedef Flattening • Removes type aliases • Distributes aliased array dimensions • Provides a way to differentiate statements from declarations – C is ambiguous about declarations and statements Reverse Engineering (Software Security) © SERG

Declaration/Statement Ambiguity void main() { typedef char buf[5]; buf (s); printf (s); } • This ambiguity makes some pre-processing impossible Reverse Engineering (Software Security) © SERG

3. Declaration Transformation A. Pre-process Variable Declarations – Adding parentheses around a declarator doesn’t change the semantics, but it does change the syntax – Remove extraneous parentheses • • Simplifies declarations Groups array dimensions B. Transform Arrays to Pointers C. Generate Pointer Initialization Reverse Engineering (Software Security) © SERG

A. Pre-process Variable Declarations Ambiguity: parsing declarations buf(s); and printf(s); can result in the same parse tree Flatten typedef Remove extraneous parentheses Reverse Engineering (Software Security) © SERG

B. Transform Arrays to Pointers • Transform array declarations to “pointer to array” • Create an initialization functions to each array • Create temporary variables to hold the original array sizes In ISO C 99, array sizes can be variable sized (an expression) Reverse Engineering (Software Security) © SERG

4. sizeof Alias Declarations • sizeof allows for expressions, not just types – Memory assigned to pointer is architecture dependent, always constant – Violates semantic preservation Output= 80 Transform Arrays to Pointers Output= 4 Reverse Engineering (Software Security) © SERG

sizeof Alias Declarations ¨ Generate new declaration with original array dimensions to replace all sizeof references Reverse Engineering (Software Security) © SERG

5. Add free and Transform return and sizeof • Fix return statements and insert free statements • A return may reference a transformed array: – If necessary, store expression in temporary variable, insert free statements for current scope, return temporary variable • 3 cases – End of scope without a return statement – return with an expression – return without an expression Reverse Engineering (Software Security) © SERG

Add free and Transform return and sizeof 1. Fix return statements and insert free statements 2. Fix sizeof constants Reverse Engineering (Software Security) © SERG

6. Insert Initialization Functions • Functions inserted at bottom of file, prototypes at top – Type flattening step removes possibility of a user defined type occurring in prototype, before actual declaration Reverse Engineering (Software Security) © SERG

Case Study Results (Most Recent) Program Arrays Fixed (KLo. C) Time Increase ( run-time) apache 226 (~94) 11. 1% (0. 001 s) bind 2408 (~298) 0. 5% (2. 690 s) bison 56 (~21) 1. 8% (0. 458 s) find 47 (~17) 0% (0. 000 s) flex 29 (~18) 0% (0. 000 s) openssh 513 (~56) 4. 3% (7. 233 s) which 2 (~2) 0% (0. 000 s) whois 7 (~1) 32. 4% (. 153 s) Reverse Engineering (Software Security) Regression Passed N/A Yes Yes N/A N/A © SERG

Conclusions and Future work • Limitations & Future work: – Extends transformation algorithm for struct declarations: • Arrays of structs • Arrays within structs (done) • Benefits – Current and future stack overflow attacks are guaranteed to fail when used against a transformed program Reverse Engineering (Software Security) © SERG

Characterizing the ‘Security Vulnerability Likelihood’ of Software Functions Reverse Engineering (Software Security) © SERG

Talk Outline • Section I – Manager/director detail – Why are we here? • Section II – User detail – What can we do? • Section III – Scientist detail – How do we do it? Reverse Engineering (Software Security) © SERG

What is a Software Security Vulnerability? • A software security vulnerability is a fault in the specification, implementation, or configuration of a software system whose execution can violate an explicit or implicit security policy. Reverse Engineering (Software Security) © SERG

Section I • Manager level detail • Software vulnerabilities are a problem; how can we avoid them? Reverse Engineering (Software Security) © SERG

How are software security vulnerabilities avoided? • Safe programming languages – Type safe languages • Adequate software design – Security Policies • Thorough testing – Testing with a high percentage of code coverage • Security auditing – Design – Implementation Reverse Engineering (Software Security) © SERG

Auditing for Software Security • Pre-release software development practices unlikely to change – Safe languages – Adequate software design – Thorough testing • Post-release auditing typically used if warranted Reverse Engineering (Software Security) © SERG

Security Auditing Issues • Large scale auditing infeasible due code size • Good source code will have 1 -3 bugs for every 100 lines of code [Beizer] • Security Auditors need to find the software security vulnerabilities in the bugs • Security audits would benefit from a tool that identify areas that are likely vulnerable Reverse Engineering (Software Security) © SERG

Improving the Security Audit • Provide a tool built on statistical measurements • Cut down the amount of code needed to be reviewed • Ensure high degree of accuracy • Bottom line: Complexity reduction Reverse Engineering (Software Security) © SERG

Section II • User level detail • How can our tools and research help the security auditor? Reverse Engineering (Software Security) © SERG

The FLF Hypothesis FL F Input FL F • Our hypothesis is that a small percentage of functions near a source of input are more likely to contain software security vulnerabilities • These functions are known as Front Line Functions Reverse Engineering (Software Security) © SERG

Front Line Functions • 60% Code Reduction • No auditor time required • High degree of accuracy Reverse Engineering (Software Security) © SERG

Discovering the FLF measurement • Collect software systems with known vulnerabilities • Perform detailed static analyses of software systems • Calculate areas of likely vulnerability from information gathered during static analyses • Build tools around these calculations Reverse Engineering (Software Security) © SERG

How are these tools used? • Run static analysis tools on source code • A database of code facts is created • The database is used to find the likely vulnerable functions • The likely vulnerable functions are outputted and ranked by proximity to input source Reverse Engineering (Software Security) © SERG

Case Study: Open. SSH GOAL: FLF Finder identifies large percentage of modified functions Open. SSH 3. 1 Functions where privilege separation will be implemented Code that was not change between revisions Open. SSH 3. 2. 3 Code changed to implement privilege separation 82% of changed functions identified by FLF Finder Reverse Engineering (Software Security) © SERG

Section III • Scientist level detail • What is the process for creating the likely vulnerable measurement • How do we identify FLFs Reverse Engineering (Software Security) © SERG

The FLF Hypothesis (revisited) FL F Input FL F • Our hypothesis is that a small percentage of functions near a source of input are more likely to contain software security vulnerabilities • These functions are known as Front Line Functions Reverse Engineering (Software Security) © SERG

Validating the FLF Hypothesis 1. Gather a set of vulnerable software systems (Experimental Systems). 2. Perform experiments on these systems with static analysis tools (GAST-MP & SGA) 3. Identify all functions which initiate user input transactions (Inputs) 4. Identify all functions which are vulnerable (Targets) 5. Calculate FLF Density (FLF Density) 6. Test the FLF Density on a set of control systems (Verification) 7. Create a tool that utilizes the FLF Density (FLF Finder). Reverse Engineering (Software Security) © SERG

Experimental Systems • 30 open source systems • 31 software security vulnerabilities • Each system has a single patch file which addresses one security vulnerability • Most recovered from Redhat’s source RPM distribution due to their incremental nature Reverse Engineering (Software Security) © SERG

Validating the FLF Hypothesis 1. Gather a set of vulnerable software systems (Experimental Systems). 2. Perform experiments on these systems with static analysis tools (GAST-MP & SGA) 3. Identify all functions which initiate user input transactions (Inputs) 4. Identify all functions which are vulnerable (Targets) 5. Calculate FLF Density (FLF Density) 6. Test the FLF Density on a set of control systems (Verification) 7. Create a tool that utilizes the FLF Density (FLF Finder). Reverse Engineering (Software Security) © SERG

GAST-MP & SGA • GNU Abstract Syntax Tree Manipulation Program ( GAST-MP ) – Source Code Analysis tool – Operates on G++’s Abstract Syntax Tree (AST) • AST can be outputted with the –fdump-tree-flag – Creates a repository of code facts • System Graph Analyzer ( SGA ) – – – Operates on the code fact repository Identifies Inputs and Targets Performs invocation analysis Calculates FLF Density Analysis of Categorical Graphs Reverse Engineering (Software Security) © SERG

Validating the FLF Hypothesis 1. Gather a set of vulnerable software systems (Experimental Systems). 2. Perform experiments on these systems with static analysis tools (GAST-MP & SGA) 3. Identify all functions which initiate user input transactions (Inputs) 4. Identify all functions which are vulnerable (Targets) 5. Calculate FLF Density (FLF Density) 6. Test the FLF Density on a set of control systems (Verification) 7. Create a tool that utilizes the FLF Density (FLF Finder). Reverse Engineering (Software Security) © SERG

Finding Inputs • An Input is a function which contains reads in external user input – For example, read • A list of external function calls were compiled to properly identify Inputs • This list could be modified to contain application specific library calls Reverse Engineering (Software Security) © SERG

Validating the FLF Hypothesis 1. Gather a set of vulnerable software systems (Experimental Systems). 2. Perform experiments on these systems with static analysis tools (GAST-MP & SGA) 3. Identify all functions which initiate user input transactions (Inputs) 4. Identify all functions which are vulnerable (Targets) 5. Calculate FLF Density (FLF Density) 6. Test the FLF Density on a set of control systems (Verification) 7. Create a tool that utilizes the FLF Density (FLF Finder). Reverse Engineering (Software Security) © SERG

Finding Targets • A Target is any function that contains a known vulnerability • Targets are found by matching code facts on subtractive lines in a patch file with code facts in the repository generated by GAST-MP --- channels. c 27 Feb 2002 21: 23: 13 -0000 +++ channels. c 4 Mar 2002 19: 37: 58 -0000 @@ -146, 7 +146, 7 @@ { Channel *c; + 1. 170 1. 171 if (id < 0 || id > channels_alloc) { if (id < 0 || id >= channels_alloc) { log("channel_lookup: %d: bad id", id); return NULL; } Reverse Engineering (Software Security) © SERG

Validating the FLF Hypothesis 1. Gather a set of vulnerable software systems (Experimental Systems). 2. Perform experiments on these systems with static analysis tools (GAST-MP & SGA) 3. Identify all functions which initiate user input transactions (Inputs) 4. Identify all functions which are vulnerable (Targets) 5. Calculate FLF Density (FLF Density) 6. Test the FLF Density on a set of control systems (Verification) 7. Create a tool that utilizes the FLF Density (FLF Finder). Reverse Engineering (Software Security) © SERG

FLF Density 1. 2. 3. 4. Create entire call graph G Transform G in DAG Label Input and Target Nodes Calculate invocation paths between Input and Target combinations and measure length 5. Calculate FLF Density by normalizing path length by function cardinality 6. For each system choose the largest FLF Density Reverse Engineering (Software Security) © SERG

Experimental Results Reverse Engineering (Software Security) © SERG

Experimental Results • Sample mean FLF Density 2. 87% – Therefore, a very small number of functions are actually likely to be vulnerable • Standard deviation of 1. 87% – The FLF density was consistent across our experimental systems • With 95% confidence the true mean is between 2. 23% and 3. 51% – There is a high probability that our experimental density is close to the TRUE FLF Density Reverse Engineering (Software Security) © SERG

Validating the FLF Hypothesis 1. Gather a set of vulnerable software systems (Experimental Systems). 2. Perform experiments on these systems with static analysis tools (GAST-MP & SGA) 3. Identify all functions which initiate user input transactions (Inputs) 4. Identify all functions which are vulnerable (Targets) 5. Calculate FLF Density (FLF Density) 6. Test the FLF Density on a set of control systems (Verification) 7. Create a tool that utilizes the FLF Density (FLF Finder). Reverse Engineering (Software Security) © SERG

Verification • The FLF Density can be used as a conservative way to highlight those vulnerability functions which do not have known vulnerabilities Reverse Engineering (Software Security) © SERG

Validating the FLF Hypothesis 1. Gather a set of vulnerable software systems (Experimental Systems). 2. Perform experiments on these systems with static analysis tools (GAST-MP & SGA) 3. Identify all functions which initiate user input transactions (Inputs) 4. Identify all functions which are vulnerable (Targets) 5. Calculate FLF Density (FLF Density) 6. Test the FLF Density on a set of control systems (Verification) 7. Create a tool that utilizes the FLF Density (FLF Finder). Reverse Engineering (Software Security) © SERG

FLF Finder • FLF Density can say what areas of code are statistically likely to be vulnerable • Automate tool to find these areas • Targets are not provided – what do we do? – Assume all functions are targets – This extremely conservative assumption is still able to reduce 60% of code! Reverse Engineering (Software Security) © SERG

Conclusion • There is credibility to the FLF hypothesis • FLF Finder can pinpoint those areas of code which are statistically likely to be vulnerable • 60% code reduction • This will greatly improve the efficiency of security code auditors! • Efficiency improvements make auditing more appealing to software developers Reverse Engineering (Software Security) © SERG