Exterminator Automatically Correcting Memory Errors Gene Novark Emery

Exterminator: Automatically Correcting Memory Errors Gene Novark, Emery Berger. Ben Zorn Microsoft Research UMass Amherst UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science • 2006

Debugging Memory Errors n n Billions of lines of deployed C/C++ code Apps contain memory errors n n n Heap overflows Dangling pointers Notoriously hard to debug n n Must reproduce bug, pinpoint cause Average 28 days from discovery of remotely exploitable memory error and patch [Symantec 2006] UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science • 2006

Coping with memory errors n Unsound, may detect errors n n Windows, GNU libc, Rx Sound, always finds dynamic errors n CCured, CRED, SAFECode n n Valgrind, Purify n n Order of magnitude slowdown Probabilistically avoid errors n n Requires source modification Die. Hard [Berger 2006] Exterminator: automatically isolate and fix detected errors UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science • 2006

Die. Hard Overview n Fully-randomized memory manager n n Replication n n Bitmap-based with random probing Increases odds of benign memory errors Different heap layouts across runs Run multiple replicas simultaneously, vote on results Increases reliability (hides bugs) by using more space UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science • 2006

Die. Hard Heap Layout size = 2 i+3 2 i+4 00000001 1010 allocation bitmap heap n Bitmap-based, segregated size classes n Bit represents one object of given size n n n i. e. , one bit = 2 i+3 bytes, etc. malloc(): randomly probe bitmap for free space free(): just reset bit UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science • 2006

Exterminator Extensions size = 2 i+3 2 i+4 00000001 1010 allocation bitmap heap Die. Hard Exterminator 2 1 3 object id (serial number) A 4 A 8 A 3 alloc site D 9 D 6 dealloc site 3 2 dealloc time UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science • 2006

The Exterminator System Error isolator runtime patches correcting allocator seed Die. Fast replica 1 correcting allocator input broadcast seed Die. Fast replica 2 output vote correcting allocator seed Die. Fast replica 3 n n n On failure, create heap images (core dump) Isolator analyzes images, creates runtime patch Correcting allocator corrects isolated errors: n n pad allocations extend object lifetimes UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science • 2006

Exterminator Isolation Algorithm n Identify “discrepancies” n Compare valid object data n n Find equivalent objects (same ID) with different contents Find corrupted canaries (free space) Check for possible buffer overflows Check for dangling pointer error UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science • 2006

Comparing Object Data n Lots of valid reasons for data to differ n n n Pointers (random target locations) File descriptors Non-transparent use of pointers n n n e. g. Red-Black tree keyed on pointer value Etc. Exterminator identifies and ignores: n n Values which differ across all replicas Valid pointers referring to same target ID UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science • 2006

Error Isolation: Buffer Overflows Replica 1: “malignant” overflow 2 5 4 3 1 2 6 1. Identify corrupt object ( 5 vs. 5 & 5 ) 2. Search for source ( = 2: 1: 2 candidate!) No object ) 3. Compare data at same ( 5 vs. 3 2 6 2 1 6 4 1 & Match! ) 5 Replica 2 & 3: “benign” overflows 3 5 2 4 2 UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science • 2006

Error Isolation: Dangling Pointer Freed, Canary value 5 2 4 3 1 Dangled ptr ? 1 n n n 6 3 6 2 5 4 Corrupted canary values for object 5 Same object, same corruption Assume dangling pointer Buffer overflow? n Extend lifetime of object Source object would be at same in all replicas n Unlikely, UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science • 2006

Error Isolation: Dangling ptr read n What if the program doesn’t write to the dangled pointer? n n n Die. Fast overwrites freed objects Canaries produce invalid reads, crashes How to identify prematurely freed objects? n n n Common case 1: read something that was a pointer, dereference it Common case 2: read numeric value, error propagates through computation No information: previous contents destroyed! UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science • 2006

Error Isolation: Dangling ptr read n Solution: Write canaries randomly (half the time) n Equivalent to extending object lifetime (until overwritten) Illegal free: (later read + deref ptr) Legal free: OK OK OK CRASH! OK : overwritten with canaries : data intact UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science • 2006

Error Isolation: Dangling ptr read n Correct frees uncorrelated with crash n For each object i, compute estimator: n P > 0. 5: dangling pointer error n Create patch when confidence reaches threshold UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science • 2006

Runtime Patches n Overflow patches n n n Allocation callsite Overflow amount Dangling pointer patches n n Allocation & Deallocation callsites Lifetime extension UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science • 2006

Correcting Allocator n n Extended Die. Hard allocator Reads runtime patches Stores pad table & deferral table On free: n Check for life extension for current object n n Place ptr, time on deferral priority queue On allocation: n n Check for overflow fix for current callsite Check deferral queue for pending frees UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science • 2006

Results n n Analytical results Empirical results n n Runtime overhead Error detection n n Injected faults Real application (Squid) UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science • 2006

Analytic results summary n Buffer overflows n n False negative & positive rate decrease exponentially with # of replicas Dangling pointers n n Write: exponentially low false +/- rate Read-only: n Confidence threshold controls false positive rate, # replicas needed to identify culprit UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science • 2006

Empirical Results: Runtime UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science • 2006

Empirical Results: Overflows UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science • 2006

Empirical Results: Dang. Ptrs. UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science • 2006

Empirical Results: Squid n Squid web cache heap overflow n n n Die. Fast detects error immediately n n Remotely exploitable Crashes glibc 2. 8. 0 and BDW collector Corrupted canary past overflowed object Exterminator’s isolator generates an object pad of 6 bytes, fixing the overflow UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science • 2006

Conclusion n Randomization + Replication = Information n Randomization bugs have different effects Exterminator exploits different effects across heaps to isolate cause Low overhead n n Automatically fix bugs in deployed programs Breaks crash-debug-patch cycle n Create 0 -day patches for 0 -day bugs UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science • 2006

Questions? n http: //www. cs. umass. edu/~gnovark/ UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science • 2006