Overview of a Google Tool Thread Sanitizer v

Overview of a Google Tool Thread Sanitizer v 2

Introduction • Race Detector based on Shadow Memory – Faster than • Valgrind • Intel Parallel Inspector (PIN) – Fully parallel – No expensive synchronization (atomics/locks) on fast path – Scales to huge apps – Predictable memory footprint – Informative reports

Data Race Example #include <stdio. h> #include <pthread. h> script. sh: ------------------- int Global[4]; #!/bin/bash void *Thread 1(void *x) { Global[0] = -1; return NULL; } void *Thread 2(void *x) { printf("Global[2] = %dn", Global[2]); printf("Global[3] = %dn", Global[3]); return NULL; } void *Thread 3(void *x) { printf("Global[0] = %dn", Global[0]); printf("Global[1] = %dn", Global[1]); return NULL; } int main() { for(int i = 0; i < 4; i++) Global[i] = i; pthread_t t[3]; pthread_create(&t[0], NULL, Thread 1, NULL); pthread_create(&t[1], NULL, Thread 2, NULL); pthread_create(&t[2], NULL, Thread 3, NULL); pthread_join(t[0], NULL); pthread_join(t[1], NULL); pthread_join(t[2], NULL); return 0; } for i in {0. . 1} do. /tsan_example sleep 1 done C/C++ Program: There is a data race on the global vector, indeed depending on the thread scheduling, T 1 can write Global[0] before T 3 read it or viceversa, printing different values.

Data Race Example Global[2] = Global[3] = Global[0] = Global[1] = ---------Global[0] = Global[1] = Global[2] = Global[3] = ---------Global[2] = Global[3] = Global[0] = Global[1] = ----- 2 3 -1 1 2 3 0 1 2 3 -1 1 Global[2] = Global[3] = Global[0] = Global[1] = ---------Global[0] = Global[1] = Global[2] = Global[3] = ---------Global[2] = Global[3] = Global[0] = Global[1] = ----- 2 3 -1 1 0 1 2 3 -1 1

Tsan: Data Race Example Global[2] = 2 Global[3] = 3 ========= WARNING: Thread. Sanitizer: data race (pid=25893) Read of size 4 at 0 x 7 f 5 ab 4 c 16 cc 0 by thread T 3: #0 Thread 3(void*) /home/simone/works/projects/thread-sanitizer/examples/tsan_example. cc: 18 (exe+0 x 0000000657 b 9) Previous write of size 4 at 0 x 7 f 5 ab 4 c 16 cc 0 by thread T 1: #0 Thread 1(void*) /home/simone/works/projects/thread-sanitizer/examples/tsan_example. cc: 7 (exe+0 x 000000065739) Thread T 3 (tid=25901, running) created by main thread at: #0 pthread_create /home/simone/works/projects/llvm/projects/compilerrt/lib/tsan/rtl/tsan_interceptors. cc: 820 (exe+0 x 0000000248 e 3) #1 main /home/simone/works/projects/thread-sanitizer/examples/tsan_example. cc: 29 (exe+0 x 00000006586 e) Thread T 1 (tid=25899, finished) created by main thread at: #0 pthread_create /home/simone/works/projects/llvm/projects/compilerrt/lib/tsan/rtl/tsan_interceptors. cc: 820 (exe+0 x 0000000248 e 3) #1 main /home/simone/works/projects/thread-sanitizer/examples/tsan_example. cc: 27 (exe+0 x 00000006583 e) SUMMARY: Thread. Sanitizer: data race /home/simone/works/projects/threadsanitizer/examples/tsan_example. cc: 18 Thread 3(void*) ========= Global[0] = -1 Global[1] = 1 -----Thread. Sanitizer: reported 1 warnings

How Thread. Sanitizer works Compiling Time • Instrument every memory access in the program prepending a function call: – __tsan_read 4(addr) • Atomic memory access using: – __tsan_atomic_callbacks • Read from vtable: – __tsan_vptr_update • Function entry and exit: – __tsan_func_entry(caller_pc) – __tsan_func_exit. • Inizialization: – __tsan_init

Thread Sanitizer: Algorithm • Direct Shadow Mapping (64 -bit linux) – Shadow = 4 * (Addr & k. Mask); Application 0 x 7 ffffff 0 x 7 f 00000 Protected 0 x 7 efffff 0 x 2000000 Shadow 0 x 1 ffffff 0 x 1800000 Protected 0 x 17 fffff 0 x 000000

How Thread Sanitizer works Run-Time Library • Shadow Cell – 64 bits word, represents a single memory access (happened) to a subset of bytes within the 8 -byte word of application memory Shadow Word Fields TID (Thread Id) 16 bits (configurable) Scalar Clock (Epoch) 42 bits (configurable) Position/Size in 8 byte word 5 bits Is. Write 1 bit • Shadow States – N Shadow Words (2, 4, or 8: represents the number of accesses to the corresponding application memory region by the threads)

Thread Sanitizer: Algorithm • State Machine – Core of the algorithm that updates the Shadow State on every memory access – Steps: • Thread’s clock is incremented and a new Shadow Word (corresponding to the current memory access) is created • State Machine iterates over all Shadow Words in the Shadow State: if one of the Shadow Words consitutes a race with the new Shadow Word a warning will be reported • The new Shadow Word is inserted in place of an empty Shadow Word or in place of a Shadow Word happenedbefore the new one (if no space a random Shadow Word is evicted)

Thread Sanitizer Algorithm: Example • 4 Shadow Cells per 8 application bytes (Shadow States traces 4 memory accesses) • Program with 3 threads TID TID Epoch Pos Pos Is. W

Thread Sanitizer Algorithm: Example • 4 Shadow Cells per 8 application bytes (Shadow States traces 4 memory accesses) First Access T 1 Write in thread T 1 E 1 0: 2 W T 1 write 2 bytes on a memory location • Program with 3 threads

Thread Sanitizer Algorithm: Example • 4 Shadow Cells per 8 application bytes (Shadow States traces 4 memory accesses) Second Access Read in thread T 2 • Program with 3 threads T 1 T 2 E 1 E 2 0: 2 4: 8 W R T 2 read 4 bytes from another memory location

Thread Sanitizer Algorithm: Example • 4 Shadow Cells per 8 application bytes (Shadow States traces 4 memory accesses) Third Access Read in thread T 3 • Program with 3 threads T 1 T 2 T 3 E 1 E 2 E 3 0: 2 4: 8 0: 4 W R R T 3 read 4 bytes from a memory location, part of that (2 bytes) was previously written by T 1

Thread Sanitizer Algorithm: Example • 4 Shadow Cells per 8 application bytes (Shadow States traces 4 memory accesses) • Program with 3 threads T 1 T 2 T 3 E 1 E 2 E 3 0: 2 4: 8 0: 4 W R R There is a RACE because there is not an “happen before” relation, between E 1 and E 3, E 1 || E 3

Thread Sanitizer: Algorithm def Handle. Memory. Access(addr, tid, is_write, size, pc): shadow_address = Map. Application. To. Shadow(addr) Increment. Thread. Clock(tid) Log. Event(tid, pc); new_shadow_word = {tid, Current. Clock(tid), is_write, size, addr & 7} store_word = new_shadow_word for i in 1. . N: Update. One. Shadow. State(shadow_address, i, new_shadow_word, store_word) if store_word: # Evict a random Shadow Word shadow_address[Random(N)] = store_word # Atomic

Thread Sanitizer: Algorithm def Update. One. Shadow. State(shadow_address, i, new_shadow_word, store_word): idx = (i + new_shadow_word. offset) % N old_shadow_word = shadow_address[idx] # Atomic if old_shadow_word == 0: # The old state is empty if store_word: Store. If. Not. Yet. Stored(shadow_address[idx], store_word) return if Accessed. Same. Region(old_shadow_word, new_shadow_word): if Same. Threads(old_shadow_word, new_shadow_word): Store. If. Not. Yet. Stored(shadow_address[idx], store_word) return else: # Different threads if not Happens. Before(old_shadow_word, new_shadow_word): Report. Race(old_shadow_word, new_shadow_word) elif Accessed. Intersecting. Regions(old_shadow_word, new_shadow_word): if not Same. Threads(old_shadow_word, new_shadow_word) if not Happens. Before(old_shadow_word, new_shadow_word) Report. Race(old_shadow_word, new_shadow_word) else: # regions did not intersect pass # do nothing

Thread Sanitizer: Algorithm • Constant-time operation – Get TID and Epoch from the Shadow Cell – 1 load from thread-local storage – 1 comparison • Similar idea to Fast. Track

Thread Sanitizer: Algorithm • Stack Trace for previous access – Per-thread cyclic buffer of event • • 64 bit per event (type + PC) Events: memory access (read/write), function entry/exit Information will be lost after sometime (cyclic buffer) Buffer size is configurable • Function interceptors – malloc, free, … – pthread_mutex, lock, … – strlen, memcmp, … – read, write, …

Pros and Cons • Pros – Speed, >10 x faster than other tools – Native support for atomics operations – Numbers: 200+ races in google server-side apps, 80+ in Go programs and lib, several races in SSL • Cons – Only 64 -bit Linux – Hard to port to 32 -bit platforms (small addressspaces)