Windows Crash Analysis Results Archana Ganapathi Department of

Windows Crash Analysis Results Archana Ganapathi Department of EECS, UC Berkeley (archanag@cs. berkeley. edu)

Motivation (1) “If a problem has no solution, it may not be a problem, but a fact, not to be solved, but to be coped with over time. ” –Shimon Peres n Windows Crashes!!! n n Determine crash causes Document crash likelihood of SW/HW components Evaluate product dependability Underlying design principles causing crashes

Motivation (2) n Build an Oracle for System Behavior n n Determine dominant failure cause Drive benchmarks Evaluate our ideas via prototypes tested using benchmarks Help us select problems to tackle

Current Data Collection Process n Dataset n n Collection Process n n 214 machines from Berkeley EECS department, ~7 months, ~1500 crashes Windows XP SP 1 Low variability in usage profile Modify group policy to auto direct crash reports to local server Easy-to-install CER software for server No prompt for user to send crash report Frequency of collection n synchronized with application and system crashes on computers

Crash count per month Milestones: Jun 14: 25 machines Jun 25: 125 machines July 9: 150 machines Aug 3: 214 machines Aug 24: fall semester begins Dec 21: fall semester ends

Sample Results App version Component version 6/17/04 1: 33 machine 1 user 1 firefox. exe. 8. 0. 0msvcrt. dll7. 0. 2600. 1106. . . 6/17/04 1: 39 machine 1 user 1 firefox. exe. 8. 0. 0msvcrt. dll7. 0. 2600. 1106. . . 6/17/04 1: 41 machine 1 user 1 firefox. exe. 8. 0. 0msvcrt. dll7. 0. 2600. 1106. . . 6/17/04 1: 43 machine 1 user 1 firefox. exe. 8. 0. 0msvcrt. dll7. 0. 2600. 1106. . . 6/17/04 1: 47 machine 1 user 1 firefox. exe. 9. 0. 0msvcrt. dll7. 0. 2600. 1106. . .

Crash Behavior on a Machine App 1 App 2 App 3 time

Common Hypotheses n n Microsoft OS is unreliable and causes most crashes Web browsers crash more than other apps Applications hang as frequently as they crash Crashes occur roughly as much in 3 rd party dlls as MS dlls

Application Categories code development custom software database document and presentation editing document archive document viewing e-mail graphic viewer i/o instant messaging multimedia OS remote connection sci computation system mgmt/security web browsing

Microsoft OS is unreliable and causes most crashes - FALSE

System Crashes n n n XP operating system crashes very rarely! More impacting…most require reboot 9/1500 isn’t enough data to study system crashes!! n Use BOINC to collect system crash data

User behavior study n n n How regularly do people use different applications? Do people behave differently if different things crash? How frequently do people proactively reboot their computer for system stability?

Web browsers crash more than other apps - TRUE Usage Frequency Crash Frequency

Web Browser usage vs crash

Limitations of this analysis n n No info on how long each application was used before crash Need metrics on system and application usage Skewed by user community’s usage patterns User survey limitations n n n Data only as accurate as user reports Difficult to estimate time spent using each application Hard to distinguish active usage from background processes

Applications hang as frequently as they crash - TRUE Crash Cause by Component

App Hang Frequency

Crashes occur roughly as much in 3 rd party dlls as MS dlls- TRUE Component Description Author Apps invoking component %crash ntdll. dll NT system functions MS Internet Explorer, Matlab 12. 28 simpl_fox_gl. exe User application 3 rd party -- 4. 80 msvcrt. dll Microsoft C runtime library MS Acrobat, Netscape 4. 03 pdm. dll Scripting component functions MS Visual Studio, Internet Explorer 3. 07 powerarc. exe Power Archiver 3 rd party Power Archiver 2. 88 acrord 32. exe Acrobat Reader 3 rd party Acrobat Reader 2. 69 ray_tracing. exe User application 3 rd party -- 2. 69 exceed. exe Exceed X Windows 3 rd party Exceed 2. 50 rpcl 3260. dll Real Player component 3 rd party Real Player 2. 50 MS Firefox, Internet Explorer 2. 50 user 32. dll Communication, message handler, timer functions

DLLs: Model citizens and bad apples n n 227 dlls used by 33 apps analyzed 37 of these dlls caused crashes in our data set n n n Worst offenders (widely used and frequent causes of crashes): ntdll. dll, kernel 32. dll, msvcrt. dll 190 model citizens Random sample of (almost) universally used dlls that have not produced crashes: CFGMGR 32. DLL NETAPI 32. DLL RPCRT 4. DLL DBGHELP. DLL PRINTUI. DLL USERENV. DLL MSIMG 32. DLL RASMAN. DLL VERSION. DLL MSSIGN 32. DLL REGAPI. DLL 32 WMI. DLL

Crash-causing Status Code NTSTATUS Code Description Count 0 xc 0000005 ACCESS_VIOLATION 525 0 xcfffffff HANG 443 0 xc 0000006 IN_PAGE_ERROR 0 xc 0000096 STATUS_PRIVILEGED_INSTRUCTION 9 0 x. D 1, 0 x. A, 0 x. C 1, 0 x 8 E IVER_FAULT 9 0 x 80000003 STATUS_BREAKPOINT 5 0 xc 000001 d STATUS_ILLEGAL_INSTRUCTION 4 0 xc 0000094 STATUS_INTEGER_DIVIDE_BY_ZERO 3 0 xc 0000091 STATUS_FLOAT_OVERFLOW 2 0 xe 06 d 7363 0 xeedfade “Trappable error in external object” 4 0 xc 000001 e STATUS_INVALID_LOCK_SEQUENCE 1 0 xc 0000090 STATUS_FLOAT_INVALID_OPERATION 1 0 xc 015000 f STATUS_SXS_EARLY_DEACTIVATION 1 0 xc 0150010 STATUS_SXS_INVALID_DEACTIVATION 1 12

Crash Behavior on a Machine App 1 App 2 App 3 time

Parameters for clustering n n n n Time since previous crash normalized by average time between crashes on machine Time since previous cluster normalized by average time between clusters on machine Size of crash cluster Time between events in a crash cluster Frequency of crash normalized by frequency of usage Complexity of application (# of dlls) Specific dlls used Crashing dlls for app

Challenges of non-numeric data n Meaningless to instantiate “average” data point for many data types: n n n Strings: discrete values, and order irrelevant to domain structure Sets (e. g. of DLL names): no natural ordering Version numbers: order irrelevant to domain structure Hexadecimal error codes: non-ordinal when used as crash labels Non-numeric components require “unnatural” distance measure n n n Strings: d(s 1, s 2) = 0 iff s 1 = s 2; d(s 1, s 2) = 1 otherwise Sets: d(s 1, s 2) = |(s 1 s 2) (s 2 s 1)| (size of symmetric difference) Version numbers: Component-wise difference scaled so difference between minor version numbers doesn’t dominate difference between major version numbers

Sample Application Clusters

Other clusters n Alisp and Firefox (!) n n Matlab and Internet Explorer n n some crashes in both caused by same DLL (but different error codes) similar DLL sets: Matlab = IE + LAPACK? Internet apps n E. g. IE, Netscape, Y! Messenger, Outlook, Acrobat Reader

Clustering lessons n Trustworthy patterns require huge amounts of structured data n n n Tradeoff between introducing domain knowledge and biasing results to confirm hypotheses Most crash histories machine/user-specific n n Need several instances of every (application, component, error code) combo Configuration management more important than blaming (insert victim here)? Crash patterns are not usage patterns

Conclusions n Challenge of obtaining relevant data n n Need continuous description of failure info Hard to convince folks to let us collect data, primarily because of privacy concerns. n n n predominant in the home user and academic communities less so in a corporate setting - better computer etiquette Need more volunteers to share data with us! n n BOINC Industry