Reasoning Systems Performance and Fault Tolerance Dan Reed

Reasoning Systems: Performance and Fault Tolerance Dan Reed Dan_Reed@unc. edu Chancellor’s Eminent Professor Vice Chancellor for IT and CIO University of North Carolina at Chapel Hill Director, Renaissance Computing Institute (RENCI) Duke University North Carolina State University of North Carolina at Chapel Hill Renaissance Computing Institute

VGr. ADS Vision Renaissance Computing Institute

Some Gr. ADS Lessons • Dynamic adaptation is really hard • Performance model decomposition is also really hard • Grid application development should not require a Ph. D. • Multivariate optimization is challenging • Implications — especially if one adapts at very low levels — separating application behavior and hardware attributes — nor should having a Ph. D. be a liability — conflicting needs and goals — Thoreau was right – simplify, simplify — consider higher level specifications for adaptation – qualitative specifications – fault tolerance and performance -> performability Renaissance Computing Institute

Challenges • Overall goal • Large-scale fault tolerance • Application susceptibility • Behavioral specification and variability • Multivariate specification and validation — raise the level of discourse about classification — prediction – observed failures and measured predictors — adaptation: resilience and recovery — sensitivity to faults and performance changes – implication for resource selection — qualitative specifications — variability detection Renaissance Computing Institute

Behavioral Classification and Adaptation • Performance variability • Fault tolerance — temporal classification – frequently, usually, seldom, … — spatial classification (processors, tasks, …) – all, most, some, a few, … — temporal/spatial relative to behavioral specification – close, far, low. BW, high. BW, reliable, unreliable — mechanisms – multivariate statistical characterization – bounding regions and isosurfaces/clustering — predictive techniques for probability of failure – resource classes and capabilities – coupled to application usage modes — resilience implementation mechanisms – adaptive checkpoint frequency – in memory checkpoints Renaissance Computing Institute

Monitoring, Prediction and Classification • Fault tolerance • Batch scheduling and checkpointing • In memory checkpointing for resilience — predictive techniques for probability of failure – resource classes and capabilities — resilience implementation mechanisms – adaptive checkpoint frequency – in memory checkpoints — checkpoint frequency based on predictors – adaptive triggering based on measurement — batch queue selection based on failure probability — collaborative approach with Jack Renaissance Computing Institute

SMART Disks • SMART • Typical SMART capabilities • Drive spin up time (for example) • By monitoring, one can identify — Self Monitoring, Analysis and Reporting Technology — on-disk monitoring and data analysis – ATA/IDE and SCSI support — head flying height, data throughput, spin up time — reallocated sector count, seek error rate, seek time performance — spin retry count, drive calibration retry count, temperature — indicative of motor or bearing failure — performance problems — failure probability Renaissance Computing Institute

ACPI Power Control • ACPI defines the following • Thermal failure is a big issue for HPC systems — Advanced Configuration and Power Management – HP, Intel, Microsoft, Phoenix and Toshiba — OS management of system power consumption — originally targeted at laptop/mobile device market — hardware registers on chip — BIOS interfaces — monitor and react in many ways – processor clock speed based on code – disk spin down to conserve power/reduce heat Renaissance Computing Institute

Power Consumption and Failure Modes • Arrhenius equation: temperature implications • Autopilot tagged sensors • • Accessible via Autopilot Manager — mean time to catastrophic failure of commercial silicon – doubles for every 10 degrees C rise above 70 degrees C — SMART (disk temperature) — ACPI (thermal zone) – active cooling policy and throttling — lm_tools (CPU and board temperature) Simple example — floating point computation launched at T=12 seconds – immediate CPU temperature increase — motherboard temperature increase at T=100 seconds — computation terminated at T=135 seconds Renaissance Computing Institute Source: Kevin Gamiel

Power Consumption and Failure Modes Load Induced Renaissance Computing Institute Source: Kevin Gamiel

Dynamic Behavioral Adaptation Renaissance Computing Institute

Diskless Checkpointing Application MPI Interface UNIX I/O MPI Fault Detection & Automatic Recovery User messages Trigger Recovery Space Optimization Data Recovery Heartbeat High Speed Interconnect Renaissance Computing Institute Redundancy Encoding Storage Choice Diskless Checkpoint Fault Tolerant MPI