Engineering Risk and Innovation Safety In Depth Hazards

Engineering Risk and Innovation Safety In Depth Hazards and Security Analysis for an Industrial Test Enclave J. R. Taylor, C. Chronopoulos, S. Piccolo S. Sarshar, J. E. Simensen

Deep Hazard Identification Motivation • When comparing risk analysis scenarios with real accidents, large gaps appear – Many real scenarios simply do not exist in risk analyses. – Many causes which are essential to understanding of an accident do not appear in risk analyses. • Prevention of failures and errors requires that causes are found and preferably eliminated. 2 DTU Management Engineering, Technical University of Denmark

Example - Deep causes of a fan failure • Control room alarm showed that Emergency Containment Cooler fan tripped. • Troubleshooting found the control power fuse for the fan's power supply breaker was loose in its fuse holder. • The fuse holder clips had been widened during installation of a new circuit breaker. • The most probable cause of the loose fuse was improper insertion. • Checks did not identify the problem. • Corrective actions – The fuse holder clips were adjusted to provide a tight fit. – Similar breakers were inspected for fuse tightness. • Handbook failure rate for fuses 3*10^-4 per year, • Actual “failure rate” 3*10^-2 for just one cause 3 DTU Management Engineering, Technical University of Denmark

Deep Causes Root cause Organisational causes Management errors and oversights. Maintenance procedures, QC 4 Human error Failure mechanism Example Contacts loose Example Loosened during maintenance Failure mode Example fuse failed open Principle Analyse sufficiently deeply to identify to find possible prevention methods DTU Management Engineering, Technical University of Denmark

Typical Industrial FMEA Ref. 1. 1. 1 5 Item Brake hydraulic hose connector Potential failure mode Brake hydraulic hose connector leaks Potential cause(s) / mechanism a) O-ring Compression Set (Creep) failure b) surface damage during assembly DTU Management Engineering, Technical University of Denmark Local effects of failure Decreased pressure to main brake hose Next higher level effect No Left Wheel Braking

Deep FMEA Ref. 1. 1. 1 6 Item Brake hydraulic hose connector Potential failure mode Leakage Potential cause(s) / mechanism Local effects of failure Safety Measures Seal leak - Nipple not adequately tightened - Cross threaded - Worn -- No condition assessment programme -- Use beyond design working life -- Re-use of second hand equipment -- Fake spare part >> Fake spare parts -- Poor earlier maintenance performance -- Poor workmanship -- Technician slip, not corrected - Foreign object in nipple when tightened -- Poor cleanliness during maintenance - Wrong tool used -- Proper tool not provided -- Correct tool "borrowed" - O-ring damage -- O-ring installed skew, trapped in nipple thread --- Technician slip, not corrected - O-ring damaged in storage -- Foreign object in nipple when tightened --- Poor cleanliness during maintenance ------ Another 32 causes Decreased pressure to main brake hose - No Left Wheel Braking Specify torque DTU Management Engineering, Technical University of Denmark Good thread lead in Specify QC tests Specify replacement interval Materials receiving QC Provide Oring placement tool

Enclave for test of analysis approaches HVAC SCADA work station Router Network (TCPIP Switch Analog I/O Digital I/O PLC Power distribution Modbus Pressure sensor Variable frequency drive Fan 7 PSU DTU Management Engineering, Technical University of Denmark Motor Power supply

Methods applied • Brain storming • Overall Functional Failure Analysis (FFA) • Deep FMEA • Installation Action Error Analysis (AEA) • Operations Action Error Analysis (AEA) • Human Machine Interface Risk Analysis • Security Sneak Path Analysis • Software Functional Failure Analysis • Software fault tree analysis 8 DTU Management Engineering, Technical University of Denmark

Human Machine Interface Risk Analysis ^ ^ 9 DTU Management Engineering, Technical University of Denmark

HMIRA example Action Error mode HMI problem Observability Recoverability Consequence Adjust pressure set point Push too long Operator watches measured value, not set point. Long response time. Set point directly observable. Pressure increases too much. Possible damage. Operator not familiar with the pressure units. Reasonable visibility Poor colour contrast. No alarm? Press wrong button “Reduce” button is near to increase button. Set point directly observable. Slow pressure change. Long response time for recovery. Pressure too low Check list of HMI problems: So far 22 interface control types, typically 8 problem types for each. 10 DTU Management Engineering, Technical University of Denmark

Fan control software – Block diagram 11 DTU Management Engineering, Technical University of Denmark

Controls Functional Semi-automated Failure Analysis Further analysis at the code level Functional failure mode Consequences Deep causes 12 DTU Management Engineering, Technical University of Denmark

Follow up & validation– published incident reports • Inspectapedia – 56 failure causes, all found in our analysis or not relevant • Nilesh Pancholi and M. G. Bhatt 21 failure causes, all found in our analysis • Marcel Kamutzki, Fan failures: five typical problems and what causes them, 74 relevant causes of which 42 were found in the analysis 13 DTU Management Engineering, Technical University of Denmark

Conclusions • In depth failure analysis is feasible for all aspects of process equipment, operation and installation. • In depth analysis is a necessary step towards failure prevention. • Time required for deep analysis can be large, total time 12 hours. • Time and effort can be reduced by semiautomated hazard identification. 14 DTU Management Engineering, Technical University of Denmark