STPA A new hazard analysis technique based on

STPA A new hazard analysis technique based on the STAMP model of accident causation

Outline • What is STPA • STPA process • Example: Robot • Example: TCAS • Comparison and Results • In-class STPA 2

STAMP-Based Hazard Analysis (STPA) • Basic premise is to prevent accidents by enforcing safety constraints on system behavior (controlling hazardous system states) • Goals (same as any hazard analysis) – Identification of system hazards and related safety constraints necessary to ensure acceptable risk • Design For Safety – Accumulation of information about how hazards can occur. – Use info to eliminate, mitigate and control hazards in system design, development, manufacturing, and operations 3

Controlling States • Since hazardous states can be prevented through appropriate control (enforcing safety constraints), this hazard analysis method seeks to find instances of Inadequate Control • Inadequate control occurs when there are state transitions to hazardous states • The commands, decisions, or actions that lead to violation of safety constraints: • “Inadequate Control Actions” 4

Inadequate Control Actions Identify inadequate control actions 1. A required control action is not provided or not followed 2. An incorrect or unsafe control action is provided 3. A potentially correct control action is provided too late or too early (at the wrong time) 4. A correct control action is stopped too soon. 5

Control Flaw Taxonomy • Design of the control algorithm does not enforce constraints – – – • Flaw(s) in creation process Process changes without appropriate change in control algorithm (asynchronous evolution) Incorrect modification or adaptation Process models are inconsistent, incomplete, or incorrect – – – Flaw(s) in creation process Flaw(s) in updating process Inadequate or missing feedback • • Not provided in system design Communication flaw Time lag Inadequate sensor operation 6

Control Flaw Taxonomy (cont) • Time lags and measurement inaccuracies not accounted for • Expected process inputs are wrong or missing • Expected control inputs are wrong or missing • Disturbance model is wrong – Amplitude, frequency, or period is out of range – Unidentified disturbance • Inadequate coordination among controllers and decision makers 7

Inadequate Control Execution Inadequate Execution of Control Actions • Communication flaw • Inadequate actuator operation • Time lag 8 © Copyright Nancy Leveson, Aug. 2006

STPA: A New Hazard Analysis Technique Based on STAMP Inadequate control Commands Controller Control Input Wrong or Missing Actuator(s) Inadequate Actuator Operation Inadequate Control Algorithm Process Input Wrong or Missing Controlled Process Failure Process Model Wrong Disturbances Unidentified or Out of Range Sensor(s) Feedback Wrong or Missing Inadequate Sensor Operation Process Output Wrong or Missing 9

How to Perform STPA 1. High-level Hazard Analysis: • Indentify Accidents • Hazards • High-level Safety Constraints 2. Identify Inadequate Control Actions • Control structure 3. Control Flaws • In the design 4. Change design to eliminate, mitigate, or control potentially unsafe control actions and behaviors. • Or accept 5. Iterate 10

Identifying and Specifying Safety Constraints • Most requirements only specify nominal behavior – Need to specify off-nominal behavior – Need to specify what system and software must NOT do • What must not do is not inverse of what must do • Derive from system hazard analysis 11 © Copyright Nancy Leveson, Aug. 2006

Example: Mobile Robot 12

Thermal Tile Robot Example 1. Identify high-level functional requirements and environmental constraints. e. g. size of physical space, crowded area 2. Identify high-level hazards a. Violation of minimum separation between mobile base and objects (including orbiter and humans) b. Mobile robot becomes unstable (e. g. , could fall over) c. Manipulator arm hits something d. Fire or explosion e. Contact of human with DMES f. Inadequate thermal control (e. g. , damaged tiles not detected, DMES not applied correctly) g. Damage to robot 13 © Copyright Nancy Leveson, Aug. 2006

Thermal Tile Robot Example (2) 3. Restate hazards as high-level safety constraints e. g. Robot must not allow humans to come in contact with DMES 4. Try to eliminate from system design 5. If cannot be eliminated or adequately controlled at system design level, will need to refine and allocate them to system components. 14 © Copyright Nancy Leveson, Aug. 2006

Design Constraints are Refined and Traced to Components 1. 4. 2. 1 Mobile Base (MB): Requirements: MB-FR 1: The mobile base shall be able to carry all the mobile robot subsystem components [2. 6. 3(73)] MB-FR 2: The mobile base shall be able to move smoothly in any direction and to cross cable covers on the floor [EA. 3(15), H 3(38), 2. 6. 2(73)] MB-FR 3: The mobile base shall be able to raise its inspection and injection equipment to the level required for servicing the tiles, from 2. 9 meters to 4 meters [EA. 2(15), 2. 6. 3(73), 2. 10. 1(81), 2. 10. 4(81)] 15 © Copyright Nancy Leveson, Aug. 2006

Design Constraints are Refined and Traced to Components (2) Design Constraints: MB-C 1: The mobile base must be no more than 2. 5 meters long and 1 meter wide. While moving, it must fit under structural beams as low as 1. 75 meters [EA. 2(15), 4. 6)] Safety-Related Design Constraints MB-SC 1: The mobile base must be able to ensure accuracy of 10 cm for positioning and 1 mm for tile servicing (inspection and injection tasks [EA. 2(15), H 4(38), 2. 6. 1(73), 2. 6. 4(73)] MB-SC 2: The mobile base design must protect against fire and explosion [H 6(39), 2. 6. 5(73), 2. 6. 6(73)] MB-SC 3: It must be possible to move the mobile base out of the way in case of an emergency [2. 9. 2(79)] 16 © Copyright Nancy Leveson, Aug. 2006

Design Constraints are Refined and Traced to Components (3) Motor Controller: 2. 9. 2 The drivetrains for locomotion are within the diameter of the wheel hub and consist of a brushless DC motor, resolver for positioning and commutation, a brake, a cycloidal reducer providing 225: 1 gear reduction with exceptional stiffness, and a locking hub that couples the output of the reducer to the wheel. The locking hub allows the operator to disengage the wheels from the drivetrain completely [MB-SC 3(20)] Rationale: In an emergency, the ability to disengage the wheels will allow towing or pushing the machine out of the way. 17 © Copyright Nancy Leveson, Aug. 2006

Define preliminary control structure and refine constraints and design in parallel. 18 © Copyright Nancy Leveson, Aug. 2006

Refinement and Allocation • After defining initial control structure, refine constraints and design in parallel. – Identify potentially hazardous control actions by each of system components that would violate system design constraints. Restate as component safety design requirements and constraints. – Perform hazard analysis using STPA to identify how safety-related requirements and constraints could be violated (the potential causes of inadequate control and enforcement of safety-related constraints). – Augment the basic design to eliminate, mitigate, or control potential unsafe control actions and behaviors. – Iterate over the process, i. e. perform STPA on the new augmented design and continue to refine the design until all hazardous scenarios are eliminated, mitigated, or controlled. • Document design rationale and trace requirements and constraints to the related design decisions. 19 © Copyright Nancy Leveson, Aug. 2006

Try to eliminate hazards from system conceptual design. If not possible, then identify controls and new design constraints. For unstable base hazard System Safety Constraint: Mobile base must not be capable of falling over under worst case operational conditions 20 © Copyright Nancy Leveson, Aug. 2006

First try to eliminate: 1. Make base heavy Could increase damage if hits someone or something. Difficult to move out of way manually in emergency 2. Make base long and wide Eliminates hazard but violates environmental constraints 3. Use lateral stability legs that are deployed when manipulator arm extended but must be retracted when mobile base moves. Two new design constraints: • Manipulator arm must move only when stabilizer legs are fully deployed • Stabilizer legs must not be retracted until manipulator arm is fully stowed. 21 © Copyright Nancy Leveson, Aug. 2006

Identify potentially hazardous control actions by each of system components 1. A required control action is not provided or not followed 2. An incorrect or unsafe control action is provided 3. A potentially correct or inadequate control action is provided too late or too early (at the wrong time) 4. A correct control action is stopped too soon. Hazardous control of stabilizer legs: • Legs not deployed before arm movement enabled • Legs retracted when manipulator arm extended • Legs retracted after arm movements are enabled or retracted before manipulator arm fully stowed • Leg extension stopped before they are fully extended 22 © Copyright Nancy Leveson, Aug. 2006

Restate as safety design constraints on components 1. Controller must ensure stabilizer legs are extended whenever arm movement Is enabled 2. Controller must not command a retraction of stabilizer legs when manipulator arm extended 3. Controller must not command deployment of stabilizer legs before arm movements are enabled. Controller must not command retraction of legs before manipulator arm fully stowed 4. Controller must not stop leg deployment before they are fully extended 23 © Copyright Nancy Leveson, Aug. 2006

Do same for all hazardous commands: e. g. , Arm controller must not enable manipulator arm movement before stabilizer legs are completely extended. At this point, may decided to have arm controller and leg controller in same component 24 © Copyright Nancy Leveson, Aug. 2006

To produce detailed scenarios for violation of safety constraints, augment control structure with process models Arm Movement Stabilizer Legs Manipulator Arm Enabled Disabled Unknown Extended Retracted Unknown Stowed Extended Unknown How could become inconsistent with real state? e. g. issue command to extend stabilizer legs but external object could block extension or extension motor could fail 25 © Copyright Nancy Leveson, Aug. 2006

Problems often in startup or shutdown: e. g. , Emergency shutdown while servicing tiles. Stability legs manually retracted to move robot out of way. When restart, assume stabilizer legs still extended and arm movement could be commanded. So use “unknown” state when starting up Do not need to know all causes, only safety constraints: - - May decide to turn off arm motors when legs extended or when arm extended. Could use interlock or tell computer to power it off. - Must not move when legs extended? – Power down wheel motors while legs extended. Coordination problems 26 © Copyright Nancy Leveson, Aug. 2006

Example: TCAS 27

Step 1: Identify hazards and translate into highlevel requirements and constraints on behavior TCAS Hazards: 1. A near mid-air collision (NMAC): Two controlled aircraft violate minimum separation standards) 2. A controlled maneuver into ground 3. Loss of control of aircraft 4. Interference with other safety-related aircraft systems 5. Interference with the ground-based ATC system 6. Interference with ATC safety-related advisory System Safety Design Constraints: – – – TCAS must not cause or contribute to an NMAC TCAS must not cause or contribute to a controlled maneuver into the ground … 28 © Copyright Nancy Leveson, Aug. 2006

Step 2: Define basic control structure 29 © Copyright Nancy Leveson, Aug. 2006

Component Responsibilities TCAS: • Receive and update information about its own and other aircraft • Analyze information received and provide pilot with – Information about where other aircraft in the vicinity are located – An escape maneuver to avoid potential NMAC threats Pilot • Maintain separation between own and other aircraft using visual scanning • Monitor TCAS displays and implement TCAS escape maneuvers • Follow ATC advisories Air Traffic Controller • Maintain separation between aircraft in controlled airspace by providing advisories (control action) for pilot to follow 30 © Copyright Nancy Leveson, Aug. 2006

Aircraft components (e. g. , transponders, antennas) • Execute control maneuvers • Receive and send messages to/from aircraft • Etc. Airline Operations Management • Provide procedures for using TCAS and following TCAS advisories • Train pilots • Audit pilot performance Air Traffic Control Operations Management • Provide procedures • Train controllers, • Audit performance of controllers • Audit performance of overall collision avoidance system 31 © Copyright Nancy Leveson, Aug. 2006

For the NMAC hazard: TCAS: 1. The aircraft are on a near collision course and TCAS does not provide an RA 2. The aircraft are in close proximity and TCAS provides an RA that degrades vertical separation. 3. The aircraft are on a near collision course and TCAS provides an RA too late to avoid an NMAC 4. TCAS removes an RA too soon Pilot: 1. The pilot does not follow the resolution advisory provided by TCAS (does not respond to the RA) 2. The pilot incorrectly executes the TCAS resolution advisory. 3. The pilot applies the RA but too late to avoid the NMAC 4. The pilot stops the RA maneuver too soon. 32 © Copyright Nancy Leveson, Aug. 2006

Step 3 b: Use identified inadequate control actions to refine system safety design constraints • When two aircraft are on a collision course, TCAS must always provide an RA to avoid the collision • TCAS must not provide RAs that degrades vertical separation • … • The pilot must always follow the RA provided by TCAS • … 33 © Copyright Nancy Leveson, Aug. 2006

Step 4: Determine how potentially hazardous control actions could occur (scenarios of how constraints can be violated). Eliminate from design or control in design or operations. Step 4 a: Augment control structure with process models for each control component. Step 4 b: For each of inadequate control actions, examine parts of control loop to see if could cause it. Guided by a set of generic control loop flaws Step 4 c: Design controls and mitigation measures Step 4 d: Consider how designed controls could degrade over time. 34 © Copyright Nancy Leveson, Aug. 2006

35 © Copyright Nancy Leveson, Aug. 2006

TCAS does not provide an RA when required to avoid an NMAC - Unit is not operational --Pilot does not turn it on -- Self-monitor turns off TCAS unit -- Component failure - TCAS does not perceive a conflict -- Current location of aircraft is incorrect TCAS thinks other aircraft is on the ground Incorrect altitude provided to TCAS Uneven terrain TCAS puts other aircraft outside protected volume -- Location of own aircraft incorrect Altimeter error Delay in receipt of information about altitude change 36 © Copyright Nancy Leveson, Aug. 2006

Comparison with Traditional HA Techniques • Top-down (vs. bottom-up like FMECA) • Considers more than just component failure and failure events (includes these but more general) • Guidance in doing analysis (vs. FTA) • Handles dysfunctional interactions and system accidents, software, management, etc. 37 © Copyright Nancy Leveson, Aug. 2006

Comparisons (2) • Concrete model (not just in head) – Not physical structure (HAZOP) but control (functional) structure – General model of inadequate control (based on control theory) • HAZOP guidewords based on model of accidents being caused by deviations in system variables • Includes HAZOP model but more general • Compared with TCAS II Fault Tree (MITRE) STPA results more comprehensive Included Ueberlingen accident 38 © Copyright Nancy Leveson, Aug. 2006

Ballistic Missile Defense System (BMDS) Non-Advocate Safety Assessment using STPA • A layered defense to defeat all ranges of threats in all phases of flight (boost, mid-course, and terminal) • Made up of many existing systems (BMDS Element) – – Early warning radars Aegis Ground-Based Midcourse Defense (GMD) Command Control Battle Management and Communications (C 2 BMC) – Others • MDA used STPA to evaluate the residual safety risk of inadvertent launch prior to deployment and test 39

Results • Deployment and testing held up for 6 months because so many scenarios identified for inadvertent launch. In many of these scenarios: – All components were operating exactly as intended – Complexity of component interactions led to unanticipated system behavior • STPA also identified component failures that could cause inadequate control (most analysis techniques consider only these failure events) • As changes are made to the system, the differences are assessed by updating the control structure diagrams and assessment analysis templates. • Adopted as primary safety approach for BMDS 40

In-Class STPA Subway Train Doors

Train Doors • What are is the system goal(s)? • What are the accidents? • What are the hazards? • Translate the hazards into safety constraints 42

43

What to do for Train Doors Exercise • To Do in your groups or individually, your choice. • See slide 10, “How to do STPA” and follow that to do a STPA hazard analysis on an existing train door design you are familiar with. (feel free to add design changes that are interesting to you) • Be sure to include: – Control structure and control loops, – process models (the controller’s model of what the system/process is doing), – The expected control inputs, measurements, sensors etc that you find to be relevant as you are going through the STPA process. – Inadequate control actions (slide 5) – Control flaws and inadequate control executions. (slides 6 -9) 44

What to do for Train Doors Exercise Once you’ve found the inadequate control actions and related control flaws and inadequate control executions, Identify – new safety constraints on the system and – new design decisions to enforce the safety constraints (and prevent inadequate control) • We’ll talk about the Train Doors STPA in class next week. No need to turn any papers, but bring what you’ve done so we can go over it as a group. • Feel free to contact me with questions. Maggie Stringfellow: sapphire@mit. edu 45