Performance Based Risk Performance is a nonfunctional software

Performance – Based Risk Performance is a non-functional software attribute that plays a crucial role in wider application domains spreading from safety-critical systems to e-commerce web sites. • We introduce the concept of performance-based risk, which is risk resulting from software failures originated from behaviors that do not meet performance requirements. • Performance failure is the inability of the system to meet its performance objective(s) • Performance-based risk is defined as: Probability of performance failure Severity of the failure 1

What do we need and what do we get? • Input – UML diagrams: Use case diagram, Sequence diagram, and Deployment diagram; – Performance objectives (requirements) • Output – Performance-based risk factor of the scenarios modeled as sequence diagrams – Identification of performance-critical components in the scenario 2

Performance – Based Risk Methodology For each Use Case For each scenario STEP 1 – Assign demand vector to each “action” in Sequence diagram; build a Software Execution Model STEP 2 – Add hardware platform characteristics on the Deployment diagram; conduct stand-alone analysis STEP 3 – Devise the workload parameters; build a System Execution Model; conduct contention-based analysis and estimate probability of failure as a violation of a performance objective STEP 4 – Conduct severity analysis and estimate severity of performance failure for the scenario STEP 5 – Estimate the performance risk of the scenario; Identify high-risk components 3

The Case Study The illustration of the methodology is done on the Flight Operations System of NASA's Earth Observing System (EOS) • NASA's Earth Observing System (EOS) is the first observing system to offer integrated measurements of the Earth's processes • The Flight Operations Segment (FOS) of EOS is responsible for the planning, scheduling, commanding, and monitoring of the spacecraft and the instruments on board • We have considered the Commanding service, to evaluate its performance-based risk. Our methodology is illustrated using the combination of “Preplanned emergency 4 command transmission” and “Handle transmission failure”

Performance – Based Risk Methodology For each Use Case For each scenario STEP 1 – Assign demand vector to each “action” in Sequence diagram; build a Software Execution Model STEP 2 – Add hardware platform characteristics on the Deployment diagram; conduct stand-alone analysis STEP 3 – Devise the workload parameters; build a System Execution Model; conduct contention-based analysis and estimate probability of failure as a violation of a performance objective STEP 4 – Conduct severity analysis and estimate severity of performance failure for the scenario STEP 5 – Estimate the performance risk of the scenario; Identify high-risk components 5

STEP 1 – Assign a Demand Vector to Every “Action” in SD Build a software execution model from the demand vectors and SD 6

The Preplanned Emergency scenario The scenario basically comprises of two sequence diagrams which are: • The preparation of command groups that are to be uplinked (SD 1) • To handle the transmission failure during uplink (SD 2) We assumed for the purpose of illustration that SD 1 is carried out once and SD 2 i. e. the retransmission takes place twice before there is a mission failure 7

Annotated SD 1 (Step 1) 8

Demand vectors for SD 1(Step 1) 9

Annotated SD 2 (Step 1) 10

Demand vectors for SD 2 (Step 1) 11

The Software Execution graph (Step 1) EOC 1 ICC 1 SD 1 IDB 1 Transmit Preplanned Command 2 Retransmit On Failure R 1 SD 2 ICC 2 EOC 5 EOC 2 SE 1 ICC 3 EOC 6 EOC 3 TC 1 T 1 EOC 7 T 2 T 1 EOC 4 12

Performance – Based Risk Methodology For each Use Case For each scenario STEP 1 – Assign demand vector to each “action” in Sequence diagram; build a Software Execution Model STEP 2 – Add hardware platform characteristics on the Deployment diagram; conduct stand-alone analysis STEP 3 – Devise the workload parameters; build a System Execution Model; conduct contention-based analysis and estimate probability of failure as a violation of a performance objective STEP 4 – Conduct severity analysis and estimate severity of performance failure for the scenario STEP 5 – Estimate the performance risk of the scenario; Identify high-risk components 13

STEP 2 – Add Hardware Platform Characteristics on the Deployment Diagram Space Craft ECOM << network - (25000 � s /KB) >> Communication Subsystem <<CPU (0. 02 � s /KB)>> Ground N/W << network - (80 � s/KB) >> EOC ICC <<CPU (0. 0025 � s/KB)>> IDB <<database (60 � s/KB)>> 14

Conduct Stand-alone Analysis (Step 2) • A stand-alone analysis of such a system consists in evaluating the completion time of the whole SD as it would be executed on a dedicate hardware platform with a single user workload. • The service time consumed by the steps (as shown in the software execution graph) is 9. 949 seconds 15

Performance – Based Risk Methodology For each Use Case For each scenario STEP 1 – Assign demand vector to each “action” in Sequence diagram; build a Software Execution Model STEP 2 – Add hardware platform characteristics on the Deployment diagram; conduct stand-alone analysis STEP 3 – Devise the workload parameters; build a System Execution Model; conduct contention-based analysis and estimate probability of failure as a violation of a performance objective STEP 4 – Conduct severity analysis and estimate severity of performance failure for the scenario STEP 5 – Estimate the performance risk of the scenario; Identify high-risk components 16

Build System Execution Model and Conduct Contention-based Analysis (Step 3) • Throughput: • Response Time: • where N is the number of requests, is the sum of all demands in the scenario, and is the maximum demand in that scenario. 17

Asymptotic bounds and Failure probability estimate (Step 3) Z 1 Z 2 N*DMAX Z 3 Response time Objective UB D LB • Failure probability (Z 1) = 0 • Failure probability (Z 2) = • Failure probability (Z 3) = 1 =0. 7958 18

Performance – Based Risk Methodology For each Use Case For each scenario STEP 1 – Assign demand vector to each “action” in Sequence diagram; build a Software Execution Model STEP 2 – Add hardware platform characteristics on the Deployment diagram; conduct stand-alone analysis STEP 3 – Devise the workload parameters; build a System Execution Model; conduct contention-based analysis and estimate probability of failure as a violation of a performance objective STEP 4 – Conduct severity analysis and estimate severity of performance failure for the scenario STEP 5 – Estimate the performance risk of the scenario; Identify high-risk components 19

STEP 4 – Conduct Severity Analysis • Functional Failure Analysis (FFA), based on UML use case and scenario diagram. • The input to FFA are • A list of events of a use case (under a specific scenario) • A list of guide words • The output is the severity level (catastrophic, critical, marginal, and minor) based on FFA in a tabulated form. 20

FFT for the Emergency Scenario in EOS-FOS (Step 4) Since we are dealing with performance-based risk, we apply only the guideword “LATE” 21

Performance – Based Risk Methodology For each Use Case For each scenario STEP 1 – Assign demand vector to each “action” in Sequence diagram; build a Software Execution Model STEP 2 – Add hardware platform characteristics on the Deployment diagram; conduct stand-alone analysis STEP 3 – Devise the workload parameters; build a System Execution Model; conduct contention-based analysis and estimate probability of failure as a violation of a performance objective STEP 4 – Conduct severity analysis and estimate severity of performance failure for the scenario STEP 5 – Estimate the performance risk of the scenario; Identify high-risk components 22

STEP 5 – Estimate the Performance Risk • The performance risk of a scenario is defined as the product of – the probability that the system will fail to meet the required performance objective for a given workload (e. g. , desired response time) estimated in STEP 3 and, – the severity associated with this performance failure of the system in this scenario estimated in STEP 4. • Performance based risk= Probability of performance failure * Severity of the failure= 0. 7958*0. 95=0. 756 23

Identify High-risk Components • Estimate the overall residence time of each component in a given sequence diagram • Sum the time of all processing steps that belong to that component in a given scenario • Normalize it with the response time of the sequence diagram • Components that contribute significantly to the scenario’s response time are the high-risk components 24

Identify High-risk Components (Step 5) • Ground and the Space(ECOM) networks are the most critical components • The service times of the other components are significantly small compared to Ground and the Space(ECOM) network components and hence are not visible on the graph 25