Lesson 3 Measures of Effectiveness for Supportability LOG

Lesson 3 Measures of Effectiveness for Supportability LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -1

Topic 1: Introduction What’s in it for me? Measures of Effectiveness (MOE) for Supportability are objective measures of a system’s ability to meet user needs and achieve the right performance and sustainment outcomes over the Life Cycle. As the system matures, the objectives identified in the early phases of the Life Cycle, are transformed into outcomes through their meeting of Key Performance Parameter (KPP) and Key System Attributes (KSA), and other Technical Performance Measurement (TPM) parameters. MOEs serve as inputs to the Affordable System Operational Effectiveness (ASOE) Model to drive its trade-off processes. Within that context, MOEs serve the Program Management, Systems Engineering, Test & Evaluation and Life Cycle Logistics Communities as the gauges by which a system’s Operational Effectiveness, Suitability and Affordability are tracked and evaluated. LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -2

Life Cycle Management Framework Technology Where Are You? Maturation & Risk What Influence Do You Have? Reduction Understand Communicate User Sustainment Needs Design, Test, and Redesign using Sustainment Metrics Monitor Performance (Program Progress, Technical Performance Metrics, Process Metrics) LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -3

MOE Life Cycle Role Requirements to Design Sustainment to Disposal MOEs enable the Closedloop Systems Engineering process. Design to Fielding to Sustainment LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -4

Affordable System Operational Effectiveness Model MOEs have the greatest influence in Design Effectiveness. MOEs LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -5

Technology Maturation & Risk Reduction (TMRR) Phase Technology Maturation & Risk Reduction Adapted from SAE GEIA-HB-0007 LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -6

Technology Maturation & Risk Reduction (TMRR) Phase Milestone B Activities ID Support System Eval & Trade-off Criterion Conduct Technology Trade-offs Assess Trade-off Impacts Output Analysis Objectives Update Logistics Product Data LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -7

MOE Lesson Approach This lesson approach emphasizes the LCL’s role in selecting MOEs and translating metrics into achievable design and Sustainment outcomes. Understand MOE Select MOEs and translate them into achievable outcomes Monitor Sustainment metrics and report deviations LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -8

Topics and Objectives TLO: Analyze Measures of Effectiveness to ensure a supportable design Objectives Topic 1: Introduction • • Welcome Where Are You? What Influence Do You Have? • Distinguish the relationships between Measures of Effectiveness, Key Performance Parameters/Key System Attributes, Measures of Performance, Measures of Suitability, and Technical Performance Measures Analyze the JCIDS Sustainment Measures of Effectiveness and their maturity over the system life cycle Topic 2: Understand Measures of Effectiveness • Topic 3: Select Measures of Effectiveness for Supportability Topic 4: Translate Measures of Effectiveness into Achievable Outcomes Topic 5: Validate a Supportable Design • • Analyze candidates for Supportability Technical Performance Measures Analyze achievable Sustainment candidates for promotion to capability and contractual design documents • Establish evaluation criteria for each Sustainment metric to validate design performance • Monitor Sustainment metrics to assure the system meets Supportability design criteria Topic 6: Summary LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -9

Topic 2: Understand Measures of Effectiveness LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -10

MOE Set Up 1. 1 Classify MOEs into Categories 1. 2 Align MOEs to Milestones Our planning considers two main areas: 1. 1 Classify MOEs – Categorize and describe MOEs in terms of KPP/KSAs, performance, suitability, and use 1. 2 Align MOEs – Describe MOEs from a milestone and artifact perspective LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -11

Set Up – Classify MOEs into Categories MOE MS A User Need MOE KPP MOEs: • Are defined by users • Define success in the operational environment • Focus on capabilities LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -12

Set Up – Classify MOEs into Categories KPP MS A User Need MOE KPP KSA Metric KPPs: • Belong to the program • Are required for all ACAT I programs Note: ACAT II programs and below with materiel solutions, shall include the Sustainment KPP or sponsor defined Sustainment metrics. ∆ AM ∆ AO SUSTAINMENT KPP, KSA, AND METRIC KPP – Availability Materiel Availability (AM) • Supportability • Process Efficiency • # of Operational End Items/Population Operational Availability (AO) • Uptime/(Uptime + Downtime) • Downtime is (total time – uptime) KSA – Reliability (R) ∆ R • Reliability • Process Efficiency • Mean Time between Failures KSA – Operation & Support Cost ∆ $ Operation & Support Cost (O&S Cost) • Maintenance • Spares • Fuel • Support Metric – Mean Down Time ∆ MDT LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 Mean Down Time (MDT) • Maintainability • Support Facilities • Producibility • Process Efficiency 3 -13

Set Up – Classify MOEs into Categories MOP, MOS, and TPM MS A User Need MOE KPP KSA Metric MOP E. g. : • Speed • Payload MOS TPM E. g. : • MTBF • MTTR E. g. : • Reliability • Maintainability LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -14

Set Up – Classify MOEs into Categories TPM Mean Time between Failure (hours) MSA TD EMD PD U pper Limit Planned Profile Variance imit L r e Low TPM Achieved to Date LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 E. g. : • MTBF • MTTR TPMs: • Assess the design process • Determine compliance to performance 3 -15

Set Up – Classify MOEs into Categories Inputs and Outputs User Needs/Requirements Process Inputs Ao. A ICD TPM Candidates • Analyze Sustainment TPM candidates and their attributes • Establish evaluation criteria Refine Outputs CDD • • TPMs Availability KPP (AM and AO) Reliability KSA O&S Cost KSA MDT Weight Speed Create/Refine Consult/Refine CDD CPD LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -16

Set Up – Classify MOEs into Categories Attributes for Good Measures/Metrics Good measures/metrics include the following attributes: Have value to team members or are an attribute essential to customer satisfaction Show a trend Are clearly defined $ Use data that is cost-effective to collect LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -17

Set Up – Classify MOEs into Categories One way to classify MOEs is by using the following categories: Product Development CDD Program Management Process Management Product Support Arrangement LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -18

Set Up – Classify MOEs into Categories Product Development CDD Product Development Examples • • • Operational Availability (AO) Weight budget MTBF Speed Range Objective Threshold MTBF Trend All good 1 2 3 4 5 6 7 8 9 Component Number LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -19

Set Up – Classify MOEs into Categories Program Management Examples Schedule performance index and variance Cost performance index and variance Manpower (planned vs. actual) Risk assessment tracking Earned Value Cumulative Dollar Variances Deliveries $10. 00 $’s in Millions • • • $0. 00 Schedule Variance ($10. 00) ($20. 00) Cost Variance ($30. 00) LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -20

Set Up – Classify MOEs into Categories Process Management Examples Product Support Arrangement Defects per 100 units • Number and cost of: • Requirement changes • Engineering change proposals • Test failures • Defect rates 30 • Cycle time 25 20 15 10 5 0 Time LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -21

Set Up – Align MOEs to Milestones The Life Cycle and Artifacts Technology Maturation & Risk Reduction 1 ICD CDD CPD Understand Communicate User Sustainment Needs 2 Project Management Plan RAM-C Rationale Report SEP LCSP Update TEMP Update Design, Test, and Redesign using Sustainment Metrics 3 Monitor Performance (Program Progress, Technical Performance Metrics, Process Metrics) Affordable Effective Suitable LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -22

Set Up – Align MOEs to Milestones MOE Maturity and the Life Cycle Technology Maturation & Risk Reduction User Need ICD • • AO AM Reliability O&S Cost MOP • Speed • Payload TPM • • • MOS • • Reliability • • Maintainability • MTBF MTTR Power Range Endurance Payload Traceability MOEs KPP/KSA Refine TPMs T&E CDD CPD MOEs mature with the life cycle and are traceable in artifacts. LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -23

Topic 3: Select Measures of Effectiveness for Supportability LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -24

MOE Analyze – TPM Candidates and Attributes Our analysis consists of three main areas: 2. 1 Analyze Candidates – Determine best TPMs for a supportable design 2. 2 Analyze Achievable TPM Attributes– Describe TPM relationship to program capability document maturity 2. 3 Establish Criteria– Describe Sustainment TPM role in design performance 2. 1 Analyze TPM Candidates LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 2. 2 Analyze Sustainment TPM Attributes 2. 3 Establish Evaluation Criteria 3 -25

Analyze – TPM Candidates and Attributes ASOE Model and TPMs The ASOE Model has four distinct intersections representing program performance and Sustainment objectives. Balancing these priorities requires using TPMs in a series of trade-offs. Technical Performance Process Efficiency Supportability • Functions • Reliability • Production • Capabilities • Maintainability • Maintenance • Support Features • Logistics Life Cycle Cost Total Ownership Cost • Operations LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -26

Analyze – TPM Candidates ASOE and Product Development 1 Capability 2 3 Concept Description 4 TPMs 1 Be on station for ‘n’ Supportability 2 Reliability/Maintainability MTBF and MTBM hour duration 1 Respond to Process operational tasking Efficiency within ‘n’ hours 3 Operational response time LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 Time in hours to start mission 3 -27

Analyze – TPM Candidates Product Development and Logistics Product Data Strike-Talon (Indenture Level A) LCN ACA COM/NAV System (Indenture Level B) LCN ACAABA Assembly ST UAV SATCOM Control (Indenture Level C) LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -28

Analyze – TPM Candidates ASOE and Product Development (cont. ) 1 Capability 1 Weigh no more than ‘n’ pounds 1 Carry and deploy 2 3 Concept Supportability payload within weight Supportability and loiter parameters 2 4 Description TPMs Support Features PDR Weight/CDR Weight 2 Support IOT&E Weight/MTBF/Mission Features/Reliability endurance time LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -29

Analyze – TPM Candidates ASOE and Program Management 1 Capability 1 Deliver Supportability analyses by x. QFYyy 2 Concept Process Efficiency 1 Deploy ‘n’ prototypes Process within budget 3 Efficiency/ Life Cycle Cost Description 3 Delivery schedule 4 Cost LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 4 TPMs Schedule variance Cost variance Earned Value 3 -30

Analyze – TPM Candidates ASOE and Process Management 1 Capability 2 3 Concept Description 1 Deploy ‘n’ prototypes 3 1 Deploy ‘n’ full 3 Production Process Efficiency/ with ≤ ‘m’ requirement Quality/Cost Life Cycle Cost 4 changes rate production units with ≤ ‘m’ defects/unit Process Efficiency LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 quality 4 TPMs # Requirement changes per phase/Cost per requirement change Defect rate 3 -31

Analyze – Sustainment TPM Attributes Thresholds and Objectives Mean Time between Failure (hours) KPPs/KSAs are expressed using threshold and objective values defined in the draft CDD: • Threshold is the absolute minimum performance the customer will accept • Objective is the performance that the customer really wants In this example, anything above the objective is questionable. Objective Planned Profile Threshold Variance Achieved to Date Planned Profile LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 In this example, the utility of anything below threshold is questionable. 3 -32

Analyze – Sustainment TPM Attributes Thresholds and Objectives (cont. ) Drive Reliability up to optimum level Objective Threshold Drive Sustainment cycle time down to optimum level LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -33

Analyze – Sustainment TPM Attributes TPM Promotion to Program Documents Product TPM Candidates Table B. (U) STRIKE TALON UCAS Suitability Attributes Attribute Parameter Development Threshold > 0. 75 at IOT&E > 0. 80 at IOC plus 2 years Development Objective Operational Availability AO, AI, AM Reliability Mean Time Between ≥ 40 Operational Mission Failure (MTBOMF) (Hours) ≥ 90 Maintainability Mean Corrective ≤ 3. 0 Maintenance Time Operational Mission Failure (MCMTOMF) ≤ 2. 0 Diagnostics Mean Flight Hours Between False Alarms (MFHBFA) >2000 hours >300 hours > 0. 90 LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 Applicable product TPMs are promoted to program capability development and capability production documents (CDD & CPD). 3 -34

Analyze – Sustainment TPM Attributes TPM Promotion to Program Management Plan Program Management TPM Candidates Applicable Program Management TPMs are promoted to the Program Management Plan (PMP). LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -35

Analyze – Sustainment TPM Attributes TPM Promotion to Contracts Process TPM Candidates Development/Production Contracts Threshold Objective LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 Applicable Process TPMs are promoted to Program Contractual Documents. 3 -36

Topic 4: Translate Measures of Effectiveness into Achievable Outcomes LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -37

MOE Analyze – Establish Evaluation Criteria Our analysis consists of three main areas: 2. 1 Analyze Candidates – Determine best TPMs for a supportable design 2. 2 Analyze Achievable TPM Attributes – Describe TPM relationship to program capability document maturity 2. 3 Establish Criteria – Describe Sustainment TPM role in design performance 2. 1 Analyze TPM Candidates LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 2. 2 Analyze Sustainment TPM Attributes 2. 3 Establish Evaluation Criteria 3 -38

Analyze – Establish Evaluation Criteria TPM Role in Design Performance Technical Performance Measures UAV Weight (kg) 53 52 51 50 49 Threshold 48 47 Objective 46 45 LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 TPMs that are mission critical and cost sensitive are used as evaluation criteria to measure progress across major life cycle milestones and Supportability reviews. 3 -39

Analyze – Sustainment TPMs TPM Maturity Technology Maturation & Risk Reduction ICD CDD CPD Understand Communicate User Sustainment Needs Project Management Plan RAM-C Rationale Report SEP LCSP Update TEMP Update Design, Test, and Redesign Using Sustainment Metrics TPMs grow in number and mature as the life cycle progresses. LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -40

Topic 5: Validate a Supportable Design LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -41

MOE Report Findings Our analysis findings fall into two broad categories: 3. 1 Monitor Sustainment Metrics – Use a surveillance process to collect and update Sustainment data 3. 2 Report Sustainment Outcomes – Qualify Sustainment TPMs that validate a supportable design 3. 1 Monitor Sustainment Metrics LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3. 2 Report Deviations 3 -42

Report Findings – Monitor Sustainment Metrics Surveillance Process Failure Reporting, Analysis, and Corrective Action System LCLs PSM SE LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -43

Sustainment Metrics Report Findings – Report Deviations Communication Structure All IPS Elements involve MOEs/TPMs and are reported to all IPTs. LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -44

Report Findings – Report Deviations Design Reviews Did the metric meet threshold? For example, at CDR (i. e. , Post-Milestone B), the TPM/metric being analyzed should be between the threshold and objective values. Technical Performance Measures UAV Weight (kg) 53 52 51 50 49 Threshold 48 47 Objective 46 45 LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -45

Is a given MTBF TPM on the correct trajectory to meet the requisite Supportability reviews? Key Question Exercise 20 minutes Individual grade LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 Directions Independently access Blackboard and select the Exercise link in the Lesson 3: Measures of Effectiveness for Supportability folder to answer two questions. 3 -46

Mean Time between Failure (hours) Lesson Exercise Debrief Question 1 Graph U pper Limit Planned Profile imit L r e Low Achieved to Date LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -47

Mean Time between Failure (hrs. x 100) Lesson Exercise Debrief Question 2 Graph 44 43 Planned Profile 42 41 Objective 40 Threshold 39 38 Achieved to Date 37 36 35 34 LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -48

Topic 6: Summary LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -49

Takeaways • • MOEs for Supportability are: o User defined o Matured through the life cycle o Used to measure product, program, and process performance o Translated by the program into KPPs/KSAs are MOEs that serve as important criteria to assess whether the design meets user requirements TPM/metrics help users determine whether their needs are met. Programs use TPMs/metrics to evaluate design maturity and progress. LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 An MOE is a mission-oriented qualitative or quantitative measure of operational success closely related to the mission objective or operation being evaluated. 3 -50

Summary TLO: Analyze Measures of Effectiveness to ensure a supportable design Objectives Topic 1: Introduction • • Welcome Where Are You? What Influence Do You Have? • Distinguish the relationships between Measures of Effectiveness, Key Performance Parameters/Key System Attributes, Measures of Performance, Measures of Suitability, and Technical Performance Measures Analyze the JCIDS Sustainment Measures of Effectiveness and their maturity over the system life cycle Topic 2: Understand Measures of Effectiveness • Topic 3: Select Measures of Effectiveness for Supportability Topic 4: Translate Measures of Effectiveness into Achievable Outcomes Topic 5: Validate a Supportable Design • • Analyze candidates for Supportability Technical Performance Measures Analyze achievable Sustainment candidates for promotion to capability and contractual design documents • Establish evaluation criteria for each Sustainment metric to validate design performance • Monitor Sustainment metrics to assure the system meets Supportability design criteria Topic 6: Summary LOG 211 Lesson 3: Measures of Effectiveness for Supportability FINAL v 1. 3 3 -51