DTE Strategy and the Developmental Evaluation Framework DEF

  • Slides: 19
Download presentation
DT&E Strategy and the Developmental Evaluation Framework (DEF) Concept & Program Implementation 5 June

DT&E Strategy and the Developmental Evaluation Framework (DEF) Concept & Program Implementation 5 June 2014

Discussion Topics n Concept – DT&E Strategy focused on evaluation and informing decisions n

Discussion Topics n Concept – DT&E Strategy focused on evaluation and informing decisions n How to Reflect in TEMP – Framework format n How to Build – Core Team implementation 2

DT&E Strategy Overview Define Inform Define Evaluation Data Test / M&S Resources Execute –

DT&E Strategy Overview Define Inform Define Evaluation Data Test / M&S Resources Execute – How acquisition, programmatic, technical and operational decisions will be informed by evaluation – How system will be evaluated – How test and M&S events will provide data for evaluation – What resources are required to execute test, conduct evaluation, and inform decisions Define Articulate a logical evaluation strategy that informs decisions Schedule DT&E story thread: decision – evaluation– test & resources

Developmental Evaluation Framework (Enclosure 4, Do. D Interim Instruction 5000. 02) Test and Evaluation

Developmental Evaluation Framework (Enclosure 4, Do. D Interim Instruction 5000. 02) Test and Evaluation Master Plan (TEMP) includes a Developmental Evaluation Framework (“T&E Roadmap”) – Knowledge gained from testing provides information for technical, programmatic, and acquisition decisions. Do. DI 5000. 02 (Interim) Developmental Evaluation Framework: – rk o ew m ) ra ok F o n eb tio d i a u alu n G v E itio al t s i u en m cq p A elo nse v fe De e (D – Identifies key data that contributes to assessing progress on: − Key Performance Parameters − Critical Technical Parameters − Key System Attributes − Interoperability requirements − Cybersecurity requirements − Reliability growth − Maintainability attributes − Developmental test objectives − Others as needed Show the correlation/mapping between: − Test events − Key resources − Decision supported

Developmental Evaluation Framework (DEF) Capability questions Decision Support Questions (DSQ) System capabilities DEO 1

Developmental Evaluation Framework (DEF) Capability questions Decision Support Questions (DSQ) System capabilities DEO 1 DEO 2 DEO 3 Technical measures TM 1 TM 2 *TM 3 KPP/KSA/CTP related System Engineering decomposition: Evaluate system capability - Inform decisions

The TEMP’s DT&E Strategy Story n TEMP tells the decision – evaluation – test/M&S

The TEMP’s DT&E Strategy Story n TEMP tells the decision – evaluation – test/M&S story – Section 3. 1 – T&E Strategy. Describe how T&E informs Acquisition Strategy decisions n Figure to accompany verbiage: Decision Support Key (DSK) Describes decisions and T&E information needed – Section 3. 3 – Developmental Evaluation Approach. Describe how system will be evaluated to inform decisions n Figure to accompany verbiage: Developmental Evaluation Framework (DEF) Links decisions – evaluation – test/M&S events – Section 3. 6 – Operational Evaluation Approach. n Decision Support Key (DSK) & Developmental Evaluation Framework (DEF) built by Chief Dev Tester – DEF Core Team is subgroup of T&E WIPT including Chief Developmental Tester and select SME’s 6

DT&E Informed Decisions Acquisition Programmatic Technical Operational Informing decisions throughout lifecycle: Same DEF concept/form;

DT&E Informed Decisions Acquisition Programmatic Technical Operational Informing decisions throughout lifecycle: Same DEF concept/form; Different decisions and system info

Decision Support Key Show decisions will be informed by answering T&E focus questions: DT&E

Decision Support Key Show decisions will be informed by answering T&E focus questions: DT&E Decision Support Questions (DSQs) and OT&E Critical Operational Issues (COIs) – TEMP Section 3. 1 – T&E Strategy – Describes how program’s Acquisition Strategy is informed by T&E Strategy 8

Developmental Evaluation Framework Inform Define Decisions Data Define Evaluation Execute Define Test / M&S

Developmental Evaluation Framework Inform Define Decisions Data Define Evaluation Execute Define Test / M&S Resources Schedule 9

Link Resources & Schedule Link key resources and schedule to DEF – Describe logical

Link Resources & Schedule Link key resources and schedule to DEF – Describe logical linkage of test/M&S events to necessary resources in Section IV – Describe linkage of decisions, evaluation, test, and M&S events to schedule in programmatic schedule in Section II 10

Core Teams: Assist Chief Developmental Tester in Applying DEF to Programs n More coherent

Core Teams: Assist Chief Developmental Tester in Applying DEF to Programs n More coherent and deliberate DT&E needs a consistent DT&E strategy development process n DASD(DT&E) led DEF Core Team – Small, focused group of SE, T&E and program acquisition SMEs n Chief Developmental Tester, Chief Engineer, Program Manager, Lead Developmental Test & Evaluation Organization lead, acquisition strategy SME, requirements SME – Develop DT&E strategy by facilitated discussion n n Decisions Supported – Key program decisions needing T&E generated knowledge to inform them Decision Support Questions (DSQ) Questions related to performance, reliability, cybersecurity or interoperability that when answered provide T&E information to decision makers n Developmental Evaluation Objectives (DEO) – system capability categories n Technical Measures (TM) – “inch deep mile wide” quantification of capabilities n T&E and M&S events – data generators for system evaluation

Before the Core Team Gather the People and Documents n Identify small, focused group

Before the Core Team Gather the People and Documents n Identify small, focused group members, including: – DASD(DT&E) Staff Specialist n Educate on concept, lead & facilitate strategy development – Chief Developmental Tester n Take ownership of Developmental Evaluation Strategy – Program Manager (or representative) n Acquisition Strategy SME – to define decisions supported – Program Chief Engineer, Requirements SME (DOORS™ expert) n Capability / requirements SME – to define capabilities and measures – Lead Developmental T&E Organization representative n Test, M&S SME – to define test and M&S events n Gather needed reference material, schedule ~2 days – DASD(DT&E): DT&E Strategy concept & example presentation – Program office: n n n Acquisition strategy & schedule Technical & ops requirements: JCIDS (ICD, CDD, CPD), TRD, SRD, A spec, SEP, PPP, and/or DOORS requirements decomposition OTA’ s OT&E evaluation framework (if available)

The Core Team Process 1 st - Defining the Decisions 1. Identify key decisions

The Core Team Process 1 st - Defining the Decisions 1. Identify key decisions & information needs – Identify programmatic, acquisition or technical decisions (PM, PEO, AT&L) throughout acquisition strategy – Articulate essence of decision being made – Determine DT&E information needed to support decision making in form of Decision Support Questions (DSQ) n Document Key in Decision Support

The Core Team Process 2 nd - Defining the System Capabilities 2. Developmental Evaluation

The Core Team Process 2 nd - Defining the System Capabilities 2. Developmental Evaluation Objectives (DEO) – System technical capabilities – Binned into Functional Evaluation Areas (performance, interoperability, cybersecurity, reliability) – Suggested starting point: major TRD paragraph headings – Expand or contract to generate top level listing of technical capabilities 14

The Core Team Process 3 rd - Defining the Measures 3. Determine appropriate level

The Core Team Process 3 rd - Defining the Measures 3. Determine appropriate level or detail for measures – Quantify “inch-deep & mile-wide” measures n Not just KPP/KSA/CTPs – Depending upon number of tech requirements: n n Technical measures TRD sub paragraph (binning of several related measures) DOORS hierarchy cut Binning of related measures 15

The Core Team Process 4 th - Defining the “Important” Measures 4. Highlight important

The Core Team Process 4 th - Defining the “Important” Measures 4. Highlight important technical measures for additional evaluation emphasis – Designate (bold & asterisk) KPP/KSA/CTP – related measures

The Core Team Process 5 th - Defining Test & M&S data sources 5.

The Core Team Process 5 th - Defining Test & M&S data sources 5. Define T&E and M&S events – Identify T&E and/or M&S events that will generate data for evaluation – Identify system security engineering activities (in PPP) – Place event nomenclature at cell at intersection of decision and capability/measure

Summary n DT&E Strategy and Developmental Evaluation Framework (DEF) tells logical story of how

Summary n DT&E Strategy and Developmental Evaluation Framework (DEF) tells logical story of how DT&E informs decisions throughout acquisition n DASD(DT&E) expects Developmental Evaluation Framework in TEMPs n Not going to leave you hanging! n Assistance available to implement in your program – Standardized concept intro brief & DAG guidance – Training / assistance with Core Team leadership Informed decisions: right information – right time – right person

QUESTIONS? Dr. Suzanne Beers sbeers@mitre. org 719 -572 -8257 19

QUESTIONS? Dr. Suzanne Beers sbeers@mitre. org 719 -572 -8257 19