Guidance for Writing Test Evaluation Master Plans TEMP

Guidance for Writing: Test & Evaluation Master Plans TEMP Preparation 101 Brief September 2009

What is a TEMP ? • • • Executive level strategy and basic planning document for all system life cycle T&E Overall T&E structure, major elements, and objectives Consistent with: – – – • • • Acquisition Strategy Technology Development Strategy (TDS) Capability Development / Production Document (CDD / CPD) System Threat Assessment Information Support Plan ( ISP) Consistent and complementary to the Systems Engineering Master Plan Sufficient detail to permit planning for timely availability of test resources required to support the T&E program Road Map for: • • • Evaluation framework Resource requirements Schedules and decisions supported ‘Contract’ Between All Stakeholders! 2

Policy / Guidance • OSD: – DOD Directive 5000. 01, The Defense Acquisition System, 12 May 2003 – DOD Instruction 5000. 02: Operation of the Defense Acquisition System, 8 Dec 2008 – Interim Defense Acquisition Guidebook, 7 July 2009 • Army: – AR 73 -1, Test and Evaluation Policy, 1 August 2006 • Chapter 8: T&E Working-level Integrated Product Team (T&E WIPT) • Chapter 10: T&E Review and Reporting Requirements – - Paragraph 10 -2. T&E Master Plan (TEMP) – DA PAM 73 -1, T&E in Support of Systems Acquisition, 30 May 2003 • Table of Contents OSD documents are accessible at DAU web site http: //akss. dau. mil/ and then using the Search engine Army documents are accessible via Army T&E page on AKO website: https: //www. hqda. army. mil/suite/page/25 3

Applicable Programs • Every Army acquisition program will have a TEMP • Only exceptions are: – Investigational drugs – Biologicals – Medical devices Includes Materiel and Information Technology Programs 4

Preparation • Responsibility: Program / Project Manager (PM) • Execution: T&E WIPT (DA Pamphlet 73 -1, Chapter 2) 5

TEMP Content • OSD TEMP Approval Memo • Approval Page • Cover Sheet • T&E WIPT Coordination Sheet • Table of Contents – – – – Part I - Introduction Part II - Test Program Management and Schedule Part III - Test and Evaluation Strategy Part IV - Resource Summary Annex A - Bibliography Annex B - Acronyms Annex C - Points of Contact Attachment 1 – Requirements / Test Crosswalk Matrix • Additional appendices / annexes / attachments as needed 6

TEMP Part I (Introduction) Preparation Responsibility Matrix TEMP PART & SECTION PM SE CD / FP 1. 1 Purpose P S 1. 2 Mission Description S P TI 1 of 5 T&E LOG ACTIVITY S 1. 3 System Description 1. 3. 1 System Threat Assessment S 1. 3. 2 Program Background P 1. 3. 3 Key Capabilities P P P S S P S Legend P: Primary Responsibility S: Support Responsibility PM: Program Manager SE: Systems Engineer CD/FP: Combat Developer/Functional Proponent TI: Threat Integrator LOG: Logistician 7

TEMP Part II (Test Program Management & Schedule) Preparation Responsibility Matrix 2 of 5 CD PM SE / FP TI TEMP PART & SECTION T&E LOG ACTIVITY 2. 1 T&E Management P 2. 2 Common T&E Data Base Requirements P S 2. 3 Deficiency Reporting P S 2. 4 TEMP Updates P S 2. 5 Integrated Test Program Summary P S S S S Legend P: Primary Responsibility S: Support Responsibility PM: Program Manager SE: Systems Engineer CD/FP: Combat Developer/Functional Proponent TI: Threat Integrator LOG: Logistician 8

TEMP Part III (T&E Strategy) Preparation Responsibility Matrix TEMP PART & SECTION PM CD SE / FP TI 3 of 5 T&E LOG ACTIVITY 3. 1 T&E Strategy P P 3. 2 Evaluation Framework S 3. 3 Developmental Evaluation Approach P P 3. 4 Live Fire Evaluation Approach P P 3. 5 Certification for Initial Operational Test P 3. 6 Operational Evaluation Approach S P 3. 7 Other Certifications P S 3. 8 Reliability Growth P P 3. 9 Future T&E P P S S P P Legend P: Primary Responsibility S: Support Responsibility PM: Program Manager SE: Systems Engineer CD/FP: Combat Developer/Functional Proponent TI: Threat Integrator LOG: Logistician 9

TEMP Part IV (Resource Summary) Preparation Responsibility Matrix TEMP PART & SECTION PM SE CD / FP TI 4 of 5 T&E LOG ACTIVITY 4. 1 Introduction 4. 1. 1 Test Articles P 4. 1. 2 Test Sites & Instrumentation P 4. 1. 3 Test Support Equipment S S 4. 1. 4 Threat Representation S S P 4. 1. 5 Test Targets and Expendables P S P S 4. 1. 6 Operational Force Test Support 4. 1. 7 Models, Simulations, & Testbeds P S 4. 1. 8 Joint Operational Test Environment S S 4. 1. 9 Special Requirements S 4. 2 Federal, State, & Local Requirements P 4. 3 Manpower / Personnel Training S 4. 4 Test Funding Summary P Table Resource Summary Table P S P S P P P Legend P: Primary Responsibility S: Support Responsibility PM: Program Manager SE: Systems Engineer CD/FP: Combat Developer/Functional Proponent TI: Threat Integrator LOG: Logistician 10

TEMP Appendices / Attachments Preparation Responsibility Matrix 5 of 5 TEMP PART & SECTION CD PM SE / FP TI T&E LOG ACTIVITY Appendix A. Bibliography P S S S Appendix B. Acronyms P S S S Appendix C. Points of Contact P S S S Attachment 1. Requirements / Test Crosswalk Matrix P S S S Other Annexes / Attachments P P P Legend P: Primary Responsibility S: Support Responsibility PM: Program Manager SE: Systems Engineer CD/FP: Combat Developer/Functional Proponent TI: Threat Integrator LOG: Logistician 11

Content Detail NOTE Following TEMP Sections will be populated with actual excerpts from 4 -Part TEMPs approved by the Office of Secretary of Defense. 12

TEMP Content Detail, cont. Part I. Introduction 1. 1 Purpose 1. 2 Mission Description 1. 3 System Description 1. 3. 1 System Threat Assessment 1. 3. 2 Program Background 1. 3. 3 Key Capabilities 13

TEMP Content Detail, cont. Part II. Test Program Management & Schedule 2. 1 T&E Management 2. 2 Common T&E Data Base Requirements 2. 3 Deficiency Reporting 2. 4 TEMP Updates 2. 5 Integrated Test Program Summary 14

TEMP Content Detail, cont. Part III. Test & Evaluation Strategy 3. 1 T&E Strategy 3. 2 Evaluation Framework 3. 3 Developmental Evaluation Approach 3. 4 Live Fire Evaluation Approach 3. 5 Certification for Initial Operational Test 3. 6 Operational Evaluation Approach 3. 7 Other Certifications 3. 8 Reliability Growth 3. 9 Future T&E 15

TEMP Content Detail, cont. Part IV. Resource Summary 4. 1 Introduction 4. 1. 1 Test Articles 4. 1. 2 Test Sites & Instrumentation 4. 1. 3 Test Support Equipment 4. 1. 4 Threat Representation 4. 1. 5 Test Targets and Expendables 4. 1. 6 Operational Force Test Support 4. 1. 7 Models, Simulation, & Testbeds 4. 1. 8 Joint Operational Test Environment 4. 1. 9 Special Requirements 4. 2 Federal, State, and Local Requirements 4. 3 Manpower / Personnel Training 4. 4 Test Funding Summary Table 4. 1 Resource Summary Matrix 16

TEMP Content Detail, cont. Appendix A. Bibliography Appendix B. Acronyms Appendix C. Points of Contact Attachment 1. Requirements / Test Crosswalk Matrix Other Annexes / Attachments 17

TEMP - SEP CONTRAST TEMP SEP • T&E WIPT produces • ATEC System Team (AST) produces • MDA (or designee) approves • ATEC Commander (or designee) approves • Serves as a T&E ‘Contract’ • Outlines HOW evaluation will be done • Issue level of detail • Measure level of detail • Can include all events • Only events necessary for evaluation • Addresses Test Plan(s) SEP Integrated TEMP T&E Strategy SEP AR 73 -1 (para 1 -4 h) – ‘The detailed information contained in the SEP supports concurrent development of the TEMP. ’ 18

Army TEMP T&E WIPT Coordination Process * Combat Developer PM T&E Manager Training Developer T&E WIPT Other Agencies INPUT Acquisition Strategy Requirements Documents: CDD / CPD System Threat Assessment Critical Operational Issues and Criteria Technology Development Strategy (TDS) Information Support Plan (ISP) T&E WIPT Draft TEMP Sent to T&E WIPT Meetings ** HQDA SMDC ATEC OSD Action Officers DTC AEC OTC ASA(ALT) ILS DUSA-TE DCS, G-1 DCS, G-2 DCS, G 3 / 5 / 7 CIO / G-6 DCS, G-8 * T&E Working-level Integrated Product Team ** Meetings repeated until TEMP coordination complete Source: Adapted from DA Pamphlet 73 - 1, Figure 3 - 1 T&E WIPT TEMP Concerns/Feedback Final T&E WIPT ‘Signing Party’ T&E WIPT Coordinated TEMP 19

TEMP Staffing and Approval Process Army Programs T&E WIPT & HQDA Staff Coordinated TEMP Program Manager Signs & Submits ACAT I and II OSD T&E Oversight or Army Special Interest Yes • PEO or Developing Agency Signs • TRADOC Signs • ATEC Signs GOAL: No Simultaneous Concurrent Staffing DUSA-TE Staffs (if necessary) Army T&E Executive Approves DUSA-TE Forwards Program Manager and T&E WIPT MDA or designated TEMP approval authority for ACAT III program DUSA-TE Distributes TIMELINES PM Signature to Army T&E Executive Signature - 30 Calendar Days or less OSD TEMP Approval Objective - 45 Calendar Days or less Source: Adapted from DA Pamphlet 73 – 1, Figure 3 - 2 OSD T&E Oversight OSD Approves - D, DT&E * - D, OT&E * ASD(NII) if space, IT, or C 3 systems 20

TEMP Staffing and Approval Process Multi-Service OSD T&E Oversight Programs – Army Lead Signatures T&E WIPT & HQDA Staff Coordinated TEMP PM Submits PEO or Developing Agency TRADOC ATEC Participating Service: OTA PEO or Dev Agency User Rep No Yes DUSA-TE Forwards Other Service Approvals Army T&E Executive Approval MDA * Approval TRADOC ATEC T&E WIPT Other Services DUSA-TE Forwards Program Manager OSD Approval - D, DT&E - D, OT&E DUSA-TE Distributes TIMELINES PM Signature to Army T&E Executive Approval Signature – 30 Calendar Days or less OSD Signature Objective – 45 Calendar Days or less Source: ACAT I or OSD T&E Oversight * ACAT II programs having MDA delegation also have TEMP delegated approval authority DA Pam 73 -1, Figure 3 -4, and Interim Defense Acquisition Guidebook, 7 July 2009 21

TEMP Staffing and Approval Process Multi-Service OSD T&E Oversight Programs – Army Participant T&E WIPT Equivalent Coordinated TEMP • Army PM, PEO or Developing Agency Signs • TRADOC Signs • ATEC Signs ACAT I or OSD T&E Oversight Lead Service Acquisition Executive Yes DUSA-TE Staffs No Lead Service Acquisition Executive TRADOC ATEC T&E WIPT Other Services Army PM, PEO or MDA DUSA-TE Distributes Lead Service Acquisition Executive Army T&E Executive Approves HQDA Staff Coordination OSD Approval - D, DT&E - D, OT&E TIMELINES TEO Receipt to Army T&E Executive Approval Signature – 20 Calendar Days or less OSD Signature Objective – 45 Calendar Days or less Source: DA Pam 73 -1, Figure 3 -5, and Interim Defense Acquisition Guidebook, 7 July 2009 22

TEMP Staffing and Approval Process Non-OSD T&E Oversight – ACAT II, III, & Army Special Interest Programs T&E WIPT & HQDA Staff Coordinated TEMP Program Manager Signs ACAT II • PEO (if not MDA) ACAT III and Non-Tactical C 4/IT Programs • TRADOC • ATEC PM Forwards Program Manager and T&E WIPT MDA or Designated TEMP Approval Authority 23

Narrative TEMP Approval Process (TEMPs requiring HQDA and/or OSD approval) • T&E WIPT develops TEMP (all members empowered; includes HQDA Staff) • Final T&E WIPT meeting (all members sign TEMP coordination sheet) • PM signs TEMP Approval Page; obtains PEO, TRADOC & ATEC simultaneous concurrence (Goal: Complete at T&E WIPT via empowered members) • PM delivers TEMP with all concurrence signatures to DUSA-TE • DUSA-TE staffs TEMP, as req’d; delivers to ADUSA(T&E) for Army approval (DUSA-TE staffing includes HQDA staff when not done during T&E WIPT staffing) • ADUSA(T&E) provides approval signature within 10 days • DUSA-TE sends TEMP to OSD (DDR&E, D, DT&E (Army goal is within 30 calendar days from PM submission signature) • OSD staffing and approval objective (DDR&E, D, DT&E (or DASD C 3 ISR & IT Acq) sends to D, OT&E. (45 calendar days from receipt to return to TEO) NOTE: TEMPs are first required at milestone / key decision point entry into the acquisition life cycle. TEMPs are required to be updated at Milestone C and the Full Rate Production Decision Review, but should also be updated when the program baseline has been breached, when the associated Joint Capabilities Integration and Development System document or Information Support Plan has been significantly modified, or on other 24 occasions when the program is significantly changed or restructured.

Ten TEMP “Do’s” and “Don’ts” (SOURCE: TEMAC T&E WIPT ‘Smart Book’) 1. Do ensure Core T&E WIPT members agree with any change(s) required to obtain TEMP approval after providing their coordination signature. 2. Do enforce configuration control over the TEMP. 3. Do have Part IV (Resources) based upon planned testing. 4. Do have the Requirements / Test Crosswalk Matrix as Attachment 1. 5. Do know the TEMP Approval Authority before developing the TEMP. 6. Do be very aware of COIC & CDD / CPD staffing approval timelines. 7. Do treat the TEMP as a CONTRACT among all Stakeholders!!! 25

“Deadly Sins” Affecting TEMP Approval (SOURCE: TEMAC T&E WIPT ‘Smart Book’) • Premature Submission: Submitting a TEMP for approval when an issue(s) precluding TEMP approval exists • Failure to Pre-Coordinate: – HQDA: Deputy Under Secretary of the Army for T&E (DUSA-TE) and HQDA Staff – OSD: Director, DT&E and Director, OT&E • Incomplete: – Requirements / Test Crosswalk Matrix missing – Classified data not integrated - submitted separately – Too many TBDs in Resources Part • COIC: Not HQDA approved and incorporated • Size: Body of plan, excluding annexes, not kept to a manageable length • Annexes: Not kept to a minimum • Reference Documents: ICD / CDD / CPD & STAR not approved / unavailable 26

T&E WIPT “Do’s” and “Don’ts” (SOURCE: TEMAC T&E WIPT ‘Smart Book’) • • • Do invite HQDA offices to participate in the T&E WIPT Do obtain appropriate OSD office(s) involvement early Do know T&E WIPT issues & be empowered Do hold program to the rigorous T&E process Do base positions on facts, vice emotions Do elevate an issue if unable to resolve within the T&E WIPT • Don’t dismiss T&E WIPT meetings as unimportant • Don’t dismiss T&E WIPT members’ comments without reasoned debate • Don't allow unresolved issues to fester within the T&E WIPT (that is, take extra effort to resolve one-on-one & face-to-face) • Don’t disband the T&E WIPT after TEMP approval 27

OSD TEMP Process 28

OSD T&E Oversight Programs TEMP Signature Process * SECDEF “ADVISE” USD(AT&L) DDR&E Director, DT&E (DT TEMP Approval) D, OT&E (OT TEMP COAPPROVAL W/ D, DT&E) NII AT&L “ADVISE” C 3 ISR & IT Acquisition DOT&E AO (DT TEMP Approval) TEMP@OSD. mil or TEMP@OSD. smil. mil DUSA-TE Forwards HQDA Approved TEMP * Non-CBDP TEMPs 29

OSD TEMP Process, cont. • Develop the TEMP through the T&E WIPT process • Add the DDT&E / DOT&E action officers to your T&E WIPT • Maintain informal dialog with the DDT&E and DOT&E action officers, to include contractor support • Use Defense Acquisition Guidebook for TEMP development • Provide sufficient details about future T&E activities (Part III, Sec 3. 9) • Be specific while eliminating the “fluff” • TEMP is a master strategy document, not a detailed test plan. TEMP answers the “how” not the “why” 30

OSD TEMP Process, cont. • Work with the Defense Intelligence Agency action officers • Plan to test the system against the threats in the System Threat Assessment • To reduce TEMP size, use pointers to other documents where more details are present, i. e. : • Software development plans, • Modeling and simulation plans • Logistic support plans • Include Modeling & Simulation in the T&E planning (Part IV, Sec 4. 1. 7) • Reduce cycle time & costs 31

OSD TEMP Process, cont. • Work to integrate / combine DT and OT to achieve mutual objectives at reduced costs • Successfully completing independent OT&E (Title 10 statute requirement) is the “final exam” in T&E planning • Be careful to not give up control of the DT&E effort to the OT&E • There will still be a requirement for a dedicated OT&E event that the OTA controls • DT&E is part of the risk reduction measures that are needed to ensure the system is ready for operational testing • If the size and scope of the dedicated OT&E can be reduced without jeopardizing DT&E objectives, then integrating / combining DT and OT makes sense 32

OSD TEMP Process, cont. • Submit electronic version of draft TEMP to OSD for staff review • Identify “showstoppers” before TEMP enters the Service formal staffing and approval process • Let DDT&E and DOT&E know early if an extension to the TEMP Due Date is needed • Final staffing at OSD is a 45 -day window for DDT&E and DOT&E signature 33

Request Your Feedback Please contact Deputy Under Secretary of the Army for T&E 102 Army Pentagon Washington, D. C. 20310 -0102 (703) 602 -3508 / 2134 DSN: 332 34