Presented to Name TE Division Executive Team TE

  • Slides: 32
Download presentation
Presented to: Name: TE Division Executive Team T&E WIPT Approved for public release; distribution

Presented to: Name: TE Division Executive Team T&E WIPT Approved for public release; distribution unlimited. Review completed by the AMRDEC Public Affairs Office (DATE and FN#). Date: 24 May 2012

Learning Objectives Upon completion of this Missile System Testing Overview student will: 1) Understand

Learning Objectives Upon completion of this Missile System Testing Overview student will: 1) Understand what a T&E WIPT is, why it is needed, and what it produces (Slides 3 -8) 2) Understand the regulatory basis for a T&E WIPT and its role in the acquisition process (Slides 9 -10) 3) Understand the changing T&E WIPT focus as it moves through the Acquisition Phases (Slide 11) 4) Understand how you start a T&E WIPT and develop the Charter with its essential parts (Slides 12 -15) 5) Understand Principal/Associate Members and their roles & responsibilities (Slides 16 -18) 6) Understand the organizational relationships between T&E WIPT members (Slide 19) 7) Understand challenges with consensus building within the T&E WIPT and tips on collaboration and conflict resolution (Slides 20 -26) 8) Understand the dynamics with outside organizations in the formal staffing of the Test and Evaluation Master Plan (TEMP) (Slides 27 -30) 2 File. Name. pptx

What is a T&E WIPT 3 • The T&E WIPT is a defined forum

What is a T&E WIPT 3 • The T&E WIPT is a defined forum serving the PM, and PMO, as the test and evaluation subject matter experts responsible for supporting the PM, and other program WIPTs, on all aspects of a program’s test and evaluation effort. This effort includes T&E program strategy, design, development, oversight, and the analysis, assessment and reporting of test results. • The T&E WIPT should be established and chartered as early as possible during the Materiel Solutions Analysis phase so that it is involved in program strategy discussions and plans. LO #1 3 3 File. Name. pptx

What is a T&E WIPT (Continued) • The PM may form lower level functional

What is a T&E WIPT (Continued) • The PM may form lower level functional working groups, who report to the T&E WIPT, to focus on specific areas such as integrated test planning, information assurance, SW T&E, reliability scoring, M&S development and Verification, Validation, and Accreditation (VV&A), and threat support. The T&E WIPTs primary focus areas through a program’s acquisition process and phases are identified on chart number 11 4 LO #1 4 4 File. Name. pptx

T&E WIPT Membership WIPT Chair: PM, DPM, or Govt. T&E Lead. Must have T&E

T&E WIPT Membership WIPT Chair: PM, DPM, or Govt. T&E Lead. Must have T&E experience relative to the product/system HQs/Sponsor/PEO representatives. Can speak for sponsoring Service/Agency OSD DT&E & DOT&E representatives Responsible and participating T&E organizations Invited SMEs relative to the WIPT issue/topic Functional representatives such as: NII, SW, M&S, IA, and JITC 5 LO #1 OTA(s) and other organizations involved in the program, such as JFCOM, especially if a So. S/Fo. S program Contractor / vendor as appropriate for type of discussions/decisions 5 5 File. Name. pptx

What are the T&E WIPT Products There are three primary T&E planning documents –

What are the T&E WIPT Products There are three primary T&E planning documents – TES, TEMP, and the Integrated Master T&E Schedule Test and Evaluation Strategy (TES) at Milestone A: – For programs on the OSD T&E Oversight List, the TES shall be submitted to the USD(AT&L) DT&E and the DOT&E for approval – TES describes the overall approach for evaluating system capabilities and addresses test resource planning – TES addresses Technology Development Phase evaluations, and the evaluation of system design concepts against the preliminary mission requirements resulting from the Analysis of Alternatives – TES includes the T&E aspects of competitive prototyping, and demonstration of technologies in relevant environments. – The Milestone A TES relies on the Initial Capabilities Document (ICD) as the basis for the evaluation strategy 6 LO #1 6 6 File. Name. pptx

What are the T&E WIPT Products Test and Evaluation Master Plan (TEMP) at Milestone

What are the T&E WIPT Products Test and Evaluation Master Plan (TEMP) at Milestone B, C and Full. Rate Production: – MDAPs, MAIS Acquisition Programs, and programs on the OSD T&E Oversight List shall submit the initial TEMP to the USD(AT&L)DT&E and the DOT&E for approval at MS B – TEMP describes the overall structure, major elements, and objectives for the T&E program – TEMP provides an integrated test schedule; and the resource requirements to accomplish the planned testing – TEMP documents the IOT&E entrance criteria, to be used to determine IOT&E readiness 7 LO #1 7 7 File. Name. pptx

What are the T&E WIPT Products Sample Integrated Test Schedule Fiscal Year Quarter 94

What are the T&E WIPT Products Sample Integrated Test Schedule Fiscal Year Quarter 94 95 96 1 2 3 4 97 98 Requirements Technology Development Acquisition Milestones Systems Engineering SRRILA x. SFR Logistics Events Prototype Major Contract Events 00 01 Post –CDR A x CDR FRR ILA PDR TES Test & Evaluation TDFA TEMP Prototype Testing Constructive M&S Virtual M&S SIL CPD x ILA 03 04 05 1 2 3 4 06 07 1 2 3 4 IOC Production & Deployment LRIP / IOTE FRP MS C FRP DR x PRR PCR IOCSR 08 09 1 2 3 4 10 11 Operations & Support MSD/ Log Demo 12 1 2 3 4 FOC Core Capability LRIP Lot 1 / IOTE support LRIP Lot 2 LRIP Lot 3 FRP Lot 4 FRP MYP L/Lead Lot 1 x 6 L/Lead Lot 2 x 9 x 14 LRIP L/Lead Lot 3 FRP L/Lead Lot 4 x 21 L/Lead Lot 5 x 24 L/Lead Lot 6 x 24 L/Lead Lot 7 x 24 L/Lead Lot 8 L/Lead Lot 9 x 10 Flight Sim Maint. Trainers EMD IBR L/Lead EMD Training Systems 02 1 2 3 4 Engineering and Manufacturing Development MS B Production DIACAP RDTE Funding Production 99 1 2 3 4 1 2 3 4 CDD Initial Trng (T&E) OA Constructive M&S Virtual M&S SIL HITL OT Training TEMP OA TEMP LFTE Report BLRIP Report ISTF LFTE (Systems) IT 5 IT 4 IT 3 AOTR / OTRR FOTE IOT&E IATO (Type Accreditation) Phase III Validation / Cert Tests Phase IV Post Accreditation LFTE (Components) IT 1 IT 2 IATT Phase I Definition Phase II Verification and Certification Testing Total • • 8 Dynamic schedule from TD through fielding Must be consistent with AS, program plan, and System Engineering Plan (SEP) Must be accurate in dates, events, and critical path sequencing Should be reviewed at every WIPT meeting LO #1 8 8 File. Name. pptx

Why is a T&E WIPT Needed IT’S REQUIRED!! • Regulatory Basis (Army Regulation AR

Why is a T&E WIPT Needed IT’S REQUIRED!! • Regulatory Basis (Army Regulation AR 73 -1): – The materiel developer (MATDEV) will form a T&E working-level integrated product team (WIPT ). The MATDEV, program executive officer (PEO), or acquisition authority, for all systems, to include C 4 I/IT, regardless of acquisition category (ACAT) level will charter the T&E WIPT as soon as the materiel need is identified. The T&E WIPT will assist the MATDEV in managing system T&E throughout the system life-cycle. The primary objectives of the T&E WIPT are to develop and document the T&E strategy in the Test and Evaluation Master Plan (TEMP). – A T&E WIPT must be established for every program to ensure that test and evaluation integration is accomplished. The primary purpose of the T&E WIPT is to optimize the use of appropriate T&E expertise, instrumentation, facilities, simulations, and models to achieve test integration, thereby reducing costs to the Army and decreasing acquisition cycle time. – The T&E WIPT supports the integrated T&E strategy, resolves issues, and assists the MATDEV in developing and coordinating the TEMP. The primary objectives are to identify and resolve issues early, understand the issues and the rationale for the approach, and document a quality TEMP that is acceptable to all organizational levels as quickly and as efficiently as possible. 9 LO #2 File. Name. pptx

Why is a T&E WIPT Needed A T&E WIPT is needed to: • Provide

Why is a T&E WIPT Needed A T&E WIPT is needed to: • Provide a forum for all key organizations to be involved in the T&E effort. • Develop and maintain the program T&E strategy. • Develop and manage an integrated test program for M&S, developmental tests, and operational tests to support evaluations. • Develop and maintain the Integrated Test Schedule. • Identify and resolve issues. • Document a quality TES/TEMP/Schedule as quickly and as efficiently as possible – necessitates that all interested parties are afforded an opportunity to contribute to the TES/TEMP development. 10 LO #2 1010 File. Name. pptx

T&E WIPT Focus Areas in the Phases of the Acquisition Process TES ICD TEMP

T&E WIPT Focus Areas in the Phases of the Acquisition Process TES ICD TEMP CDD CPD TEMP T&E WIPT Focus Areas §WIPT Initiation §Technology Development Strategy (TDS) §Test Evaluation Strategy (TES) §ID emerging T&E capability requirements §ID T&E resources §Develop T&E requirements in RFP §Annual Report 11 LO #3 §Test and Evaluation Master Plan (TEMP) §Execute T&E Program §Provide T&E results for OIPT/DAB §CDD requirements for testability and evaluation §TRL Evaluation §T&E requirements in RFP §Annual Report §Test and Evaluation Master Plan (TEMP) §Execute T&E Program §Provide T&E results for OIPT/DAB §Support SE Tech Review (PDR) §CPD requirements for testability and evaluation §Define system capabilities and limitations §Discovery and deficiencies §Annual Report §Test and Evaluation Master Plan §Verification of corrections for deficiencies §T&E results for OIPT/DAB §OTRR §IOT&E §Annual Report §Follow-on DT and OT §Verification of corrections for deficiencies §Develop T&E programs to support upgrades, modifications, increments 11 File. Name. pptx

Starting a WIPT • • • 12 The PM should establish the program T&E

Starting a WIPT • • • 12 The PM should establish the program T&E WIPT – Identify functions/missions – Identify members/stakeholders – Appoint T&E WIPT Chair – Charter and empower the WIPT How initial membership decided – Who are the stakeholders? • Classical stakeholders include the User, Operational Tester/Evaluator, PM/Materiel Developer, Prime Contractor, Functional Experts (Logistics, PQM, etc…), Niche Experts (safety, E 3, etc…), ASA(ALT), HQDA staff – OSD Oversight? Chair generates charter, gets concurrence, and calls first official meeting LO #4 File. Name. pptx

What should you do if you are tasked by the PM to initiate a

What should you do if you are tasked by the PM to initiate a WIPT? • • • 13 Information you need: – Funding – is it adequate and available? – Documentation – Signed ADM, ICD, what is authority to proceed? – PM Direction – verbal or written Determine contact information for the following organizations to get an Action Officer assigned: – DOT&E – DDT&E – DUSA T&E – ATEC – TRADOC (TCM) Note, above list applies to ACAT 1 D programs. Smaller programs require a subset of this list. File. Name. pptx

T&E WIPT Charter • Getting WIPTs off to a good start is critical. A

T&E WIPT Charter • Getting WIPTs off to a good start is critical. A key activity is developing a charter. • The charter documents the WIPT mission, membership, roles and responsibilities, products, and establishes the timeframe in which the effort is to be completed. • The WIPT chairperson should draft the charter, to be ratified by the membership at the outset of the WIPT, and approved by the program manager. 14 LO #4 14 File. Name. pptx

T&E WIPT Charter (Continued) A charter should: • contain a clear mission statement, to

T&E WIPT Charter (Continued) A charter should: • contain a clear mission statement, to include the specific purpose and objectives of the WIPT; • provide recognition of the purpose of the WIPT in a larger context; • identify the output product, process, or service to be provided; • identify the customer or recipient of the output product, process, or service; • identify the timeframe by which the product is to be produced, the process completed, or the service provided; • identify WIPT membership, to include all the crossfunctional disciplines necessary to achieve the objectives of the WIPT and to produce the product, complete the process, or provide the service; • consider any need for training of the WIPT membership, particularly those new to the WIPT process; • address membership performance objectives that characterize a high-performance WIPT; • address product ownership and membership accountability; • address the use of metrics as a means of creating and maintaining team focus; 15 LO #4 15 File. Name. pptx

Typical Charter Outline 16 LO #4 File. Name. pptx

Typical Charter Outline 16 LO #4 File. Name. pptx

WIPT Member Relationship considerations • • 17 PM is in charge WIPT Chairman leads

WIPT Member Relationship considerations • • 17 PM is in charge WIPT Chairman leads meeting TEMP signatures are the key players ATEC, OSD, DA, and User have multiple functions (although they may have only one representative present – ATEC: AEC, OTC, Test Ranges – OSD: DOT&E, DAS(DT&E) – DA: DUSA (T&E), ASA(ALT), G 1 -G 8, PEO STRI – USER: TRADOC, TCM, Battle Labs SMEs are there to … Contractor is there to … Many people are there to observe and learn Many people are there to represent a small (but to them significant) part of the test program LO #5 File. Name. pptx

Characterization of a WIPT Key Players PM T&E Lead ATEC User OSD ? Significant

Characterization of a WIPT Key Players PM T&E Lead ATEC User OSD ? Significant Support Observers 18 LO #5 File. Name. pptx

Defense Acquisition Guidebook Suggested TEMP Format PROGRAM TITLE/SYSTEM NAME ACRONYM ACAT Level ************************************ SUBMITTED

Defense Acquisition Guidebook Suggested TEMP Format PROGRAM TITLE/SYSTEM NAME ACRONYM ACAT Level ************************************ SUBMITTED BY __________________________ Program Manager DATE CONCURRENCE __________________________ Program Executive Officer DATE or Developing Agency (if not under the Program Executive Officer structure) __________________________ Operational Test Agency DATE __________________________ User's Representative DATE Do. D COMPONENT APPROVAL __________________________ Do. D Component Test and Evaluation Director DATE __________________________ Do. D Component Acquisition Executive (Acquisition Category I) DATE Milestone Decision Authority (for less-than-Acquisition Category I) Note: For Joint/Multi Service or Agency Programs, each Service or Defense Agency should provide a signature page for parallel staffing through its CAE or Director, and a separate page should be provided for OSD Approval ************************************ OSD APPROVAL __________________________ ODUSD(A&T)/DDT&E __________________________ D, OT&E 19 ____________ DATE LO #5 File. Name. pptx

What are the T&E WIPT Organizational Relationships T&E WIPT organizational relations are both internally

What are the T&E WIPT Organizational Relationships T&E WIPT organizational relations are both internally to the PMO, and externally to all the stakeholders regarding the T&E effort, plans, resources, and results reporting External relationships include organizations involved in product/system certifications, ranges and laboratories, special functional requirements such as environmental, personnel and system safety standards and requirements, and the user community PM SE WIPT SW T&E Sub. WIPT T&E WIPT IA Sub-WIPT SME s (as needed) Other WIPTs Other USD Other Sub. Other USD WIPTs Other USD External Organizations (as needed) - - as directed or needed 20 LO #6 2020 File. Name. pptx

Challenges and Consensus Building within the T&E WIPT • Understand the Three Tenets of

Challenges and Consensus Building within the T&E WIPT • Understand the Three Tenets of WIPTs: • The PM is in charge of the program. • WIPTs are advisory bodies to the PM. • Direct communication between the program office and all levels in the acquisition oversight and review process is expected as a means of exchanging information and building trust. 21 LO #7 21 File. Name. pptx

T&E WIPT Timeline KO Meeting, Charter Signed TES for MS A TEMP for MS

T&E WIPT Timeline KO Meeting, Charter Signed TES for MS A TEMP for MS B TEMP for MS C • Development of TEMP is the primary mission of the WIPT • Kick-off might not start until after MS A decision • Typically takes 9 -12 months to develop and sign TEMP • Done concurrently with CDD/CPD and COICs • Should address all OSD initiatives (Hot buttons) • Must adequately address budget issues • Make sure draft material is seen by principals, not just Action Officers 22 LO #7 At T-60 days have good draft and generate punch list Signing Party Wo Lis rk off t is sue Punc h s En s pro ure A v bri ide d ction e sig fing t raft a Office nee o r n esp d/or rs s ons ible Ge n Ch erate art er PM etc O, AT inp EC, uts KO Develop Draft TEMP (Strawman) Re v Co iew ntr a tes t pl ctor ans Re v tes iew o t pl the DT , O ans: r T, L F Oth M& er iss DO S, RA ues: etc E, Fun M, din g, Determine Test Strategy File. Name. pptx

Challenges 23 • No one wants to bring up issues until it is time

Challenges 23 • No one wants to bring up issues until it is time to sign • 90% of the work gets done in the last 10% of the time • Many people are primarily worried about their rice bowl • Some WIPT members may not be truly empowered • Poor communications • Hidden agendas • Personalities LO #7 File. Name. pptx

Roles and Responsibilities 24 • Assist the PM in developing strategies and in program

Roles and Responsibilities 24 • Assist the PM in developing strategies and in program planning, as requested by the PM • Establish IPT plan of action and milestones (POA&M) • Propose tailored documentation and milestone requirements • Review and provide early input to documents • Coordinate WIPT activities with the OIPT members • Resolve or elevate issues in a timely manner • Assume responsibility to obtain principals’ concurrence on issues, as well as with applicable documents or portions of documents LO #7 File. Name. pptx

Six Operating Principles for Success Oriented WIPTs • CHARTERING, LAUNCH, INITIATION Getting teams off

Six Operating Principles for Success Oriented WIPTs • CHARTERING, LAUNCH, INITIATION Getting teams off to a good start is critical. Key activities include charter preparation, conducting WIPT training, and preparing a POA&M (Plan of Action & Milestones). When combined, these activities contribute to early and sustained success. • GOAL ALIGNMENT The team leader should ensure the goals and objectives of team members are consistent with project goals and objectives. An effective mechanism to provide performance feedback to team members and their functional organization should be established. • OPEN DISCUSSIONS WITH NO SECRETS Each member brings to the team unique expertise that needs to be recognized by all. Because of that expertise, each person’s views are important in developing a successful program, and these views need to be heard. Teams must have full and open discussions with no secrets. Cooperation is essential! 25 LO #7 25 File. Name. pptx

Six Operating Principles for Success Oriented WIPTs (Continued) • EMPOWERED, QUALIFIED TEAM MEMBERS Empowerment

Six Operating Principles for Success Oriented WIPTs (Continued) • EMPOWERED, QUALIFIED TEAM MEMBERS Empowerment is critical to making and keeping the agreements essential to effective WIPTs. All representatives assigned to WIPTs at all levels must be empowered by their leadership. They must be able to speak for their superiors, the “principals, ” in the decision-making process. • DEDICATED/COMMITTED PROACTIVE PARTICIPATION Consistent team participation by people with institutional knowledge of the functional areas is necessary for success. Therefore, WIPTs should be organized to allow all stakeholders to participate and right sized to the issue. Membership should be limited to the minimum essential to enhance communication and trust. Other organizations may be added as required based on the needs of the program. Contractor participation shall be in accordance with Service/Agency guidance. • 26 ISSUES RAISED AND RESOLVED EARLY Team members should openly raise and discuss issues at the earliest possible opportunity. The WIPT should try to resolve issues within the WIPT, seeking additional functional expertise when necessary. In the spirit of teaming and cooperation, issues should not be worked “off-line” beyond the purview of the IPT. When an issue cannot be resolved within the WIPT, the PM should raise the issue as quickly as possible to a decision-making level where resolution can be achieved. LO #7 26 File. Name. pptx

T&E WIPT “Do’s” and “Don’ts” Do invite Service headquarters offices to participate in the

T&E WIPT “Do’s” and “Don’ts” Do invite Service headquarters offices to participate in the T&E WIPT Do obtain appropriate OSD office(s) involvement early, especially DT&E and DOT&E Do know T&E WIPT issues & be empowered Do hold program to the rigorous T&E process Do base positions on facts, vice emotions Do elevate an issue if unable to resolve within the T&E WIPT Don’t dismiss T&E WIPT meetings as not important Don’t dismiss T&E WIPT members’ comments without reasoned debate Don't allow unresolved issues to fester within the T&E WIPT (that is, take extra effort to resolve one-on-one and face-toface) Don’t disband the T&E WIPT after TEMP approval 27 LO #7 27 File. Name. pptx

Formal Staffing of the TEMP 28 • What signatures are required after the coordination

Formal Staffing of the TEMP 28 • What signatures are required after the coordination memo is signed and the PM has signed (start of formal staffing process)? • Have the signees been pre-briefed or just kept aware by their representative? • What are their hot buttons and have they been addressed? LO #8 File. Name. pptx

Staffing Flow Chart 29 LO #8 File. Name. pptx

Staffing Flow Chart 29 LO #8 File. Name. pptx

T&E WIPT Timeline KO Meeting, Charter Signed TES for MS A TEMP for MS

T&E WIPT Timeline KO Meeting, Charter Signed TES for MS A TEMP for MS B TEMP for MS C • Development of TEMP is the primary mission of the WIPT • Kick-off might not start until after MS A decision • Typically takes 9 -12 months to develop and sign TEMP • Done concurrently with CDD/CPD and COICs • Should address all OSD initiatives (Hot buttons) • Must adequately address budget issues • Make sure draft material is seen by principals, not just Action Officers 30 LO #7 At T-60 days have good draft and generate punch list Signing Party Wo Lis rk off t is sue Punc h s En s pro ure A v bri ide d ction e sig fing t raft a Office nee o r n esp d/or rs s ons ible Ge n Ch erate art er PM etc O, AT inp EC, uts KO Develop Draft TEMP (Strawman) Re v Co iew ntr a tes t pl ctor ans Re v tes iew o t pl the DT , O ans: r T, L F Oth M& er iss DO S, RA ues: etc E, Fun M, din g, Determine Test Strategy File. Name. pptx

Army Staffing • • • 31 What is DUSA T&E looking for: – Reqts

Army Staffing • • • 31 What is DUSA T&E looking for: – Reqts Docs (CDD/CPD) and COICs are submitted with the TEMP – COIC and KPPs/KSAs clear, achievable, measurable and properly decomposed into lower level measures with traceability What is ASA(ALT)/AAE looking for: – Affordability, Program schedule realism What is User looking for: – Validation of benefit to Warfighter What is ATEC Looking for: – Ability to evaluate • Traceability of test requirements is key • Sufficient data sources (CT, DT, OT, LFT&E, M&S) How do you coordinate through complete Army Staff: – G 3, G 4, G 8, etc LO #8 File. Name. pptx

OSD Staffing 32 • What is DOT&E looking for: – Sufficient data for an

OSD Staffing 32 • What is DOT&E looking for: – Sufficient data for an evaluation of effectiveness, suitability and survivability – Reliability Growth – Experimental Design logic/rationale • What is DASD(DT&E) looking for: – Optimal use of ranges, – Coordination with other services – System performance to the KPP/KSA – Reliability Growth – Experimental Design logic/rationale – Sufficient data and test rigor to assess readiness for operational test LO #8 File. Name. pptx