Systems Engineering Best Practices Nicholas M Torelli Deputy

Systems Engineering = Best Practices Nicholas M. Torelli Deputy Director, Human Capital and Specialty Engineering Systems and Software Engineering (SSE) Office of the Deputy Under Secretary of Defense for Acquisition and Technology Presented to 2009 Systems Engineering Summit: Harnessing Best Practices Sponsored by the Huntsville Regional Chapter of INCOSE, US Army, RDECOM, AMRDEC, DAU, and NASA March 3, 2009

Do. D Vision for Systems Engineering Systems engineering principles and disciplines are fully accepted and assimilated into the Do. D acquisition workforce positioning the Do. D for acquisition excellence leading to a stronger national defense. SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 2

SSE’s 2009 Priorities • Acquisition work force development • Renewed focus on early application of systems engineering to affect affordability and total ownership cost • Increased level of visibility for our role in developmental test and evaluation • Systems of Systems engineering tools and techniques • Integration of program protection activities into acquisition oversight to address cyber threat • Measuring results of our efforts SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 3

Do. D SE Best Practice Continuum Policy & Guidance Developing the Workforce & Advancing SE Practice Program Support & Assessment Teamwork & Collaboration SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 4

Overview of Acquisition Policy Changes* • • Strategic Guidance Mandatory Materiel Development Decision (MDD) Mandatory competing prototypes before MS B Mandatory PDR and a report to the MDA (“the sliding PDR”) [PDR Report to MDA if before MS B; formal PDR Assessment by MDA if after MS B] Configuration Steering Boards at Component level to review all MS A MS B MS C requirements changes Joint Concepts CBA ICD Materiel Solution MDD Analysis JCIDS Process • Technology Development CDD Engineering and Manufacturing Development or PDR CPD Production and Deployment O&S Full Rate Production Decision Review Renewed emphasis on manufacturing during system development: – Re-titles SDD phase to EMD with two sub phases: Integrated System Design and System Capability and Manufacturing Process Demonstration – Establishes consideration of manufacturing maturity at key decision points • • Mandatory system-level CDR with an initial product baseline and followed by a Post-CDR Report to the MDA Post-CDR Assessment by the MDA between EMD sub phases * Do. DI 5000. 02, 8 December 2008 SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 5

Acquisition Policy Opportunities for SE • Early SE engagement with programs • Program Support Reviews (PSRs) Pre-MS A/B/C • Risk Reduction activities (e. g. , Technical Risk assessment in Ao. As, Competitive Prototyping) • SE Technical Reviews - Informed Trades for Feasible Solutions • Developmental Test and Evaluation – Integrated DT/OT – Updated T&E Strategy at MS A SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 6

New Systems Engineering Enclosure • Codifies several previous SE policy memoranda • Codifies a number of SE-related policies and statutes since 2003: – – – Environmental Safety and Occupational Health Modular Open Systems Approach Data Management and Technical Data Rights Item Unique Identification Spectrum Supportability Corrosion Prevention and Control • Introduces new policy on Configuration Management SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 7

Do. D Guidance • • • Defense Acquisition Systems Engineering Developmental Test and Evaluation (DTE) Modeling and Simulation (M&S) Safety System Assurance http: //www. acq. osd. mil/sse/pg/guidance. html SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 8

Do. D SE Best Practice Continuum Policy & Guidance Developing the Workforce & Advancing SE Practice Program Support & Assessment Teamwork & Collaboration SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 9

OSD Systems Engineering Strategy DSB DT&E Report & 231 Report Recommendations • • Integrated DT/OT Reliability Improvement Early SE Access to Relevant Data for Evaluation & Decision Making DT&E Revitalization Milestones A/B Critical Correct SE Staffing Component Development Planning Pre-MS A analysis OSD Policy & Guidance NRC Study Recommendations • • Reliability Workforce Development Systemic Analysis Recommendations* Enhanced SE Pre-MS B • Achievable Acquisition Strategy • Enhanced Gate Review Process • Enhanced Staff Capabilities *Based on 3, 700 Program Assessment findings from 40 Programs Support Reviews SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 10

Workforce Development The Defense Acquisition Community 126, 033 Government and Military Certified Professionals 500, 000+ Defense Industry Personnel SSE Functional Leader for SE, T&E, and PQM workforce SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 11

SE Human Capital Strategy • FY 08 NDAA Section 852: Do. D Acquisition Workforce Development Fund: >$300 M per year across Do. D – SE, PQM and T&E initiatives to Recruit, Train, and Retain the workforce – Do. D Human Capital Initiative - Published Annex for SPRDE, PQM, and T&E Career Fields • SE “core competency” assessment effort; completion Summer 2009 • “Program Systems Engineer” career path • Partnership with INCOSE SE Certification Program – “CSEP-Acq” aligned with Defense Acquisition Guidebook • Equivalency granted for DAU Courses SYS 101 and SYS 202 – Expanding potential for Industry “certifications” • what the future could hold SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 12

Human Capital Initiatives (Defense Acquisition Workforce Development Fund 1) Defense Acquisition Workforce Development Fund Recruiting & Hiring Training & Development Retention & Recognition Intern Programs Training Enhancement & Capacity Expansion Retention & Recognition Recruiting Incentives Acquisition Workforce Management & Student Information System Career Broadening & Academic Programs Outreach Programs Competency Management & Assessments Journeyman Hiring Workforce Planning Highly Qualified Experts 1 Based on NDAA Section 852, Defense Acquisition Workforce Development Act SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 13

Workforce Size Notional Do. D Systems Engineering Population 25 -35 35 -45 45 -55 Workforce Age SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) >55 14

Notional Do. D Systems Engineering Workforce Strategy Train: Mentor Recruit: Highly Qualified Experts Retain Recruit: Journeymen Re In crui te t: rn s Workforce Size Retain: Rotational Training Assignments Education with Industry Qualification Cards Incentives Hazardous Duty Pay for PSEs? ? 25 -35 35 -45 45 -55 Workforce Age SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) >55 15

NDIA Systems Engineering Division E&T Committee Mission: Strengthen Systems Engineering capabilities through education, training and experience across government, industry, and academia. Contact Govt Committee Chair: Dr. Don Gelosh, CSEP-Acq ODUSD (A&T) SSE / HC 703 -695 -0472 donald. gelosh@osd. mil SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 2009 Task: To address the issue of “self-proclaimed” Systems Engineers, the E&T Committee is attempting to determine the core essential set of competencies that distinguish systems engineers from domain engineers. 16

SE Competency Model Life Cycle View Systems Thinking Unit of Competence Element(s) Analytical Essence of Systems Behavior Structure Goal Analytical Problem Solving Lvl Problem Solving Method Factors that Affect the SE Competency Profile Unit of Competence Element(s) Lvl Material Solution Analysis Technology Development EMD Production & Deployment Systems Engineer Composite Competency Map SE Technical Competence Analytical Stakeholder Requirements Analysis Analytical Element(s) Lvl Minimum Essential SE Criteria Architecture Design Implementation Comp. Level Unit of Competence wth Gro Competence Technical Management Decision Management Technical Management Risk Management Technical Management Configuration Management Technical Management Element(s) Professional / Interpersonal Unit of Competence Element(s) Lvl Time SE Technical Mgmt Unit of Competence Operations & Support Domain Knowledge Lvl Technical Planning SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 17

Research to Practice Pro vid Policy Guidance Education & Training Ongoing Development Acquisition Community (Do. D and Industry) Del ive r. N e. L ess o Lea r SERC Governance & Collaborators • Co-Sponsors • Government PMO • Tasking Activities • Industry • Associations • Academia ew /I ces mpro ve ses & T d SE M ool s (M etho ds, PTs ) Pro ns SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) ned and Cha llen ges SERC 18

Need for SE Research • State-of-the-practice must keep up with application needs of Do. D acquisition – Methods, processes, and tools to enable effective acquisition and sustainment of systems – Leveraging Modeling & Simulation for SE • Size and complexity of modern systems drive need for commensurate extensions of SE, – To System of Systems (So. S), to complex systems, to “architecting, ” to net-centric sets of services – Validation & Verification challenges • SE “theory and practice” should be inclusive of, and establish linkages to, challenged sub-specialty areas – Software engineering, reliability engineering, system safety, costing, etc. SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 19

Stevens-SERC Team Stevens Institute of Technology Lead University Partners • • Auburn University Air Force Institute of Technology Carnegie Mellon University Fraunhoffer Center at UMD Massachusetts Institute of Technology Missouri University of Science and Technology (S&T) Pennsylvania State University Southern Methodist University • • • Texas A&M University Texas Tech University of Alabama in Huntsville University of California at San Diego University of Maryland University of Massachusetts University of Southern California University of Virginia Wayne State University The Do. D Systems Engineering Research Center will be responsible for systems engineering research that supports the development, integration, testing and sustainability of complex defense systems, enterprises and services. Stevens has MOUs to develop enhanced SE courseware for competency development within Do. D with AFIT, NPS and DAU. Further, SERC members are located in 11 states, near many Do. D facilities and all DAU campuses. SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 20

Initial SERC Research Tasks • “Assessing Systems Engineering Effectiveness in Major Defense Acquisition Programs (MDAPs)” – USC task lead, with support from Stevens, Fraunhofer Center, MIT, University of Alabama at Huntsville – Characterize SE effectiveness within the context of Do. D acquisition and identify methods of measurement suitable for application to project execution organizations (e. g. , defense contractors), program management organizations (e. g. , project managers and program executive offices) and oversight organizations (e. g. , OUSD(AT&L)) (OSD) • “Evaluation of Systems Engineering Methods, Processes, and Tools (MPT) on Department of Defense and Intelligence Community Programs” – Stevens task lead, with support from USC, University of Alabama at Huntsville, and Fraunhofer – Examine and recommend areas for advancing current SE methods, processes, and tools (MPTs) as they are applied across the Do. D acquisition life cycle focusing on three different development environments: individual weapons systems, So. S, and network-centric systems (NSA) SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 21

Ways Your Organizations Can Become Involved… • Identify SE research challenges • Provide funding to sponsor research as a Tasking Activity • Collaborate on Do. D-sponsored research by identifying pilot programs to participate, provide relevant acquisition information, identify subject matter experts • Make use of research findings to improve systems engineering on acquisition programs SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 22

Do. D SE Best Practice Continuum Policy & Guidance Developing the Workforce & Advancing SE Practice Program Support & Assessment Teamwork & Collaboration SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 23

Our World of Stakeholders • Do. D – – Components and Agencies Product Centers Program Executive Offices Acquisition Programs (PMs and CSEs) – Practicing PMs and SEs – Other DAWIA Career Fields • Industry – First tier contractors (e. g. , Lockheed Martin, Boeing, Northrop-Grumman, Raytheon) – Second tier contractors – Practicing SEs – SE tool vendors SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) • Education & Training Institutions – Do. D organizations: DAU, AFIT, NPS – US universities with SE programs (~60) – SE short course providers • Professional and Industrial Associations – DAU Alumni Association – INCOSE – NDIA SE Division, other assoc Div – IEEE Standards Committee; Systems Council – AIAA SE Committee – Tech. America – ISO 24

Do. D Challenge: “A”cquisition Process • ICD, CDD, CPD • Future Threat and Op Tempo • Planning, Programming, and Budgeting Process • FYDP • POM Process • Appropriation and Authorization of Funds Budgeting Requirements “Sweet Spot” • Acquisition Policy, Guidance, and Oversight. • MDAP Decision Authority Acquisition SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 25

Do. D Goal: Increase the Overlap Budgeting Requirements “Sweet Spot” Acquisition SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 26

Systems & Software Engineering Organization Overview SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 27

Opportunities for SSE Engagement SSE Statutory Direction Policy & Guidance Congress • Systems Engineering • DT&E Program Support • Program Support Reviews • OIPT, T&E and SE WIPTs Sec Def Requirements Developers Service Acquisition Executives • AOTR, Post-CDR Review & Assessment PEOs Workforce Planning Program Offices • Competency Models • Certification Reqts • Education & Training Emerging Concepts • Systems of Systems • SE Research AT&L Direction ICD, CDD, CPD DAB, ITAR, DSAB, OIPT Prime Contractors Second Tier Contractors Education & Collaboration Infrastructure Professional / Industry Associations (NDIA, INCOSE, AIA, ITEA, Tech. America, etc. ) DAU, Academic Institutions, SERC Outreach PSR, SEP, TEMP, Technical Reviews CSEP-Acq, Research, Industry-University SE Workforce Initiatives, Industry-Gov’t Projects • SE Forum • Engagement Strategy SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 28

Do. D Human Systems Integration (HSI) Problem • HSI is not being consistently integrated within Do. D acquisition resulting in increased ownership costs and lower system effectiveness. Approach • • Assign DUSD(A&T) Executive Authority to define policy, as necessary, to implement changes. Maximize the good work that has been done (policies, plans, methods, tools and standards). Engage Do. D stakeholders and strengthen industry affiliations including NDIA, Tech. America and INCOSE. Identify gaps and apply necessary resources to resolve gaps or barriers to successful HSI implementation. Cross-Do. D and Technical Community Collaboration SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 29

Collaboration with INCOSE • International Panel: “What Defines a Systems Engineer? Comparing and Contrasting Global Perspectives on Systems Engineering Competency” – – – – July 19 – 23, 2009 Moderator: Dr. Don Gelosh, Human Capital Strategy and Planning, OSD Directorate of Systems and Software Engineering Dr. Arthur Pyster, Distinguished Research Professor, School of Systems and Enterprises, Stevens Institute of Technology; Deputy Executive Director, Do. D Systems Engineering Research Center; and member of the INCOSE Board of Directors Dr. John Snoderly, Program Director, Systems Engineering, Defense Acquisition University Samantha Brown, President-Elect of INCOSE and Systems Engineering Innovation Centre (SEIC), Loughborough, UK Mark Kupeski, Director, Complex Systems Integration, IBM Global Business Services Dr. U. Dinesh Kumar, Professor in Quantitative Methods and Information Systems, Indian Institute of Management Bangalore Professor Stephen Cook, Director, Defence and Systems Institute, University of South Australia SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 30

INCOSE Multi-Level Professional Certification Senior Level Foundation Level Discipline Extensions Entry Level ESEP CSEP-Acq CSEP ASEP Expert Systems Engineering Professional CSEP w/ US Do. D Acquisition Certified Systems Engineering Professional Associate Systems Engineering Professional SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 31

CSEP with US Do. D Acquisition Extensions • Targeted towards Systems Engineers who support or work in a US Department of the Defense acquisition environment • Same core CSEPs experience, education, and knowledge requirements • Additional acquisition knowledge items tested • Available since July 2008 SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 32

Do. D Best Practices Clearinghouse (BPCh) The Do. D Acquisition Best Practices Clearinghouse (BPCh) facilitates the selection and implementation of systems engineering and software acquisition practices appropriate to the needs of individual acquisition programs. https: //bpch. dau. mil/ The BPCh uses an evidencebased approach, linking to existing resources that describe how to implement various best practices. SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 33

We Need You! • • • Browse the Clearinghouse for applicable best practices Provide feedback on practices Submit additional evidence for an existing practice Submit a lead or a possible best practice Tell others about the Clearinghouse Consider becoming an editor to help vet practices submitted in your area of expertise SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 34

Do. D SE Best Practice Continuum Policy & Guidance Developing the Workforce & Advancing SE Practice Program Support & Assessment Teamwork & Collaboration SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 35

OSD’s Program Support Reviews… “The Source of Systemic Root Cause Analysis Data” Rigor in process … Defense Acquisition Program Support Methodology (DAPS) v 1. 1 Plan Conduct Analyze Tools & Materials • Tailorable, Scalable Methodology (DAPS v 1. 1) • Templates • Criteria • Questions • Training Materials • Execution Guidance • Subject Matter Expertise Report …ensures integrity SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 36

Program Support Review Data Set (since March 2004) PSRs/NARs completed: 57 AOTRs completed: 13 Nunn-Mc. Curdy Certification: 13 Participation on Service-led IRTs: 3 Technical Assessments: 13 Reviews planned for CY 08: Data PSRs/NARs: 10 AOTRs: 1 NM: 1 derived from diverse & broad program set SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 37

OSD Systemic Analysis: Data Model Tactical, Program and Portfolio Management Use collective set of program findings… Program Review Findings Program Effects & Root Cause Program Unique Solutions … to identify systemic issues Strategic Management at the root cause level… Systemic Issues … and develop recommendations that mitigate problems at their source! Do. D Acquisition Community SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) Systemic Root Causes • Policy/Guidance • Education & Training • Best Practices Corrective Actions • Other Processes (JCIDS, etc) • Oversight (DAB/ITAB) • Execution (staffing) 38

Deriving Systemic Issues… PSR findings 44 Programs “Negative” Findings Analysis Pareto & Pivot Tables) Findings Prelim. Analysis Results Systemic Issues Hypothesis testing: Re-analyze, review… • Use “Core” and “Systemic” Root Cause tagging to cull data into focused subsets • Analyze and trend using Pareto charting and Pivot Tables Occurred on 50% or more of the programs SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 39

SRCA Milestones & Results Sep 07 – Feb 08 Mar 08 – Oct 08 – Dec 08 OUSD/SSE root cause analysis yields systemic issues and preliminary recommendations OUSD/SSE sponsored NDIA Task Group to develop executable acquisition recommendations NDIA Task Group concludes effort and proposes actions for both government and industry 3 Recommendation Areas Early & sufficient planning Acquisition Strategy & Planning (ASP) Decision Gate Review (DGR) Clear, enforceable execution criteria SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) Enhanced Staff Capability (ESC) People: the right skills and numbers 40

New Opportunities for Independent Reviews What’s relevant: Mandatory Milestone A for all “major weapon systems” MS B after system-level PDR* and a PDR Report to the MDA EMDD with Post-CDR* Report and MDA Assessment PSR and AOTR in policy MS A Strategic Guidance Joint Concepts CBA ICD MS B Materiel Solution Technology CDD MDD Analysis Development Potential Independent Technical Reviews - PSRs and AOTRs Program Support Reviews (PSRs) All ACAT ID and IAM To inform the MDA on technical planning and management processes thru risk identification and mitigation recommendations To support OIPT program reviews and others as requested by the MDA * PDR – Preliminary Design Review Engineering and Manufacturing Development and Demonstration CPD Production and Deployment OTRR O&S Full Rate Production Decision Review Assessments of Operational Test Readiness (AOTRs) All ACAT ID and special interest programs To inform the MDA, DOTE, & CAE of risk of a system failing to meet operational suitability and effectiveness goals To support CAE determination of materiel readiness for IOT&E * CDR – Critical Design Review SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) MS C * OTRR – Operational Test Readiness Review 41

Do. D SE Best Practice Continuum Policy & Guidance Developing the Workforce & Advancing SE Practice Program Support & Assessment Teamwork & Collaboration SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 42

Always Our Focus The Mission: Delivering Timely and Affordable Capabilities to the Warfighter The Defense Acquisition Community 126, 033 Government and Military Certified Professionals 500, 000+ Defense Industry Personnel For additional information: http: //www. acq. osd. mil/sse SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 43

Do. D Guidance (1/2) • Defense Acquisition – – • Defense Acquisition Guidebook (DAG) Integrated Defense Acquisition, Technology, & Logistics Life Cycle Management Framework, Version 5. 3. 2: December 8, 2008 Systems Engineering – – – – DAG Chapter 4, Systems Engineering Plan (SEP) Preparation Guide, Version 2. 01, April 2008 SEP Frequently Asked Questions (FAQs), February 8, 2008 SE WIPT Brief: An Overview of Technical Planning and Systems Engineering Plan (SEP) Development, Version 1. 0, March 20, 2008 SE Working Integrated Product Team (WIPT) Generic Charter Template Systems Engineering Guide for Systems of Systems, Version 1. 0, August 2008 Systems Engineering Assessment Methodology, Defense Acquisition Program Support (DAPS), Version 2. 0, October 21, 2008 Guide for Integrating Systems Engineering into Do. D Acquisition Contracts, Version 1. 0, December 11, 2006 Risk Management Guide for Do. D Acquisition, 6 th Edition, Version 1, August 2006 Integrated Master Plan / Integrated Master Schedule Preparation and Use Guide, Version 0. 9, October 21, 2005 Program Manager's Guide: A Modular Open Systems Approach (MOSA) to Acquisition, Version 2. 0, September 2004 Modular Open Systems Approach (MOSA) Program Assessment and Rating Tool (PART), Version 1. 02 Do. D Guide for Achieving Reliability, Availability and Maintainability, August 1, 2005 Program Reliability and Maintainability Review Template, Version 1. 0, August 15, 2008 Designing and Assessing Supportability in DOD Weapon Systems: A Guide to Increased Reliability and Reduced Logistics Footprint, October 24, 2003 Technical Review Checklists http: //www. acq. osd. mil/sse/pg/guidance. html SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 44

Do. D Guidance (2/2) • Developmental Test and Evaluation (DTE) – DAG Chapter 9, Integrated Test and Evaluation – Guide on Incorporating Test and Evaluation into Do. D Acquisition Contracts, FINAL DRAFT FOR FINAL APPROVAL VERSION, September 2008 • Modeling and Simulation (M&S) – Do. D 5000. 59 -M, Do. D Modeling and Simulation (M&S) Glossary, January 15, 1998 – Recommended Practices Guide for Verification, Validation, and Accreditation (VV&A), Build 3. 0, September 2006 – Guide for Modeling and Simulation for the Acquisition Workforce, Version 1. 01, October 2008 • Safety – System Safety — ESOH Management Evaluation Criteria for Do. D Acquisition, Version 1. 1, January 2007 – Joint Systems Safety Review Guide for USSOCOM Programs, Version 1. 1, October 12, 2007 – Unmanned System Safety Guide for Do. D Acquisition, Version 0. 92, June 27, 2007 – Joint Weapons and Laser Safety Review Guide (Draft), Version 0. 92, September 4, 2007 – ESOH in Acquisition: Integrating ESOH into SE, Version 3. 0, January 2008 • System Assurance – – DAG Chapter 7, Acquiring Information Technology and National Security Systems DAG Chapter 8, Intelligence, Counterintelligence, and Security Support Do. D 5200. 1 -M, Acquisition Systems Protection Program, March 16, 1994 Engineering for System Assurance, Version 1. 0, October 2008 http: //www. acq. osd. mil/sse/pg/guidance. html SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 45

Do. D Best Practices Clearinghouse (BPCh) Program Goals Useful Information Active Knowledge Base Help finding, selecting, and implementing practices appropriate to User’s situation; fill the gap between “what” and “how” Not just another practice list; experience data updated, expanded refined; encourages organic growth A Single Source For answers about practices, how to apply them, when they are good to use; lessons learned; and risks to avoid Repository Living Knowledge Integration with Do. D communities of practice (ACC) and experts SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) Validated practices Consistent, verifiable information 46