UNCLASSIFIED FOUO Office of the Secretary of Defense

UNCLASSIFIED // FOUO Office of the Secretary of Defense Test Resource Management Center: Opportunities for HBCU/MI Mr. Geoff Wilson Deputy Program Manager Test and Evaluation / Science and Technology Program (T&E/S&T) TRMC Office of the Secretary of Defense September 12, 2017 Taking the Pentagon to the People – University of the District of Columbia UNCLASSIFIED // FOUO 1

UNCLASSIFIED // FOUO Discussion • TRMC Overview • Technical Workforce Challenges • TRMC-Directed Research Opportunities for HBCU/MIs UNCLASSIFIED // FOUO

UNCLASSIFIED // FOUO Discussion • TRMC Overview • Technical Workforce Challenges • TRMC-Directed Research Opportunities for HBCU/MIs UNCLASSIFIED // FOUO

UNCLASSIFIED // FOUO TRMC Organization As of 1 Sep 2017 Under Secretary of Defense for Acquisition, Technology & Logistics The Honorable Ellen M. Lord Chief Financial Officer Chief Operating Officer Deputy EA for Cyber Test Ranges Director, TRMC Mr. G. Derrick Hinton (Acting) PM, NCR Complex Expansion Deputy PM, NCR Complex Expansion Principal Deputy, TRMC Mr. Paul D. Mann (Acting) Deputy Director, Major Initiatives and Technical Analyses PM, T&E/S&T // PM, JMETC Deputy Director, T&E Range Oversight Agency Range Oversight Air Force Range Oversight PM, CTEIP Army Range Oversight Navy Range Oversight Deputy PM, CTEIP PM, REP Director, TENA SDA Deputy PM, T&E/S&T UNCLASSIFIED // FOUO Deputy PM, JMETC Director, NCR Complex Deputy Director, NCR Complex

UNCLASSIFIED // FOUO TRMC Missions Do. D (Charter) Directive 5105. 71 MRTFB OVERSIGHT / T&E INFRASTRUCTURE • Plan for and assess the adequacy of the MRTFB to provide adequate testing in support of the development, acquisition, fielding, and sustainment of defense systems • Support the Department’s objective of ensuring compliance with Do. DD 7000. 14 -R • Review proposed significant changes to T&E facilities and resources of the MRTFB before they are implemented by the Do. D Components • Issue guidance to the Do. D Components, through the USD(AT&L), with respect to MRTFB planning • Maintain an awareness of other T&E facilities and resources, within and outside the Do. D, and their impacts on Do. D requirements • Serve as Executive Agent for Cyber Test Ranges STRATEGIC PLAN • Complete a strategic plan for T&E not less often than once every 2 fiscal years BUDGET CERTIFICATION • Submit report to the SECDEF containing the comments of the Director concerning all such proposed budgets, together with the Director’s certification as to whether such proposed budgets are adequate PROGRAMS • Administer the CTEIP (Central Test and Evaluation Investment Program) and T&E/S&T Program CAPABILITIES • Manage and operate the JMETC (Joint Mission Environment Test Capability) Multiple Independent Levels of Security (MILS) Network and the Regional Service Delivery Points (RSDP) cloud computing environments • Manage and operate the NCR (National Cyber Range) (IAW RMD 407 A 1, Issue #1, Title: Cyber, Jan 12, 2015) to provide test capability and capacity for the T&E Community Statutory UNCLASSIFIED // FOUO Regulatory 5

UNCLASSIFIED // FOUO Major Range and Test Facility Base (MRTFB): The “Critical Core” 23 Sites: Army-8; Navy-6; Air Force-7; Defense Agency-2 Legend: Updated 22 Jul 2015 Army, Navy, AF, Defense Agency NUWC Keyport (Nanoose & Dabob ranges) Keyport, WA Cold Regions Test Center Fort Greely, AK NAWC-WD China Lake, CA 30 th Space Wing Vandenberg AFB, CA NAWC-WD Point Mugu, CA • Created by Congress • Army – 5. 5 M Acres Land • 25, 000+ Personnel • Navy – 1 M Acres Land, 165 K sq mi Sea, 103 K sq mi Air • ~$4 B in Funding Annually • Air Force – Western Ranges ALONE – 5 M Acres Land Nevada Test and Training Range Nellis AFB, NV Utah Test and Training Range Hill AFB, UT West Desert Test Center Dugway Proving Ground, UT Yuma Test Center Yuma Proving Ground, AZ Electronic Proving Ground Fort Huachuca, AZ 412 th Test Wing Edwards AFB, CA U. S. Army Kwajalein Atoll Reagan Test Site DISA, JITC Fort Huachuca, AZ Tropic Regions Test Center, various locations PMRF Kauai, HI Aberdeen Test Center, Aberdeen Proving Ground, MD DISA Ft. Meade, MD NAWC-AD Patuxent River NAS, MD Arnold Engineering Development Complex, Arnold AFB, TN 45 th Space Wing, Patrick AFB, FL 96 th Test Wing (Includes 96 th Test Group), Eglin AFB, FL White Sands Test Center, White Sands Missile Range, MN UNCLASSIFIED // FOUO Atlantic Undersea Test and Evaluation Center, Andros Island, Bahamas 6

UNCLASSIFIED // FOUO Discussion UNCLASSIFIED // FOUO

UNCLASSIFIED // FOUO Test & Evaluation / Science & Technology (T&E/S&T) Program Mission: Develop Technologies Required to Test Future Warfighting Capabilities l Established in FY 02 l l l Joint DDR&E / DOT&E Initiative Transitioned to TRMC in FY 05 l l RDT&E Budget Activity 3 funds Purpose l l l Annual Broad Agency Announcements (BAAs) l Tri-Service working groups l High Risk / High Payoff R&D for Testing Foster technology transition to major Do. D test ranges Risk reduction for test capabilities developments l l l Academia Industry Government Laboratories 71 Active Projects Validate requirements Evaluate proposals Facilitate technology transition Central Oversight – Distributed Execution Eight Test Technology Areas High Speed Systems 15 Active Projects Unmanned & Autonomous Systems 4 Active Projects Spectrum Efficiency 8 Active Projects Advanced Instrumentation 12 Active Projects Directed Energy Cyberspace Electronic Warfare 9 Active Projects 2 Active Projects 16 Active Projects C 4 I & Software Intensive Systems 5 Active Projects *Outyear Planning Profile Shaping Technology into Tomorrow’s T&E Capabilities UNCLASSIFIED // FOUO 8

UNCLASSIFIED // FOUO Game-Changing Technologies Challenges in Hypersonic T&E • Hypersonic flight environment is difficult to replicate on the ground • Hypersonic flows are difficult to accurately model Scramjet Test at NASA Langley Material Development Challenges • • • Arc Jet Heater UNCLASSIFIED // FOUO Extreme heat requiring Thermal Protection System (TPS), shell made of advanced materials TPS materials tested in Arc Jet Heaters up to 180 atmospheres & 13, 000 deg. R Arc jets achieve these extreme temperatures by flowing high pressure air over an electric arc and through a supersonic nozzle Bottom Line: Designing test facilities is just as challenging as designing hypersonic flight vehicles Hypersonic Vehicle in Flight

UNCLASSIFIED // FOUO Game-Changing Technologies Managing T&E of Autonomous Systems We Must Transform T&E Culture We Must Develop Accepted • Cross-domain standards, metrics, and test methodologies • Human-Robot interactions • Multiple interacting domains (Air, Land, Sea, Undersea) ranges Georgia Tech Research Institute Study To assess and develop an approach to gaps and required investments in T&E infrastructure for autonomous systems UNCLASSIFIED // FOUO 10

UNCLASSIFIED // FOUO Test and Evaluation Range in Action Bidirectional Secure Range Network OTHER RANGES CEC Unmanned Systems & Networks TAC NET C 2 CONTROL NET Mobile Apps & Adaptive Networks Joint Task Force UNCLASSIFIED // FOUO C 4 I Providers Range Control Facility

UNCLASSIFIED // FOUO High Pressure High Temperature Gas Tunable Diode Laser Absorption Spectroscopy Temperature (HTDLAS) Stanford University, Air Force Research Lab (AFRL), & Air Force Office of Scientific Research Description: Developing Tunable Laser Absorption Spectroscopy (TLAS) instrument for temperature measurements in high pressure, high temperature Air Enables: • Non-intrusive air temp measurements at high pressure and temperatures • Nitric oxide (NO) based TLAS to provide the needed temperature measurements with low experimental uncertainties • Evaluation of turbine and scramjet engine combustion properties TDLAS Flange Current Status: Completing final report and delivering HTDLAS instrumentation to Hypersonic Aerothermal Propulsion Clean Air Testbed (HAPCAT) FY 14 Major Accomplishments ü Complete shock tube spectral studies, including 3 -color absorption ü Create lookup tables for temperature vs spectral absorption FY 15 - FY 17 Major Accomplishments ü Investigate one-, two- and three-color (wavelength) nitric oxidebased TLAS (NO-TLAS) for low and high temperature ranges ü Identify optimum wavelengths ü Obtain shock tube data on 3 separate NO spectral lines ü Develop HAPCAT installation requirements q Design the hardware needed to install HTDLAS at HAPCAT q Install & perform shakedown testing at HAPCAT q Final Report and HTDLAS instrument delivered to HAPCAT Key Future Events: Q 4 FY 17: Install & perform shakedown testing at HAPCAT Q 4 FY 17: Final Report and HTDLAS instrument delivered to HAPCAT Deliverables (Mo/Yr): q Final Report Sep/17 q HTDLAS Instrumentation Sep/17 Potential Transition Partner / Date: Hypersonic Aerothermal Propulsion Clean Air Testbed (HAPCAT) UNCLASSIFIED // FOUO

UNCLASSIFIED // FOUO Cyber Threat Automation & Monitoring (CTAM) Florida International University-ARC Miami FL Description: Developing technologies to detect, monitor, and analyze malware behavior during cyber attacks in a virtualized T&E environment Enables: • Virtual machine introspection, data collection, and monitoring through a centralized hypervisor system • Artificial Intelligence / Machine Learning technology to monitor and analyze malware in virtualized testbed using classification and anomaly detection algorithms • Visualization and generation of reports to assess the potential damage to operational cyberspace assets Current Status: Development of XEN and QEMU hypervisors with full introspection instrumentation package for various Operating Systems FY 16 Major Accomplishments ü CTAM T&E Infrastructure Development ü Advanced Cyber Analytics and Virtualization Platform Design ü Test Control Center Virtual Machine Mgt Module Key Future Events: August 2017 : Technical Interchange Meeting FY 17 Major Accomplishments ü Smart Memory Acquisition and Virtual Machine Introspection ü Machine Learning Algorithms and Models ü Test Control Center Development and Platform Integration ü Phase 1 Demo and Cyber Fellows Induction Ceremony q Test Control Center Enhancements q Scaling of Virtualization Platform q Advanced Machine Learning Algorithms Implementation q Malware Analysis & Repository Deliverables (Mo/Yr): ü Phase 1 Final Report Apr/17 q Phase 2 Demonstration Dec/17 q Phase 2 Final Report Jan/18 Potential Transition Partner / Date: National Cyber Range, Regional Service Delivery Point (RSDP), Avionics Cyber Range (ACR) and Cyber. TASE UNCLASSIFIED // FOUO

UNCLASSIFIED // FOUO T&E/S&T Program Broad Agency Announcements • In-cycle Process – Declared schedule for white paper submissions (deadline) – Requires an allocation of available funding – Priority over out-of-cycle white paper submissions • Out-of-cycle Process – White papers submitted anytime after "in-cycle" deadline or to define a BAA cycle where a “new start” budget has not been quantified – Offerors required to contact Executing Agent before submitting white paper − Ensures interest; mitigates unnecessary effort by Offeror − Can address whether a ‘possibility' exists for funding • All BAAs include an “Other Test Technology” category to enable offerors to propose test technology not included in the list of topics identified by the Government UNCLASSIFIED // FOUO

UNCLASSIFIED // FOUO T&E/S&T BAA Location www. fbo. gov Posted Date: [change to blank] Keyword: TRMC Out-of-Cycle Only: Must Discuss Concept with Executing Agent Prior to Submittal UNCLASSIFIED // FOUO

UNCLASSIFIED // FOUO FY 18 Project Selection Schedule Activity T&E/S&T Project Selection Process Schedule JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN Draft BAA Topic Areas BAA Topics Finalized BAA Topics Thru Legal BAA Topics Announced Call for White Papers White Paper Submissions White Paper Reviews Proposals Requested from Selected Offerors BAA Pre-FY Cycle Proposal Submissions Proposals Review Contract Clarifications Executing Agents Recommendations to PM PM New Start Decisions Contract Negotiations & Awards Program Manager Action Executing Agent Action Contracting Officer Action UNCLASSIFIED // FOUO Offeror Action 16

UNCLASSIFIED // FOUO Successful White Papers / Proposals • Satisfy T&E need • Meet S&T challenge • Demonstrate technical feasibility • Convey breadth of technology application • Imply utility via desired application/T&E CONOPS • Compatible with TTA portfolio UNCLASSIFIED // FOUO

UNCLASSIFIED // FOUO National Cyber Range Complex (NCRC) Background Mission: Improve the resiliency of our warfighters in the cyber-contested battlespace by conducting testing, training, and mission rehearsal events in operationallyrepresentative cyberspace environments • Developed by Defense Advanced Research Projects Agency (DARPA) in the 2009 -2012 timeframe • Transitioned from DARPA to the Do. D Test Resource Management Center (TRMC) in October 2012 • Provides secure facilities, innovative technologies, repeatable processes • Creates custom-fidelity, mission representative cyberspace environments • Facilitates the integration of the cyberspace T&E infrastructure through partnerships with key stakeholders across government, industry, and academia TRMC institutionally funds the NCR to support the full spectrum of Do. D T&E and training customers UNCLASSIFIED // FOUO 18

UNCLASSIFIED // FOUO National Cyber Range Complex Vision • Common Cyber Testing Ontology, Taxonomy, and Lexicon – How we communicate • Common Concept of Operations – How the cyber ranges operate • Common Cyber Test Data Model – How to describe a cyber test • Common Cyber Descriptions (Object Definition and Transforms) – How components of the cyber test are defined • Common Cyber Universe Description – How the environment that surrounds the components of the cyber test are defined • Common Instrumentation and Control – How data is collected and assessed – How cyber tests are controlled SCADA SITES JIOR JMN TECHNICAL CHARACTERISTICS • Realistic: Large-scale, high-fidelity virtualized cyber environments operating actual software integrated with hardware-in-the-loop capabilities • Repeatable: Archived, reusable environments, procedures, parameters, and event restoration checkpoints to facilitate test-fix-test verification • Rapid: Standard tools and processes to automatically create, re-create, and modify mission-specific environments • Isolation: Cryptographic segregation of multiple, concurrent cyber environments at varying security classifications • Sanitization: Restore all assets to a known, clean state – not just range infrastructure, but also mission system equipment E n a b l e d b y a C y b e r - S a v v y W o r k f o r c e Manage and operate the National Cyber Range (IAW RMD 407 A 1, Issue #1, Title: Cyber, Jan 12, 2015) to provide test capability and capacity for the T&E Community UNCLASSIFIED // FOUO 19

UNCLASSIFIED // FOUO Sanitization is an Example of Multi-Pronged Approach to Mature Best-of-Breed Technologies for Utilization • Communicating technology development needs in multiple forums promotes best-of-breed solutions • Multi-pronged approach to augmenting sanitization capabilities includes the following initiatives: • Developing broader methodologies and tools to sanitize a range of mission system hardware provides several benefits: – Increased throughput / rapid turnaround – Increased event scope – Higher fidelity to provide greater operational realism • • – Framework for Automated and Verified Sanitization (FAVS) (Lockheed Martin) § Effort progressing to establish and demonstrate tools and procedures to restore systems to fully mission capable status (with confidence) § Initial emphasis is on T&E, training, and mission rehearsal—in the future, this could provide real-time sanitization of systems in the operational environment – Avionics Attestation (TRMC / AF/TE, AFRL) § Determine whether or not an aircraft preparing for a mission contains malware or maldata § Minor modifications in avionics memory enable attestation to detect a change before a cyber attack occurs – Customized server components for NCR (Dell; unsolicited) § Tailored hardware and software to reduce sanitization time for NCR infrastructure assets Critical research area; expect to spend $50 M over the next 5 years GOAL: Reduce Dependence on Virtualized Systems UNCLASSIFIED // FOUO 20

UNCLASSIFIED // FOUO Discussion • TRMC Overview • Technical Workforce Challenges • TRMC-Directed Research Opportunities for HBCU/MIs UNCLASSIFIED // FOUO

UNCLASSIFIED // FOUO Technical Workforce Demand • The Do. D Cyber T&E Community anticipates hiring a minimum of 200 new technical employees over the next 5 years (FY 18 -FY 22). • These jobs include positions at the National Cyber Range Complex, as well as at Service cyber ranges. • The number of new hires could be as high as 300—or higher—depending on demand for cyber testing and training. UNCLASSIFIED // FOUO 22

UNCLASSIFIED // FOUO Accelerate Professional Development of Technical Staff • Targeted curriculum • Student internships • Jump-start clearance process UNCLASSIFIED // FOUO 23

UNCLASSIFIED // FOUO Discussion • TRMC Overview • Technical Workforce Challenges • TRMC-Directed Research Opportunities for HBCU/MIs UNCLASSIFIED // FOUO

UNCLASSIFIED // FOUO Major S&T Challenges How will we test future weapons? As the S&T Community innovates weapons… The T&E Community must innovate test capabilities. How do we test a learning system with changing behavior? • Autonomy & Robotics • Electronic Warfare How do we replicate a multispectral environment? • Cyber Warfare • Future of Computing How do we test systems maneuvering at hypersonic speeds? • Micro-electronics How do we measure energy on target? • Hypersonics • Directed Energy How do we measure cyber resiliency? • …? We look for your help to innovate our test capabilities UNCLASSIFIED // FOUO 25

UNCLASSIFIED // FOUO University Outreach Ø Focus on Spectrum Efficiency and Network Centric Systems Ø Software Defined Radio (SDR) Project Ø Decision Engine for Structured and Unstructured Data (DESU) Ø Ø Focus on Cyberspace Test Technology Established Cyber Test Lab with existing T&E test tools Established Cyber T&E Fellow Program Ongoing T&E/S&T project: CTAM – Cyber Threat Automation and Monitoring System Ø 6 TRMC summer interns Ø Focus on Unmanned & Autonomous Systems Test Technology Ø Developed Autonomous System Predictive Performance Test Technology Ø Won ASD (R&E) Autonomy Center of Excellence Ø Ongoing T&E/S&T project: DIPT – Data-driven Intelligent Prediction Tool for T&E of Unmanned Autonomous Systems Ø 3 TRMC summer interns UNCLASSIFIED // FOUO 26

UNCLASSIFIED // FOUO University Support to Test Technology Development l Arizona State University l New Mexico State University l University of Delaware l Brigham Young University l North Carolina A&T Univ. l University of Florida (Gainesville) l Carnegie Mellon University l North Carolina State Univ. l University of Hawaii at Manoa l Case Western Reserve Univ. l Ohio State University l University of Illinois l Columbia University l Oklahoma State University l University of Iowa l Cornell University l Oregon State University l University of Kansas l Embry-Riddle Aeronautical Univ. l Pennsylvania State University l Univ. of Maryland (College Park) l Fayetteville State University l Princeton University l University of Massachusetts l Florida International University l Purdue University l Univ. of Minnesota l Florida State University l Rochester Inst. of Technology l University of Missouri l George Washington University l Stanford University l University of Pittsburgh l Georgia Tech Research Institute l Texas Tech University l University of Tennessee l Harvey Mudd College l Univ. of Alabama, Huntsville l University of Texas – El Paso l Johns Hopkins University l University of Arizona l University of Virginia l Lincoln University l University of Arkansas l University of Washington l Mass. Institute of Technology l University of California Irvine l Utah State University l Middle Tennessee State Univ. l Univ. of California Los Angeles l Vanderbilt University l Morgan State University l Univ. of California San Diego l Virginia Tech l Naval Post Graduate School l University of Central Florida l West Virginia University UNCLASSIFIED // FOUO 27

UNCLASSIFIED // FOUO TRMC Points of Contact Mr. Derrick Hinton Acting Director, TRMC Acting Executive Agent (EA) for Cyber Test Ranges george. d. hinton 2. civ@mail. mil 571 -372 -2761 Test Resource Management Center (TRMC) 4800 Mark Center Drive, Suite 07 J 22 Alexandria, VA 22350 -3700 Phone: 571 -372 -2700 http: //www. acq. osd. mil/dte-trmc Mr. G. Derrick Hinton (Acting) Director, TRMC EA, Cyber Test Ranges george. d. hinton 2. civ@mail. mil 571 -372 -2761 Mr. Chip Ferguson Deputy EA, Cyber Test Ranges bernard. b. ferguson. civ@mail. mil 571 -372 -2697 Mr. Paul Mann (Acting) Principal Deputy, TRMC DD, T&E Range Oversight paul. d. mann 2. civ@mail. mil 571 -372 -2773 Mr. AJ Pathmanathan Director, National Cyber Range (NCR) Complex arjuna. pathmanathan. civ@mail. mil 571 -372 -2702 Mr. George Rumford Deputy Director, Major Initiatives & Technical Analyses PM, Test & Evaluation / Science & Technology (T&E/S&T) george. j. rumford. civ@mail. mil 571 -372 -2711 Dr. Robert Tamburello Deputy Director, National Cyber Range (NCR) Complex robert. n. tamburello. civ@mail. mil 571 -372 -2753 Mr. Geoffrey Wilson Deputy PM, T&E/S&T geoffrey. w. wilson 2. civ@mail. mil 571 -372 -2679 Mr. Christopher Paust PM, Central Test & Evaluation Investment Program (CTEIP) christopher. w. paust. civ@mail. mil 571 -372 -2732 Ms. Denise De La Cruz Lead, Army Range Oversight and Workforce/STEM Lead denise. b. delacruz. civ@mail. mil 571 -372 -2733 UNCLASSIFIED // FOUO As of 14 Apr 2017 28

UNCLASSIFIED // FOUO T&E/S&T Program Significant Test Technology Needs (1 of 2) High Speed Systems Test Electronic Warfare Test § Advanced Thrust Measurement Techniques § High-Speed EO/IR Scene Generation § In-Flight Sensing of Atmospheric Conditions § Real-time, High-Fidelity, High-Density RF Signal Generation § Hypersonic Flight Erosion Test Methods § Reconfigurable AESA Threat Simulators § Thermal Protection System Ablation Testing § High Power EW Amplifiers § High Power/Bandwidth 2 -Color IR Emitter § Improved Computational Tools Directed Energy Test C 4 I & Software Intensive Systems Test § Neutron Environment Simulator to replicate required nuclear effects § Automated Testing of Systems and Environments § In Chamber HPM Source § HEL Direct Detection Target Board § Rapid Analysis of Voluminous Unstructured Data (“Test Data to Decision”) § HEL Target Imagery with Active Adaptive Optics § Removing Distributed Testing Environment Effects § Automatically Characterize Required M&S Fidelity in Tests UNCLASSIFIED // FOUO 29

UNCLASSIFIED // FOUO T&E/S&T Program Significant Test Technology Needs (2 of 2) Unmanned & Autonomous Systems Test Cyberspace Test § Predicting autonomous behavior for testing scenarios § Hypervisors for Kinetic Systems and Cyber-Physical Networks § Enhancing safety of autonomous testing § Defining Cyberspace Test Measurements § Testing of autonomy components § Improved, Scalable Cyberspace Test Environments § Measure autonomous system reliability and performance § Adapt Test Environment to Cognitive Autonomous Systems Spectrum Efficiency Technology § Multiple Access Schemes for Serial Streaming Telemetry § Cyberspace T&E Visualization § Advanced cyberspace instrumentation § Improved cyberspace analysis Adv. Instrumentation Sys. Technology § TSPI without GPS (or with limited GPS) § Measuring body armor / helmet effectiveness § Modulation Agile Radio for Test Applications § Measuring cognitive workload § Advanced Spectrum Management Techniques § Steerable Multiband Antenna (L-, S-, & C-band) for Airborne Test Articles § Improved weapon characterization and scoring § Mitigating test range impacts from alternative energy solutions UNCLASSIFIED // FOUO 30