NASA IVV Software Assurance Tools SWAT S W

NASA IV&V Software Assurance Tools (SWAT) S W A T Data-Driven IV&V Decision Support

• What is Decision Support? • Data Availability (On-Demand, Linked, Warehouse) • Data Presentation • Analysis, Reporting and Metrics • Related IV&V Efforts • Questions S W A T Outline 2

S W A T What is Decision Support? • For our purposes, Decision Support will be defined as helping people make decisions that are supported by relevant data • A decision is defined as the choice of one among a number of alternatives • Decision making refers to the overall process of making a choice, which includes assessing available data and identifying alternatives 3

S W A T Data Availability • Data must first be made available for decision support Data Availability On-Demand Data Linked Data Warehouse 4

S W A T On-Demand Data • IV&V can support daily decision making with basic on-demand data access • Example #1 – Project PM may ask “Based on the issues we have identified so far, where should we focus IV&V resources? ” Ø ORBIT (Integrity) Queries configured to show current project IV&V Issues by category and severity • Example #2 – Project PM may ask “Which of my team members should I allocate to an upcoming milestone review? ” Ø JIRA Dashboard view configured to show current IV&V project tasking status by assignee 5

• Example #3 – IVVO Lead may ask “Which project risks currently require greater attention? ” S W A T On-Demand Data ØRisk. Manager 5 x 5 matrix view of risks filtered for the IV&V Office level 6

S W A T Linked Data • Linked Data for our purposes is defined as structured data from multiple sources that has been related in some way which can make other useful knowledge available 7

S W A T Linked Data • Each data entity is given an individual identity for access, e. g. a URL can be used if accessing through the web • The advantage comes from leveraging the relationships between data, not just the individual content itself • Linked data concepts vs. separate “silos” of data in spreadsheets for example provides the following benefits: Ø Data remains in an appropriate context, as it is referenced through a common interface (e. g. web service or centralized data store ) Ø Data is linked (related), enabling information to be combined across the organization or even from outside the organization Ø Data can still be accessible from other applications that depend on live data, while not preventing static data dumps where applicable (e. g. export to operational data store) 8

• Vocabulary S W A T Linked Data ØDefining a formal and shared vocabulary is necessary to document relationships (ontology) ØPermits “inferences” to be made when analyzing data • Search ØSome value can be gained from just traversing linked data ØGreater value is provided by some search (query) capability of linked data 9

IVVO Lead: “What types of IV&V issues or risks have we been identifying for missions with similar characteristics? ” S W A T Linked Data linked or related by project (or mission) SQL Server Report generated based on data that is extracted from multiple data sources (e. g. IV&V issue tracking system, Risk. Manager tool, IV&V Projects/Mission database) into a centralized operational data store 10

IV&V Project PM: “Which of my team resources may be overloaded with active issue follow up and additional tasking? ” S W A T Linked Data linked or related by analyst Dashboard display of team assignments for active IV&V issues (ORBIT) and other tasking (JIRA) 11

S W A T Data Warehouse • Historical analysis and trending must be based on data that is not just a single snapshot in time, rather data this is accumulated over a time period • Robust and strategic IV&V decision making can be supported through a data warehouse approach that adds a time (historical) dimension to the data 12

S W A T Data Warehouse 13

TQ&E Lead: “Is there any correlation between IV&V issue generation rates and the maturity of methods employed by IV&V teams? ” Senior Management: “Do projects with senior level analysts generate more issues/higher severity issues? ” S W A T Data Warehouse Data is analyzed over a period of time Business Intelligence (BI) tool configured to aggregate the necessary facts for analysis of any relationship 14

S W A T Data Presentation • Data must also be presented for decision support • Data can be presented in any number of ways for analysis: Ø Dashboard views of data in simple table and chart format, including custom interfaces that access other data sources to populate these views Ø SQL Server Reports delivered through a web interface Ø Excel can be populated from outside data to leverage additional analysis capability Ø Visualized data can be presented through Business Intelligence (BI) dashboards to help expose data patterns, trends and correlations 15

S W A T Analysis, Reporting and Metrics • Online Analytical Processing (OLAP) – multidimensional data and aggregations for analysis • Ad hoc reporting provided through such mechanisms as SQL Server Reporting Services (SSRS), with reports provided in web interface and e-mail subscriptions • Metrics (e. g. IV&V method utilization over time by projects) • Export to more lightweight options like Excel for specific analysis 16

S W A T Related IV&V Efforts • IVVO Data Management – related FY 14 initiatives identified • Analysis Management Framework (AMF) domain model for IV&V analysis execution • Development of dashboards for SWAT developed tools that pull from various data sources • Risk. Manager and COMPASS tool web services • Atlassian gadgets for JIRA and Confluence 17

Contact us at: ivv-swat@lists. nasa. gov S W A T Questions 18