IVV Facility IVV Status for THEMIS June 23

IV&V Facility IV&V Status for THEMIS June 23, 2006 1

Overview • Brief Description of Work and Communication Process • Status for High Severity Technical Issues of Memoranda (TIMs Sev 1 & 2 ) • Detail for High Severity ( Sev 2 ) TIMs • Risks 1001 & 1002 Status • Backup Slides with additional detail IV&V Facility 2

IV&V Tasks for THEMIS IV&V Facility Requirements Analysis System and software requirements are verified to be, for example, complete, consistent, traceable and testable. Software Design Analysis Software design models and algorithms are, for example, checked to provide implementation of associated requirements and for handling off-nominal functionality. Software Code Analysis Code is verified that it is free of implementation errors and that it fulfills the requirements. Tools used, for example, Beyond Compare and Understand. Test Program Analysis Test artifacts are verified to cover all requirements levied on the software. • Weekly meetings with Project are held to work through Technical Issues of Memoranda (TIMs). Other regular meetings attended include Spacecraft, Systems and Mission Ops. • Work is driven by receipt of artifacts. Assessment of artifacts is tracked through TIMs and Risks. • Phase Completion Reports are delivered to Mission Manager, Project Manager, IV&V Liaison • Project Risks are reported to the Project first in written draft through weekly meetings, then formally in Monthly Software Status Report 3

TIMs Status Severity 1 TIMs No open Severity 1 TIMs. IV&V Facility Severity 2 TIMs Status: IV&V action remains to verify tests received: TIM 1357: Incorrectly Deleted Requirement: ACS Req. 3. 2. 1. 8. 11 as a Trace to FSW. AC. 09 Status: The Project has stated they have addressed the testing issue documented in TIM 1357. IV&V is in the process of verifying the artifacts received. Status: IV&V continues to have limited concern on these issues however, the problem of memory management described is unlikely to occur. These will not prevent an IV&V “go” recommendation for launch. Project accepts risk, if any, associated with these TIMs. TIM 1537: Lock flaw in function DS_clear_bulk TIM 1536: Lock flaw in function DS_three_copy TIM 1535: Memory-scrub lock management design flaws 4

THEMIS Risks Risk 1001: Configuration Management – Closed IV&V Facility Risk 1002: BAU Testing - reduced from red to yellow May 2006 If these problems are not mitigated, then BAU software development cost may increase ~$300 K, launch may be delayed by a month and 5 high/medium critical functions may be impacted. • In its current state, the project has a moderate frequency of requirements formulation, requirements tracing, and related test-content issues. • It should be noted that the Project has quickly corrected specific errors identified by IV&V in the FSW executable-test-script errors. The project's progress in addressing these specific errors between the testing of FSW v 2. 504 and FSW v 3. 10 testing has been excellent. IV&V has been unable to determine, to date, via objective evidence that all of the FSW and BSW requirements have been adequately tested. Status: There are 99 TIMs ( all Sev 3 ) associated with this risk. Based on the most recent analysis of artifacts available to IV&V, the status of the TIMs associated with this risk is as follows FSW (67 TIMs) - Closed: 55 TIMs - To Be Verified: 12 TIMs BSW (32 TIMs) - Closed: 3 TIMs - In Dispute: 29 TIMs 5

Summary – Ongoing work with project continues to progress well. Current processes & communications are sufficient to address remaining IV&V concerns. IV&V Facility – While there remains concern on memory management in TIM 1535 -37, IV&V can not verify the problem case will occur. If the problem does occur there remains concern that recovery may be limited. These TIMs will not prohibit a “go” recommendation for launch. – IV&V has been unable to determine, to date, via objective evidence that all of the FSW and BSW requirements have been adequately tested. Further progress is expected to be realized over the next few weeks. 6

IV&V Facility Backup Slides 7

Requirements Analysis in General IV&V applies the following criteria – – – – – IV&V Facility Correct – accurately represents stakeholders’ needs Complete – no unspecified or incomplete requirements Consistent – no conflicts or incompatibilities between requirements. Sponsored – traceable to system or user requirements Testable – defined in terms of explicit, verifiable criteria; atomic requirements Understandable – comprehensible to all stakeholders and self supporting; proper use of standard terminology; definitions provided for all domain-specific terminology Precise – free from ambiguity Design Independent – define what the system is to do, not how to do it; avoids specifying requirements in terms of design elements Organized – grouping of related requirements; partitioning of unrelated requirements Traced – each requirement is traced to an appropriate element of design and/or test procedure 8

Design Analysis in General • IV&V verifies that the design specification – Adequately provides for correct and complete implementation of the associated requirements – Provides enough detail for a developer to correctly implement the design – Correctly accounts for all constraints levied upon the software by the system design • IV&V confirms that the design specification partitions the CSCI into appropriate design entities, defines the key relationships between entities, and provides the following attributes for all design entities – A unique identifier, entity type, and purpose/requirements fulfilled – Function definition and interface definition – Subordinates, dependencies, resources used, and internal data • In addition, IV&V verifies – Function – functionality provided by design entities will satisfy the applicable requirements – Timing relationships – execution frequency meets needs, I/O homogeneity, appropriate tasking priorities, and protected data sharing – Off-nominal functionality – design correctly handles each potential off-nominal scenario – Interactions – interactions between design entities, including proper use of the interfaces and the absence of non-deterministic behaviors and race conditions – Design quality – impacts maintainability and verifiability of the design IV&V Facility 9

Code Analysis in General IV&V Facility • IV&V analyzes source code – Verify a correct implementation of design entities fulfilling all requirements – Verify that code is free of implementation errors • Key code analysis objectives include – Verify that software will produce the intended result for combinations of inputs and conditions • Confirm logic and algorithms are correct • Confirm data types and semantics are consistent and free from errors across interfaces, function calls, computations, and assignments • Identify cases of unintended functions or side effects – Verify that results are reliable/repeatable • Investigate for race conditions and nondeterministic behaviors under a dynamic tasking environment • Confirm data transmission timing, protection of shared resources, and the absence of type mismatches – Confirm error handling is present and correctly detects and handles • Out-of-range, late, or missing inputs • Task overruns and hardware exceptions • Violations of preconditions, such as prerequisite hardware or software states priority 10

Test Analysis in General • IV&V analysis of developer test programs is most often applied to software requirements verification testing (a. k. a. Formal Qualification Testing) – In addition, IV&V often reviews integration or system level testing – Unit level tests are rarely reviewed due to resource constraints – Analysis often covers test plans, descriptions, procedures, scripts, and results • The primary objectives of IV&V test analysis are to ensure – Tests cover all requirements levied on the software – Tests fully exercise all critical functionality required of the software • Key problem areas include – Coverage of requirements, especially in cases where requirements are subject to change – Negative predicate testing (implicit converse scenarios) – Response to fault injection and off nominal conditions – Stress tests and extended duration tests – Negative interactions between components or software states – Regression testing IV&V Facility 11

IV&V Facility Severity definitions 1 a) Prevent the accomplishment of an essential capability b) Jeopardize safety, security, or other requirement designated critical. IV&V Facility 2 a) Adversely affect the accomplishment of an essential capability and no work-around solution is known b) Adversely affect technical, cost or schedule risks to the project or life cycle support of the system, and no work-around solution is known 3 a) Adversely affect the accomplishment of an essential capability but a work-around solution is known b) Adversely affect technical, cost, or schedule risks to the project or life cycle support of the system, but a work-around solution is known 4 a) Result in user/operator inconvenience but does not affect a required operational or mission essential capability b) Result in inconvenience for development or maintenance personnel, but does not affect the accomplishment of these responsibilities 5 Any other affect 12

Detail for TIMs 1535 -37 • Disposition of Sev 2 TIMs 1535 -37 • Since diagnostics are not recorded, in the event of a problem occurring, reactive recovery is limited. Since function exit status is not checked this limits the ability to diagnose or to notify • • 1) TIMs 1535 -37 will remain as Sev 2 TIMs. No workaround exists if error occurs. Proposed workaround is reboot. In any case, reboot does not guarantee problem is fixed and will not recur sometime later. Without diagnostics it is difficult to get to root cause. • 2) IV&V believes these issues as existing in the current code Build 3. 1 have the potential for adversely affecting the essential capability to read/write from/to memory it is unlikely to occur. At this time IV&V is unable to define a test case that presents a concrete failure scenario. TIMs will not be closed however, these Sev 2 TIMs 1535 -37 will not prevent an IV&V recommendation for "go" for launch. IV&V Facility 13

Artifacts Reviewed Below is a list of THEMIS artifacts which IV&V has reviewed or is currently reviewing. BAU Requirements • FSW SRS Revs 1. 1, 2. 0, 2. 2, 3. 0 (in progress) • Flight Software User’s Guide v 1. 0 (in progress) • Boot SRS Revs 1. 0, 2. 1, 2. 2 Design • BAU CDR Presentation (6/15/04) Code • FSW Build 2. 504 • FSW Build 3. 003 • FSW Build 3. 1 (in progress) • Boot Build 0 • Boot 2. 510 Test • • IV&V Facility IDPU Requirements • SRS Revs D, E, F Design • THEMIS IDPU FSW Design document Code • IDPU FSW Phase 1. 04 • IDPU FSW Phase 2. 01 • IDPU FSW Phase 3. 03 • IDPU FSW Phase 4 (in progress) Test • IDPU CPT Plan (in progress) Ground • None BAU FSW Build 2 BAU FSW Build 3. 000/3. 003/3. 10 (in progress) BAU FSW Build 2. 504 CPT BAU FSW Test Plan v 1. 0 System Requirements • MRD Revs C, D, E, F, G, H 14

IV&V Analysis Methods IV&V Facility 15

Tool Descriptions • Poly. Space is a code analysis tool which utilizes Abstract Interpretation to provide exhaustive, automatic checking of the following run-time errors in C, C++, and Ada code. – Attempt to read a non-initialized variable – Access conflicts for unprotected shared data in multi threaded applications (if threading model is configured) – Referencing through null pointers – Out-of-bounds array access – Out-of-bounds pointers – Illegal type conversion (long to short, float to integer) – Invalid arithmetic operations (e. g. division by zero or the square root of a negative number) – Overflow / underflow of arithmetic operations for integers and floating point numbers – Unreachable code IV&V Facility 16

Tool Descriptions • Klokwork Inspect is a static code analysis tool which will identify the following errors in C and C++ code – Array bounds violation – Assignment in condition – Buffer overflows – No-effect statements (i. e. k<j; ) – Invalid memory de-allocations – Non-void functions not returning a value – Void functions returning a value – Return a reference to a local variable – Unused (labels, parameters, variables) – Memory leak – Null pointer – Semi-Colon misplaced for (int i=0; i<100; i++); – Using freeing memory (dereferencing, freeing, passing) – Uninitialized variables – Unreachable code IV&V Facility 17

Tool Descriptions • splint is a C/C++ static source code analyzer that detects coding errors and code constructs that are frequently associated with coding errors. It is particularly useful for detecting mixed-type operations and non-portable constructs • Understand for C/C++ is an interactive C/C++ source code analyzer that generates dynamic cross-reference reports of code components. Understand can detect language errors. The tool uses an extensive data dictionary to underpin its interactive analysis of source code structure and its determination of source components relationships. • Beyond Compare is a utility tool for comparing things like text files, folders, zip archives, FTP sites, etc. Software developers and testers can use it to manage source code, keep folders in sync, compare program output, and validate CD copies. IV&V uses its ability to publish meticulously detailed and dynamic comparison reports between source code components of Build Releases as part of the code implementation analysis phase. IV&V Facility All of the analysis tools, especially if deployed in scripted mode, produce false positives under normal language and filter settings. Manual analysis, guided by mature engineering judgment, is required to determine actual errors and significant issues in the extensive reports generated. 18

BAU Functions IV&V Facility 19

IDPU Functions IV&V Facility 20

Risk 1002: References 1) THEMIS Boot Software Specification (THEMIS-BSW-SRS), Release 2. 2, 1 August 2005 IV&V Facility 2) THEMIS Flight Software Test Plan, THEMIS-FSW-STP, v 1. 0, 3 June 2004 3) THEMIS Flight Software Test Procedures, BAU Build 2 4) B. W. Boehm, et al. Software Cost Estimation with COCOMO II. Prentice-Hall, 2000. 5) THEMIS BAU Software Development Plan 6) THEMIS FSW SRS v 2. 2, August 2005 7) NASA IV&V. Work Instruction for IV&V Services Risk Management. IVV 09 -4 -1, Rev. A. Effective date: 22 August 2005 8) THEMIS Project. THEMIS Continuous Risk Management Plan. Contract Number: NAS 5 -02099. CDRL 05. 19 January 2004. 9) THEMIS Flight Software Requirements Specification v 3. 0. February 2006. 10) THEMIS BSW Test Procedures. v 1. 5. Received May 2006. 11) THEMIS FSW Requirements-to-Test Procedure mapping matrix. Received by the IV&V contractor from the NASA IV&V Facility, May 2006. 12) THEMIS BSW Requirements-to-Test Procedure mapping matrix. Received by the IV&V contractor from the NASA IV&V Facility, May 2006. 13) THEMIS FSW v 3. 10 source code 14) THEMIS FSW Software Users Guide. v 1. 0. Draft. May 2006. 21

Risk 1002: If-Then Statement IV&V Facility In its current state, the project has a moderate frequency of requirements formulation, requirements tracing, and related test-content issues. If these problems are not mitigated, the BAU software development cost increase may be ~$300 K, launch may be delayed by a month (past the nominal launch date of 19 Oct 2006), and ~5 high/medium critical functions may be impacted (Boot, Health & Safety, Command Processing, Data Storage, Telemetry Processing ) The maturity of the test artifacts, documentation, and processes for BAU remains less than would be expected for a project in final testing. This statement is based on Refs. 5, 9, and 11 -14, and on a comparison of the maturity of these artifacts and processes with the norms of the ~400 software development projects constituting the database in Ref. 4. 22

Risk 1002: Likelihood, Consequence & Criticality based on Project’s Criteria Summary This Risk TIM uses the likelihood, impact, and Criticality/Risk definitions of Ref. 8. IV&V Facility A) In Ref. 8, "Likelihood = 3" means "Possible" B) In Ref. 8, "Impact (aka known as Consequence) = 4" can be assessed on several dimensions, including technical, schedule, and cost: Technical (=3): "Baseline Mission" Schedule (= 3 -4): "PCA of 2 weeks" to "Launch Delay of 4 weeks" Cost: (= 4) "Cost increase of $200 K - $500 K" C) In Ref. 8, Likelihood = 3 and Impact (consequence) = 3 (or Impact = 4) implies that the Criticality is Medium (Yellow). This represents significant reduction from the initial Risk Evaluation (Red) for this Risk Item. 23

Risk 1002: Rationale for Likelihood IV&V Facility Likelihood assignment (= 3) • The project continues to defer the resolution of BSW requirements formulation issues (clarity and testability) in Ref. 1 (TIMs 1337, 1400, 1402 -1407, 1410 -1412, 1414 -1415, 1426, 1428, 1431 -1438, 1440 -1441, 14431446). Thus, it is not possible to know whether "final" BSW testing has actually tested the requirements in Ref. 1 that are referenced in these TIMs. This specific feature of this Risk Item has changed very little since the Risk was initially published: of the 32 BSW TIMs in Submitted or To Be Verified state at the orginal publication of this Risk Item, an IV&V review of Refs. 2 and 10 closed only ~10%; the other 29 TIMs in the original list are in In Dispute state. • Since this Risk was originally published, the project processes have reduced to one (from 16) the number Severity 3 TIMs open on Ref. 9 (TIM 1370). • The NASA IV&V Facility provided a mapping from requirements to test procedures for both FSW and BSW (Refs. 11 and 12) in May 2006, thereby establishing a baseline for these mappings. 24

Risk 1002: Rationale for Consequence (Technical) IV&V Facility Consequence/impact assignment (=3): Technical • An Excel spreadsheet that identifies the critical functions (and their CFL scores [Red=high, Yellow=medium, Green=Low]) that are associated with the TIMs identified in the "Description of Issues" section of this Risk item is attached and is accessible under the "Relationships" function of this Risk item. Approximately 5 (down from ~10) high/medium critical BAU FSW functions remain affected. • Since this Risk was originally published, the project processes have reduced to one (from 16) the number Severity 3 TIMs open on Ref. 9 (TIM 1370). 25

Risk 1002: Rationale for Consequence (Cost) IV&V Facility Consequence/impact assignment (= 4): Cost • The project continues to defer the resolution of BSW requirements formulation issues in Ref. 1 (TIMs 1337, 1400, 1402 -1412, 1414 -1415, 1426, 1428, 14311438, 1440 -1441, 1443 -1446, 1370). Based on an analysis of Refs. 1, 10, and 12, IV&V has been unable to resolve these issues. BAU thus continues to have moderate requirements instability in the sense of Ref. 4 predicts that moderate requirements instability will result in a cost increase of ~20% above nominal. It is evident from the historical BAU staff loading that the BAU development cost is at least $1. 5 M, implying a cost increase of ~$300 K over nominal, as defined by Ref. 4. • Since this Risk was originally published, the project processes have reduced to one (from 16) the number Severity 3 TIMs open on Ref. 9 (TIM 1370). 26

Risk 1002: Rationale for Consequence (Schedule) IV&V Facility Consequence/impact assignment (=3 -4): Schedule • The project continues to defer the resolution of BSW requirements definition issues (TIMs 1337, 1400, 1402 -1412, 1414 -1415, 1426, 1428, 1431 -1438, 1440 -1441, 1443 -1446, 1370). Based on an analysis of Refs. 1, 10, and 12, IV&V has been unable to resolve these issues. BAU thus continues to have moderate requirements instability in the sense of Ref. 4 predicts that a moderate requirements instability will result in a schedule increase of ~8% above nominal. The development schedule for BAU FSW is at least 1. 5 years. A 8% increase in project schedule would therefore be ~1 month (past the nominal launch date of 19 Oct 2006), as defined by Ref. 4. • Since this Risk was originally published, the project processes have reduced to one (from 16) the number Severity 3 TIMs open on Ref. 9 (TIM 1370). 27