PreSoftware Assurance Symposium Facility Initiatives Briefing IVV Facility

Pre-Software Assurance Symposium Facility Initiatives Briefing IV&V Facility Independent Verification & Validation of Programmable Logic Devices 8 July 2005 NASA IV&V Facility James Cercone, Ph. D. , P. E. , WVU-Tech Michael Beims, SAIC July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 1

IV&V Facility Pre-Software Assurance Symposium Facility Initiatives Briefing Outline • Review IV&V of PLD Research Project Objectives and Framework • Review of detailed technical findings and VHDL defect taxonomy • Provide overview of Work Instruction development July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 2

IV&V Facility IV&V PLD Status NASA-STD-8739. 8 Software V&V is concerned with ensuring that software being developed or maintained satisfies functional and other requirements and that each phase of the development process yields the right products. …. . IV&V is performed by an organization that is technically, managerially, and financially independent of the development organization. For NASA, IV&V is performed and/or managed by the NASA IV&V Facility. …“Software includes programs and operational data contained in hardware (e. g. firmware, programmable logic, and programmable gate arrays). ” July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 3

IV&V Facility IV&V PLD Status IEEE STD 1012 -1998 IEEE Standard for Software Verification and Validation, provides supporting information regarding the integration of IV&V into every step of the Software Development Life Cycle. The IEEE standard, like the NASA Standard, also cites firmware and microcode in its definition of software: “This standard applies to software being developed, maintained, and reused …. The term Software also includes firmware, microcode, and documentation. ” July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 4

IV&V Facility IV&V PLD Status IEEE Std 1076™-2002 Abstract: VHSIC Hardware Description Language (VHDL) is defined. VHDL is a formal notation intended for use in all phases of the creation of electronic systems. Because it is both machine readable and human readable, it supports the development, verification, synthesis, and testing of hardware designs; the communication of hardware design data; and the maintenance, modification, and procurement of hardware. Its primary audiences are the implementers of tools supporting the language and the advanced users of the language. Keywords: computer languages, electronic systems, hardware design, VHDL July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 5

NESC Project Activities performed by IV&V Facility From Project Plan (SAIC Document #ISTO-05 -98 -192), Section 3 – Activities Completion Date 1. Identify the FPGA design logic faults from: • NASA and industrial sites, • Document Artifacts, and • Comparison of typical FPGA logic design methods with proven software engineering methodologies, including those used for design and peer review Year 1, 2 Q 2. Identify existing software engineering methodologies that can be directly applied to FPGA designs by tracing common defects to their underlying cause Year 1, 3 Q 3. Suggest enhancements to developers’ design and peer review methodologies Year 2, 1 Q 4. Provide field prototyped training materials for performing PL software V&V Year 2, 4 Q 5. Successfully complete a pilot project Year 2, 4 Q Primary Goals: • Develop an IV&V strategy for PLD’s • Provide field proven PLD Work Instruction (WI) to the IV&V practitioner July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 6

IV&V Facility Activities/Results thus far (9 -1 -05 through 8 -8 -05) Activities Thus Far • Better understanding of PLD’s at WVU Tech and SAIC – Primarily via literature searches and attendance at workshops – Presentation to IV&V CAWG – WVU Tech obtained and learned IDE’s (Integrated development Environment) for both Actel and Xilinx PLD’s. WVU Tech has also obtained and learned Active HDL • • • Limited analysis and simulation of NASA project data IV&V has mapped PLD’s into a better framework for IV&V WI development Identifying a taxonomy of defects in VHDL Domain – Via Literature Search – By comparing VHDL releases for the same chip – Evaluating SW code defects that can be “mapped” to PLD VHDL defects Had initial discussions with JWST as candidate for VHDL Pilot Project July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 7

IV&V Facility Activities/Results thus far (9 -1 -05 through 8 -8 -05) Results Thus Far • Scoped PLD IV&V Framework to understand 05 accomplishments and identify potential future year activity – Based on increased understanding, and – Realization that existing IV&V Code Analysis WI is insufficient for PLD analysis • Defect taxonomy, in process of refinement, for presentation at MAPLD July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 8

IV&V Facility Development Environments Idealized Software/PLD Development Requirements • SRS is a CM’d document • Rigorous flowdown is common Design • SDD is a CM’d document • Manual or tool generated Common PLD Development Requirements Design/code and simulate at functional level Code • Different types of code (C, C++, etc) • Mature tools avail to aid in development, verification Test • Performed on implemented code • Unit, Subsystem, System testing follows req’ts flowdown • Rigorous process Design/code and simulate After chip layout • Part of subsystem • Hardware artifact (e. g. EQ spec, product functional spec) • It is at this stage that Idealized/Common development Testing after PLD is programmed • Performed on implemented PLD • Unit, Subsystem, System testing follows req’ts flowdown processes diverge • Design process • IDE • Target • PLD’s also have timing concerns that are rare in software development, such as • Synthesized versus native components • Race Conditions • Adequately buffering data July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 9

Current Year (2005) Activities against PLD IV&V Framework IV&V Facility Common PLD Development Verification Tasks Validation Tasks Requirements Similar to FSW but artifact expectations need to be articulated by IV&V tbd Design/code and simulate at functional level 1) Ensure syntax is correct 2) Identify typical errors 3) Develop/deploy WI: Programming Standards and Defect ID • VHDL • Verilog • Schematics Traceability of Requirements Identify key timing areas and independently simulate Design/code and simulate After chip layout Testing after PLD is programmed Identify any known issues with IDE and ensure potential errors not present in developed product Verify tests performed by developer (using simulation to generate test cases) Re-simulate key timing functions tbd, independent testing? Our current WI activity focuses on verification of VHDL Design • Develop Work Instruction • Flesh out Work Instruction with Pilot Project • Deploy Work Instruction at Facility July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 10

Ideas for Future Year Projects IV&V Facility Common PLD Development Verification Tasks b Validation Tasks Requirements Similar to FSW but artifact expectations need to be articulated by IV&V tbd Design/code and simulate at functional level 1) Ensure syntax is correct 2) Identify typical errors 3) Develop/deploy WI: Programming Standards and Defect ID • VHDL • Verilog a • Schematics Traceability of Requirements Identify key timing areas and independently simulate Design/code and simulate After chip layout Testing after PLD is programmed Identify any known issues with IDE and ensure potential errors not present in developed product Verify tests performed by developer (using simulation to generate test cases) b c Re-simulate key timing functions tbd, independent testing? d Future Projects • Develop WI for verification of Verilog or Schematic designs (item “a”) • Provide WI for Requirements Analysis or Test Analysis of PLD’s (item “b”) • • • This is probably a straightforward extrapolation of current WIs for FSW Have the breadth of knowledge on development tools and all target PLD’s (item “c”) Provide insights on timing/validation aspects of PLD implementation (item “d”) July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 11

IV&V Facility Complexity is a Challenge for all Design Representations ? Functional Trace / Performance Test July 8, 2005 Design Trace / Functional Test IV&V of Programmable Logic Devices – Beims, Cercone 12

IV&V Facility Pre-Software Assurance Symposium Facility Initiatives Briefing Review of detailed technical findings and VHDL defect taxonomy July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 13

Sample Findings IV&V Facility Potential VHDL“Hot Spots” visible in semantics Entity Are Signals defined in the port list as out type signals given values? Are Signals that are defined in the port list as inout type signals used for both – reading and writing? Are Signals defined in the port list as in type signals used in the architecture? July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 14

Sample Findings IV&V Facility Potential VHDL“Hot Spots” visible in semantics Process Is there a series of sequential statements followed by a branching structure? Is there a branching structure followed by a series of sequential statements? Is each process sensitive list made up of the signals from the Entity’s port list? July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 15

Sample Findings IV&V Facility Potential VHDL“Hot Spots” visible in semantics If Structures Having elsif and no else statement Having neither an elsif or else statement Is there unreachable code inside an else statement? When using a compound if statement, are all possible conditions covered in subsequent elsif and else statements. How deep is the nesting of if structure? Testing Signals in the condition that are not part of the process’s sensitive list July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 16

Sample Findings IV&V Facility Potential VHDL“Hot Spots” visible in semantics Signal Assignment Is the same set of Signals assigned values in each of the if-else sections? Is the same set Signals assigned values in each of the case structures when and when others => clauses? Are all Signals in a component’s port list mapped values during a Component’s instantiation? July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 17

IV&V Facility July 8, 2005 Sample Findings Static Metrics Analysis of Public Code IV&V of Programmable Logic Devices – Beims, Cercone 18

Sample Findings IV&V Facility Static Metrics Analysis of Public Code Note ! July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 19

Sample Findings IV&V Facility Static Metrics Analysis of Public Code … July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 20

Sample Findings IV&V Facility Static Metrics Analysis of Public Code Observations related to Code Changes (absolutes for the public code analyzed) A code change occurred if: • There were 12 or less files • Average number of lines per file was greater than 400 • There were less than 3 “package” statements • There were less than 11 “entity” statements • There were less than 20 “component” statements • There were less than 13 “architecture” statements • There were less than 20 “clk’event” statements • There were any “while” statements • There were any “wait” statements • There were any “after” statements July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 21

Sample Findings IV&V Facility Static Metrics Analysis of Public Code Observations related to Code Changes (normalized for the public code analyzed) A code change occurred if: • 5% or more of the total lines were “signal” statements • 5% or less of the total lines were “in” statements • 5% or less of the total lines were “out” statements • 3½% or more of the total lines were “if” statements • ¼% or more of the total lines were “case” statements July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 22

Sample Findings IV&V Facility July 8, 2005 Static Metrics Analysis of Public Code IV&V of Programmable Logic Devices – Beims, Cercone 23

Sample Findings IV&V Facility July 8, 2005 Static Metrics Analysis of Public Code IV&V of Programmable Logic Devices – Beims, Cercone 24

Sample Findings IV&V Facility July 8, 2005 Static Metrics Analysis of Public Code IV&V of Programmable Logic Devices – Beims, Cercone 25

Sample Findings IV&V Facility July 8, 2005 Sample taxonomy of semantic defects visible in VHDL IV&V of Programmable Logic Devices – Beims, Cercone 26

Sample Findings IV&V Facility July 8, 2005 Sample taxonomy of semantic defects visible in VHDL IV&V of Programmable Logic Devices – Beims, Cercone 27

Sample Findings IV&V Facility Sample taxonomy of semantic defects visible in VHDL Note: NASA experts’ recommended practice prevents an ‘out of sync’ by insuring that resets are never very close to a clock edge. This design is seen in NASA flight software as a D-FF with reset. July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 28

Sample Findings IV&V Facility Sample taxonomy of semantic defects visible in VHDL Synthesis vs Simulation difference visible in VHDL Multiplexer with missing sensitivity signal (signal “b”) process(a, sel) if sel = '1' then out <=1; else out <= b; end if; end process www. synplicity. com/literature/pdf/HDLDesign. Methods. pdf July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 29

Fault Detection Matrix High Confidence – IV&V success IV&V Facility Actual False True False/Positive True/Positive Potential Hot Spot identified/ No defect Potential Hot Spot identified/ Design defect exists False/Negative True/Negative No Hot Spot identified/ Defect exists No Hot Spot identified/ No Defect Tested Positive Negative Less Confidence Mission Risk July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 30

IV&V Facility Fault Detection Matrix Examples True/Positive Potential Hot Spot identified/ Design defect exists “Pre-processor directives and usage are often not controlled by coding standards. 1) #ifdef statements can be left active or inactive and permit non-flight code (e. g. test code) to be compiled. Instances of such errors have been found in Mars program code. 2) #define statements can be left in the code from testing leaving test values or conditions active in the flight code. Instances of such errors have been found in Mars program code. ” July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 31

Fault Detection Matrix Examples IV&V Facility False/Negative No Hot Spot identified/ Defect exists process (A, B) begin if (cond 1) X <= A + B; elseif (cond 2) X <= X – B; end if; end process; July 8, 2005 If neither cond 1 nor cond 2 is true, then X will retain its value. . . basically, X is stored in a latch In general, latches are not usually recommended in synchronous designs IV&V of Programmable Logic Devices – Beims, Cercone 32

IV&V Facility July 8, 2005 Taxonomy of Common Visible Defects IV&V of Programmable Logic Devices – Beims, Cercone 33

VHDL Code Severity Chart IV&V Facility Severity July 8, 2005 Description A FPGA code will not perform desired tasks. Mission is jeopardized. B Serious hindrance to mission accomplishment. A serious cost effect to project. I. E. (No “quick fix”) C Adversely affects FPGA code performance. A minimal cost effect to project but a “quick fix” is possible. D Annoying effect to user but FPGA code is operable. E Any other effect. IV&V of Programmable Logic Devices – Beims, Cercone 34

IV&V Facility Pre-Software Assurance Symposium Facility Initiatives Briefing Overview of Work Instruction Development July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 35

Work Instruction Development IV&V Facility Background Considerations VHDL specific considerations Taxonomy of potential “Hot Spots” Clock and Reset Lines Sensitivity Lists Features not consistently supported between IDE’s Non-implemented features (i. e some attributes) July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 36

VHDL Code Severity Chart IV&V Facility July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 37

VHDL Code Severity Chart Examples IV&V Facility Example of High Functional Criticality The FPGA with this defect is the only functional capability for the Satellite to deploy the solar panels. If the FPGA does not perform this function, the satellite will run out of power, causing loss of the mission. July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 38

VHDL Code Severity Chart Examples IV&V Facility Example of Low Functional Criticality The FPGA controls one side of a dual redundant path to the telemetry transponder. If the FPGA fails, then the telemetry is routed via CPU control, causing a momentary delay in telemetry, if all telemetry is buffered through on-board storage. (Note: since this is a design (software) defect, if there were two identical FPGA's controlling this functionality, instead of an FPGA and the CPU, then the redundant FPGA can be expected to fail in the same manner and there is no functional redundancy, making this a High Functional Criticality defect. ) July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 39

Compliance Issues IV&V Facility Example of Vendor Specific Degree of Compliance • “major differences between XVHDL and Express is IEEE VHDL-93 compliance. XVHDL is a fully IEEE VHDL-93 compliant tool. Express supports many of the most commonly used VHDL-93 synthesis constructs, but is not yet fully compliant; it remains officially compliant with the IEEE VHDL-87 standard. ” • http: //www. xilinx. com/xlnx/xil_ans_display. jsp? i. Language. ID=1&i. Country. ID=1&get. Page. Path=5144 (7/21/2005) July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 40

Compliance Issues IV&V Facility Example of Vendor Specific Compliance (two examples) • Signal Declaration – Supported ("register" or "bus" type signals are not supported) • Attribute – Only supported for some predefined attributes: HIGH, LOW, LEFT, RIGHT, RANGE, REVERSE_RANGE, LENGTH, POS, ASCENDING, EVENT, LAST_VALUE – Otherwise, ignored. • http: //www. xilinx. com (7/21/2005) July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 41

Defect Taxonomy (VHDL Verification at Functional Design), and Pilot Deployment IV&V Facility • Draft of VHDL programming standards geared toward defect identification – Defects commonly detected by compilers are not included – Includes syntax, timing margins, clock boundaries • This draft is in process of update, peer review – Align defects with known coding defects – Test draft product against actual VHDL text • Developer places multiple revs of VHDL on website • The results will be presented at MAPLD in September, 05 Work Instruction in 05 addressed VHDL verification. Future tasks needed to address: Verilog and Schematic verification plus validation. Note: GLAST LAT used Actel VHDL for design and this served as basis for IR&D project. MRO project used Xilinx Verilog for design. July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 42

Preliminary Considerations for Defect Detection in VHDL Based Designs IV&V Facility • Materials needed to start the verification process are: • Design Documentation to analyze performance • Actual VHDL Code • Code Pedigree » (Reused modules, designers, level of experience…) • Development and Analysis Tools State diagrams. • Clock Trees • NASA and IEEE Standards July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 43

Preliminary Considerations for Defect Detection in VHDL Based Designs IV&V Facility • V&V Process and Procedure at the Code Level – – – Static Metric Analysis Code Coverage (particularly for behavioral level designs) Verification of Clock and Reset Tree’s (if provided) Check for compliance to NASA Standards Check for device resource usage • (synthesized vs. board components such as MAC’s, SR, and DFF’s) – Check of IDE specific restrictions – Check VHDL specific “Hot Spots” July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 44

IV&V Facility Pre-Software Assurance Symposium Facility Initiatives Briefing Conclusion • Review IV&V of PLD Research Project Objectives and Framework • Review of detailed technical findings and VHDL defect taxonomy • Provide overview of Work Instruction development July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 45

IV&V Facility Background Slides 46

What is IV&V ? IV&V Facility Independent • Technical: IV&V prioritizes its own efforts • Managerial: Independent reporting route to NASA Headquarters • Financial: Budget is allocated by NASA to the IV&V Facility such that IV&V effectiveness is not compromised Verification (Are we building the product right? ) • The process of determining whether or not the products of a given phase of the software development lifecycle fulfill the requirements established during the previous phase • The product is internally complete, consistent and correct will support the next phase Validation (Are we building the right product? ) • The process of evaluating software throughout its development process to ensure compliance with software requirements. This process ensures: – Expected behavior when subjected to anticipated events – No unexpected behavior when subjected to unanticipated events – System performs to the customer's expectations under all operational conditions July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 47

Maximizing Project V&V and IV&V Facility • The Project V&V goes end-to-end – Needs sufficient depth to help ensure that they have build the product right and built the right product • IV&V needs to be the second line of defense – Select the most critical functionality and then IV&V to the appropriate depth -- not exceeding the IV&V point of diminishing returns (maintaining reasonability) – The cut-off point should be where we have found the critical defects and also gained enough confidence in the software to support mission assurance requirements and launch recommendations • Project Teams should compare our criticality rankings to their knowledge of the development as an independent source and explore differences • Project Teams should look at activity just below the IV&V line to ensure adequate V&V resources July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 48

IV&V Facility • PLD Updated Framework to perform IV&V was updated to – Take into consideration that development of PLD’s is different than development of software, – Address verification and validation tasks explicitly • PLD’s are developed in an environment that combines design, code and simulation simultaneously – Timing aspects much more critical in PLD’s • Major differences needed to be addressed: – PLD development is part of subsystem development – comparable artifacts not consistently generated – PLD design can occur in many forms • Schematic (representation similar to chip/board design) • VHDL (representation similar to Ada) • Verilog (representation similar to C/C++) – Target system matters • Syntax different even when same language used – Development environment matters • From a syntax standpoint • From a capability standpoint (e. g. software motivated or hardware motivated) July 8, 2005 • Which. IV&V of Programmable Logic Devices Beims, Cercone. IDE’s) revision of IDE (multiple releases for– hw motivated 49

Updated IV&V Framework for PLD Development IV&V Facility Common PLD Development Verification Tasks Validation Tasks • • Requirements Similar to FSW but artifact expectations need to be articulated by IV&V tbd Design/code and simulate at functional level 1) Ensure syntax is correct 2) Identify typical errors 3) Develop/deploy WI: Programming Standards and Defect ID • VHDL • Verilog • Schematics Traceability of Requirements Identify key timing areas and independently simulate Design/code and simulate After chip layout Testing after PLD is programmed Identify any known issues with IDE and ensure potential errors not present in developed product Verify tests performed by developer (using simulation to generate test cases) Re-simulate key timing functions tbd, independent testing? The above updated IV&V framework has more detail – Allows us to clearly understand what we have accomplished, and what lies ahead – Strategy is to perform accomplished tasks well – For each task performed, 1. Develop Work Instruction (WI), flesh out internally 2. Test on pilot project 3. Deploy updated WI The fast pace of PLD product evolution requires additional considerations – Important to have cognizance of market trends – Update WI appropriately with trends that will be implemented in spacecraft in the near term July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 50

Simulink Example IV&V Facility http: //www. transtech-dsp. com/software/simulink. asp July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 51

Small Microprocessor Example IV&V Facility July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 52

Examples of Rare Software Defects IV&V Facility • PLD’s also have timing concerns that are rare in software development, such as: • Synthesized versus native components – Seen in Stinger real time seeker simulations. Debate over whether a native compiler matrix multiplication routine was sufficiently predictable versus a matrix multiply built up in separate Fortran 77 instructions. • Race Conditions – Seen in the “Response to a Setting Satellite Vehicle” scenario in the Space Shuttle GPS’ firmware. • Adequately buffering data – Seen in the Space Shuttle Primary Avionics Software Systems’ Mid Frequency Executive where every variable must be analyzed for time homogeneity and treated accordingly. • In General, “PLD–like” defects are seen in hard, real time software systems, which in turn are the primary candidates for migration to PLD’s in the near future (recent past? ) July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone 53