Design Integrity Concepts 2005 MAPLD 1 Design Integrity

Design Integrity Concepts 2005 MAPLD 1 Design Integrity Concepts

Unit Agenda • Consistent terminology, consistent results – Introduction and definitions • What does it have to do? – Specifying the design • Letting constraints work for you – Proportional design • More than the sum of its parts – Partitioning a design • You mean we’re still working on it? – Sustaining a design • What’s the exit strategy? – Verifying a design • What do you mean, it doesn’t do what we thought? – Validating a design 2005 MAPLD 2 Design Integrity Concepts

Who is this guy? • John Stone – Chief Engineer (Department of Space Systems – Sw. RI®) – 18 years experience (Exclusively space-flight) • • • C&DH design (hardware and software) Instrument electronics design Box-level integration and test Hardware project management Product assurance (parts, radiation, reliability) Process improvement – Broad interest in improving what we do 2005 MAPLD 3 Design Integrity Concepts

What do we want to accomplish? • Discuss techniques that may help us to do our jobs better – Based on general principles – Broad applicability (logic, board, box, system) • Foster necessary discussion / brainstorming – To extent possible (as time allows) – No one person has all of the answers – Suggestions appreciated 2005 MAPLD 4 Design Integrity Concepts

Consistent Terminology, Consistent Results Introduction and Definitions 2005 MAPLD 5 Design Integrity Concepts

Agenda – Introductions and Definitions • Design integrity – A working definition • Why is this important? – A tale of two designs – Has anything changed? – Why should I care? • The miracle cloud • The alternative – Overview – Implications – Additional Definitions • Summary 2005 MAPLD 6 Design Integrity Concepts

Definition and Goal • Design – The invention and disposition of the forms, parts, or details of something according to a plan. (AH dictionary online) • Integrity – The state of being unimpaired; Soundness; The quality or condition of being whole or undivided; completeness (AH) • This seminar is intended to talk about techniques and issues that ensure the soundness and completeness of both the end product and the means used to produce it. 2005 MAPLD 7 Design Integrity Concepts

Design 1 – Radarsat 1 ACP 2005 MAPLD 8 Design Integrity Concepts

Radarsat 1 ACP Overview • Program dates: late 1990 – late 1992 • Specifications – Processor: 8 MHz 80386/80387 – Memory: 128 k x 8 SRAM, 192 kx 8 EEPROM, 16 k x 8 PROM – Interfaces: A/D (16), D/A (4 -12), Synchronous serial (3 input, 3 output), RS-232 GSE – Function: Attitude control processor for the RADARSAT 1 satellite – Logic Implementation MSI logic + PALs (16 L 8, 16 R 6, 16 R 8) • Additional functions (cross-strap, control) external 2005 MAPLD 9 Design Integrity Concepts

PAL Reminder 2005 MAPLD 10 Design Integrity Concepts

Design 2 - Command Telemetry Board (CTB) 2005 MAPLD 11 Design Integrity Concepts

CTB Overview • Program Dates: early 2001 – late 2003 • Specifications: – Processor: RTX 2010 (16 MHz) – Memory: 4 M x 8 (random), various FIFOs (16 k x 8) as necessary – Interfaces • • • MIL-STD-1553 B Synchronous Serial (command / telemetry) Analog input High-level discrete (output) Low-level discrete (input / output) – Functionality: S/C command/telemetry (level 0 and active) – Logic Implementation: 4 54 SX 32 S FPGAs • Additional resources (in same box): RAD 750, Mass Memory, Instrument Interface Card 2005 MAPLD 12 Design Integrity Concepts

What’s Changed? • Capability / Complexity • Logic Density • Specificity – RADARSAT (1 small specification with interface focus) – CTB (1 large specification with interface, s/w, operations focus) • Software Centricity • Initial Errors – RADARSAT: 3 jumpers; 1 PAL design change – CTB: 14 FPGA revisions • 2 spec change • 5 -6 mistakes • 6 -7 data dependency 2005 MAPLD 13 Design Integrity Concepts

What’s Not Changed? • • Overall program schedules Proportional budget Expectation of correctness Pain from mistakes 2005 MAPLD 14 Design Integrity Concepts

What Explains the (error) Difference? • Engineers aren’t as capable? – Insulting! • Everything is just more complex? – Copout! • Methodology? – Methodology hasn’t changed • Always inadequate, we just got lucky • Adequate for old designs, no longer adequate – Methodology has changed • Used to be adequate, but we lost the recipe • Design philosophy of systems has changed? – Predicated on maximum flexibility – Expectation of extreme complexity – Over-specification – almost impossible to meet 2005 MAPLD 15 Design Integrity Concepts

What Do These Examples Illustrate? • The incidence of initial correctness for designs seems to be decreasing – Design changes seem to be more common – Problems late in the verification/validation cycle seem to be more frequent • Perhaps a combination of the factors presented explains this, but … – Desired complexity is not going to decrease – Budgets are not going to get bigger – The expectation of excellence isn’t going to go away • The only solution is to develop and improve a consistent methodology for implementing robustly designed products – Based on basic principles – Applicable to a variety of conditions 2005 MAPLD 16 Design Integrity Concepts

Why Should I Care? • Why do we work? – Self-actualization (fun, monetary reward, interest) • Why do people want us to work for them? – They need what we produce • What do people want engineers (especially in Aerospace) to produce? – A quality product that satisfies the customer’s needs • How do they want us to produce such a product? – Consistently and efficiently 2005 MAPLD 17 Design Integrity Concepts

The Layman’s View – The Miracle Cloud 2005 MAPLD 18 Design Integrity Concepts

The Miracle Cloud Method • Note that too many engineering schools teach this approach without meaning to • Advantages to the miracle cloud method – Total creative freedom • Disadvantages to the miracle cloud method – Product quality is variable • Team makeup dependent • Team mood/morale dependent (Monday morning car) • Luck dependent – Product is not produced in a repeatable manner – Product is not produced in an efficient manner • Result – Quality low – Customer Satisfaction Low 2005 MAPLD 19 Design Integrity Concepts

How Do We Replace the Miracle Cloud? • • Provide structure to the development effort Evaluate the effort and the product produced Improve the effort and the product Repeat 2005 MAPLD 20 Design Integrity Concepts

Definitions of Importance • From Q 9000 -2000 • Process – A set of interrelated and interacting activities which transforms inputs to outputs [in our case ideas to devices] • Product – The result of a process 2005 MAPLD 21 Design Integrity Concepts

Implications From These Definitions • If we want a consistent product, we must have a consistent process • If we want to improve a product, we must improve the process • If our company has no (or inadequate) process and we must produce a quality product, then we must establish a process [personal responsibility] • Developing, imposing, and improving a process is not an end (in and of itself) it is only a means to an end 2005 MAPLD 22 Design Integrity Concepts

A Model for Discussing the Design Process 2005 MAPLD 23 Design Integrity Concepts

Notes on the Model • Feedback / iteration are not shown for clarity • Model may be recursive – Board development process includes FPGA requirement definition, FPGA development, instantiation, etc. – Board verification process includes the FPGA validation product • Successes and failures are cumulative – Good requirements + successful development => successful instantiation – Bad requirements + failed development => failed instantiation • Complexity multiplies – Complex requirements increase design complexity which, in turn, increases verification complexity • Processes are absolute gates to the next stage of development 2005 MAPLD 24 Design Integrity Concepts

Implications From the Model • All processes must be addressed at all levels of design [there are no shortcuts!] – Does not imply same formality at all levels – Does imply same rigor at all levels • Up front work on requirements is essential! – Must provide adequate time and money – Must gain team buy-in to the process* – Benefits compound throughout the rest of the activities • Simplicity is an essential virtue! – Complexity inevitably multiplies • A product is not qualified until both verification and validation are complete 2005 MAPLD 25 Design Integrity Concepts

Additional Useful Definitions (courtesy of Q 9000 -2000) • Specification – A document* stating requirements, needs, or expectations that are obligatory • Quality – The degree to which a set of inherent characteristics fulfill requirements • Customer satisfaction – Customer’s perception of the degree to which the customer’s requirements have been fulfilled • Verification – Confirmation, through the provision of objective evidence, that specified requirements have been fulfilled • Validation – Confirmation, through the provision of objective evidence, that the requirements for a specific intended use or application have been fulfilled • Objective evidence – Data supporting the existence or verity of something • Continual Improvement – recurring activity to increase the ability to fulfill requirements • Note the importance of Requirements 2005 MAPLD 26 Design Integrity Concepts

Summary • I have no assurance that my product will have consistent quality without: – Well-defined requirements – A well planned approach to implementing the requirements – A clearly defined plan for verification and validation of the requirements – The ability to improve the process that produces the product • Without quality product, customer satisfaction is impossible • Without customer satisfaction, I won’t work! • Therefore, I must care about ensuring design integrity 2005 MAPLD 27 Design Integrity Concepts

What Does It Have to Do? Specifying the design 2005 MAPLD 28 Design Integrity Concepts

Agenda – Specifying the Design • • • Basic concepts and implications What should we include in a specification? What should we avoid in a specification? The role of the design engineer Summary FPGA specifications 2005 MAPLD 29 Design Integrity Concepts

Basic Specification Concepts • #1 A specification has a limited set of purposes: – It is intended: • To capture product requirements that are essential to meeting a need – functionally and interfacially • To ensure traceability of the requirements from higher levels • To provide a fixed framework for verifying product conformance • To provide a framework for derived requirements – It is not intended: • • To impose a detailed implementation for a design To provide unnecessary theory of operation statements To become a user’s manual As a box to check on the schedule checklist 2005 MAPLD 30 Design Integrity Concepts

Basic Specification Concepts (cont. ) • #2 Specification language must meet certain criteria – Structure • Similar requirements must be grouped together • All requirements must stand on their own – Verbs must have a consistent specific meaning • Shall, must -> impose imperatives • Should, might -> define preferences • Will -> defines objective information – Phraseology • Phrases must be unambiguous • Phrases must define one (and only one) requirement • Phrases must be short and understandable – All must agree on the meaning of what is written 2005 MAPLD 31 Design Integrity Concepts

Basic Specification Concepts (cont. ) • #3 Specifications are inherently intrusive – Requirements exclude options in a design • The number and specificity of requirements determine the level of exclusion – Specifications impose non-negotiable verification requirements • Every requirement must be verified • The narrower the requirement, the more specific the verification • The more requirements, the more work needs to be performed – A well designed specification assists the design and verification process – A poorly designed specification • Unnecessarily shuts down trade space • Increases verification time and effort • Makes for an unhappy engineer 2005 MAPLD 32 Design Integrity Concepts

What is Included in a Specification? • Interfaces – Number – Type – Timing / Handshaking • Interrelationships – Temporary storage – Data Flow – Software Support (careful) • System impact – Quality and Reliability – Parts, materials, and processes 2005 MAPLD 33 Design Integrity Concepts

Examples of Things to Include • Interfaces – The unit shall provide 16 low-level discrete interfaces – The unit shall configure 8 low-level discrete interfaces as pulsed interface – Pulse interfaces shall generate pulses between 2 and 20 ms long • Interrelationships – The unit shall provide on-board temporary storage sufficient for 2 packets from the XYZ interface – The unit shall forward data from the XYZ to QRS interfaces on a packet by packet basis – The unit shall provide an interrupt to the S/W upon receipt of a complete package from the XYZ interface • System Integration – The unit shall have a. 75 probability of success as calculated per the parts count method of MIL-HDBK-217 – The Unit shall use only EEE parts that meet the requirements of EEEINST-002 2005 MAPLD 34 Design Integrity Concepts

What Should Not Be Included in the Specification? • Implementation details (unless essential to safety or reliability) – The unit shall implement standard pyrotechnic fire circuit #2 as found in …(marginally acceptable) – The unit shall use 4 7206 FIFOs made by IDT corporation – 370 ns after the last bit of a packet is received the unit shall raise bit 3 of interrupt control register 21 • Requirements based on inherent capability of devices used – The unit shall provide a 14 -bit A/D converter for interface X (when based upon the part being used) – Note: Decisions on precision or accuracy etc. should be made for interface characteristics (physical measurement range and accuracy) 2005 MAPLD 35 Design Integrity Concepts

What Shouldn’t Be Included in the Specification (cont)? • Ambiguous, unclear, or interpretive requirements – The unit shall be constructed using techniques found in generally accepted space hardware guidelines – The unit shall interface to other components per figure 2 (picture based requirements) – The unit shall provide the best performance possible • Overly complex requirements – The unit reset shall be immediately triggered upon expiration of the watchdog timer unless the watchdog timer has timed-out five times (8 times if in mode 2) in which case the circuitry shall look at discrete bit 2 to determine whether to delay 3 s (discrete bit 2 = 1) before timing out • Extraneous information – The unit shall interface with the visible CCD used in spectrographic mode – The unit shall replace the functionality provided by p/n 276401 -01 2005 MAPLD 36 Design Integrity Concepts

Why Shouldn’t These Be Included? • This type of requirement increases cost [time, money, emotional] to develop a product – Increase design effort (less trade space, inefficient design, overly complex design) – Increase verification effort [planning, implementing, configuring] due to design complexity – Increase probability of re-design or re-build [validation uncertainty] • This type of requirement reduces the overall quality of the product developed – Implemented design not per intention [interpreted requirements] – Implemented design is not 100% verifiable to specification [unclear requirements] – Implemented design requires significant number of waivers and deviations 2005 MAPLD 37 Design Integrity Concepts

The Role of the Design Engineer • The design engineering team must “buy in” to the specification development process – They have to deal with the consequences if it is done poorly – Lack of “buy in” produces a non-controlled process • “Buy in” requires early review and participation by the design team • Therefore, design engineers have a significant role to play in the specification development process • What is this role? 2005 MAPLD 38 Design Integrity Concepts

Role of the Design Engineer (cont. ) • Know how to write a specification – Allows constructive review of the imposed specification – Improves qualities of derived requirements • Try to understand the “why” behind the various requirements – Improves efficiency of design trade studies – Allows intelligent suggestion of alternatives • Suggest alternatives when necessary – Expose more efficient ways of producing equivalent functions – Support trade offs between software and hardware functionality 2005 MAPLD 39 Design Integrity Concepts

Role of the Design Engineer (cont. ) • Think forward – Take the lead in defining derived requirements – Ask: • What implications does this have for the design? • What implications does this have for testability? • Will this let me sleep at night? • Push back – Seek clarification of ambiguous requirements – Inform others of impractical or costly driving requirements – Ask for relief from impossible requirements • Remain engaged in the process – Be thorough in review – Don’t be passive with suggestions 2005 MAPLD 40 Design Integrity Concepts

Summary • Specifications are critically important to the design engineer and must not be ignored • Design teams must be intimately involved in the specification development process • Detailed design and implementation will succeed or fail largely based on successful application of the specification process 2005 MAPLD 41 Design Integrity Concepts

FPGA Specifications - Rationale • FPGAs are a major part of modern day spaceflight designs – Primary control functionality – Equivalent of multiple boards of traditional circuitry – Major schedule driver • FPGAs are impossible to verify completely from external signals – Too many buried modes and functions – Too much dependence on simulation • Correcting FPGAs is conceptually simple – Temptation to sloppiness • “First time right” is an essential virtue – The FPGA specification is a means to achieve “first time right” 2005 MAPLD 42 Design Integrity Concepts

FPGA Specification Prototype • Interface Section – Specific pinout requirements – I/O type definition (Names, Direction, Logic levels) – Imposed Software requirements (addressing, etc. ) • Do not include non-imposed addressing / register mapping / software interfaces • Functionality Section – – Interface interaction requirements Data flow requirements Exclusion requirements Do not include • Implementation details • Theory of operations • Usage instructions 2005 MAPLD 43 Design Integrity Concepts

FPGA Specification Prototype (cont. ) • Fine timing section – All timing imposed by board peripheral circuits – Include • Min and max delay between I/Os • Set-up and hold for controlled peripherals • Fine timing requirements should be an input to the FPGA development effort, not outputs of the concluded design 2005 MAPLD 44 Design Integrity Concepts

Why Include Fine Timing? • Reduces influence of changes external to FPGA (encapsulation) – Board implementation can change significantly – FPGA implementation can change significantly – Changes ok as long as interfaces remain stable - but fine timing must be a controlled item • Simplifies verification – Verification between implemented FPGA performance and fixed requirements defined at the board level can be easily accomplished – Reduces reliance on complex back-annotated simulations at the board level for FPGA specific verification – Increases reliability of static timing analysis • Note – this only works if a structured design approach is used such that valid requirements can be imposed on the FPGA early in the process. 2005 MAPLD 45 Design Integrity Concepts

Stand-Alone or Integral? • When should an FPGA specification stand alone? – When the FPGA design engineer is not the board design engineer – When there are multiple interrelated FPGAs on a board – When the FPGA design will be used in multiple board applications – When the FPGA will become an ASIC – When the development schedule between the board and FPGA are disjoint – When the complexity of the FPGA is greater than that of the board support circuitry 2005 MAPLD 46 Design Integrity Concepts

Stand Alone or Integral (cont. )? • When should an FPGA be specified as part of the board specification? – When the FPGA is so intertwined with the peripheral circuitry that writing a stand-alone specification is not practical – When the FPGA functionality is easily expressed by simple gate level schematics – When the FPGA and board are designed by the same person – When heritage design with adequate specifications exist 2005 MAPLD 47 Design Integrity Concepts

Letting Constraints Work For You Proportional Design 2005 MAPLD 48 Design Integrity Concepts

Agenda – Proportional Design • • • Conceptual background Types of constraints Examples The proportional design mindset Summary 2005 MAPLD 49 Design Integrity Concepts

Conceptual Background • Three parts to solving a problem: – Need, solution set, constraints • All parts have a role to play in the solution • Ignoring any of them will lead to problems 2005 MAPLD 50 Design Integrity Concepts

Conceptual Background (cont. ) • Example – Need: means of conveyance to work – Solution set: Skateboard, bicycle, bus, jogging shoes, midsize sedan, luxury car, helicopter – Constraints: Distance (6 miles), $, not on bus route, $, not in very good shape, $ – Solution: 1992 Honda Accord (120 kmiles, 4 k$) – The constraints guide selection of the solution from the solution set • The particular solution is not necessarily – The cheapest (roller skates) – The most desired (Lexus LS 400) – What is perceived as best for society (bus) • But … the best overall fit to the needs 2005 MAPLD 51 Design Integrity Concepts

Conceptual Background (cont. ) • Definitions – Constraint: the state of being checked, restricted, or compelled to avoid or perform some action (AH) – Proportional: corresponding in some degree or intensity (AH) • Proportional design is design that results in a product “sized” appropriately to the needs and restrictions of the specification • The concept of proportional design: – Accepts the reality of constraints – Attempts to optimize the solution given the constraints – Accepts that the constraints provide benefits (more later) • • More efficient designs More thorough designs More correct designs Caveat – All other things being equal 2005 MAPLD 52 Design Integrity Concepts

Types of Constraints • External (mass, power, cost, quality) • Internal – Derived (packaging, architecture, component availability, maximum clock speed) – Self-imposed • Design rules/guidelines (free space, clock use, logic structure, HDL language) • Documentation style (pre-design, post design) • Component acceptability (maturity of part, limited use of various features 2005 MAPLD 53 Design Integrity Concepts

Examples (1) • Problem: provide decoding logic for memory map – 0 -3 FFF = SRAM; 4000 -4 FFF = Peripheral; E 000 -FFFF = PROM • Constraint: use minimum amount of logic • But what about … – Unused addresses, future expansion, etc. • Doesn’t matter – given the constraints 2005 MAPLD 54 Design Integrity Concepts

Examples (2) • Problem: provide all combinational / sequential logic for the RADARSAT ACP • Constraint: Only low density high speed logic available (16 X 8 PALs, MSI/SSI logic) • What was forced by the constraint? – Careful mapping of peripherals into available address space – Careful partitioning between: • Programmable logic and MSI/SSI • MSI/SSI functionality – Efficient data bus partitioning (tri-state enable issues) – Special attention to component delays at the gate level 2005 MAPLD 55 Design Integrity Concepts

The Proportional Design Mindset • Constraints inevitably foster attention to detail (creativity “inside the box”) – With respect to methodology – With respect to level of planning – With respect to implementation • Attention to detail is of inherent value because it produces carefully structured, well-thought out designs – Improved up-front correctness – Decreased design post-processing time (simulation, verification, validation, lab time) – Efficient designs that meet the stated requirements – Increased reliability • Therefore, constraints are welcomed, whether externally imposed or self-imposed 2005 MAPLD 56 Design Integrity Concepts

The Proportional Design Mindset (cont. ) • Examples of self-imposed constraint – Ignoring achievable flexibility (when not necessary) – Removing non-specified capability – Avoiding gratuitous cleverness (especially with abstract design techniques) – Rejecting brute force solutions without analysis 2005 MAPLD 57 Design Integrity Concepts

The Proportional Design Mindset (cont. ) • Characteristics of the right mind set – – – Planning before starting Reviewing before finalizing Simplifying ruthlessly Making the design do only what it must Viewing resources as precious commodities to be used only to the extent needed – Understanding the implication of the design’s level of abstraction – Being satisfied with the result 2005 MAPLD 58 Design Integrity Concepts

The Proportional Design Mindset (cont. ) • Why aren’t self-imposed constraints more common? – They aren’t absolutely essential because we have: • Lots of logic space [FPGAs, ASICs] • Lots of memory space [DOS file systems, complicated operating systems] • Lots of bandwidth [fast data busses, general purpose communications protocols] – They don’t match the current paradigm • • Flexibility is all-important [re-use, re-configure, adapt] Specifications are malleable late in the game Software changes, why can’t hardware? We can catch problems in simulation and reprogram the part – They aren’t fun – We don’t train people to value constraints and work within them • This is unfortunate because constraints can make our job easier without degrading the end product 2005 MAPLD 59 Design Integrity Concepts

Summary • The proportional design mindset is important because it: – Focuses on fulfilling needs, not wants [specification orientation] – Deepens understanding of the final design [ownership oriented] – Avoids unnecessary effort [efficiency oriented] – Fosters simplicity that aids verification and validation [quality oriented] 2005 MAPLD 60 Design Integrity Concepts

More Than the Sum of Its Parts Design Partitioning 2005 MAPLD 61 Design Integrity Concepts

Agenda – Design Partitioning • Definitions and examples • Why partition an electronic design? • Guidelines for partitioning an electronic design • Why isn’t it done more often? • Summary 2005 MAPLD 62 Design Integrity Concepts

Definitions and Examples • Partition – to divide into parts or shares – Examples: budgeting, outlining, WBS development • Design partitioning refers to the deconstruction of a design into various sub-designs in an ordered and logical manner • Goal – to simplify the whole by optimizing the parts and thus increase: – Efficiency – Reliability – Maintainability 2005 MAPLD 63 Design Integrity Concepts

Why Partition (General)? • Complexity interferes with ready comprehension – Comprehension of a complex system depends on ability to impose order upon it – But … a given mind has a finite capability to impose order which depends on the quantity and structure of the data related to the system – The more complex the system, the more difficult its comprehension • Partitioning a design introduces a piece-wise reduction in complexity – Reduces quantity and complexity of design to manageable “chunks” – Improves comprehension of the various parts of the design – Increased comprehension of the parts leads to better comprehension of the whole • Better comprehension of the whole increases the ability to – Verify correctness of the design – Correct errors in the design – Update the design when necessary 2005 MAPLD 64 Design Integrity Concepts

Why Partition (cont. )? • Programmatic advantages – Refines scope of work • Identifies unexpected effort • Identifies reuse possibilities • Identifies staffing requirements – Identifies schedule dependencies • Improves allocation of resources • Identifies parallelization and schedule enhancement opportunities – Promotes management visibility 2005 MAPLD 65 Design Integrity Concepts

Why Partition? - Design • Improved interface organization / more formal structure – Interfaces between functions have more predictable characteristics – Expansion by addition of functions is more controlled • Enhanced functional encapsulation – Individual functions have predictable results when invoked – Functional design enhancements have limited side-effects when installed – Effects of faults are more easily predicted and mitigated • Efficient design implementation – Functional (fewer functions and types of functions) – Data flow (fewer, more sensible data busses) – Control flow (simpler address decoding and state machines) • All are side effects of additional thought put into “How? ” 2005 MAPLD 66 Design Integrity Concepts

Why Partition? - Correctness • Simpler inspection – Functionality may be “obvious at a glance” – Error space is more limited • Within the function or within the interconnect • Simpler Qualification – Verification can begin with encapsulated modules or circuit subsets – Overall functionality correctness becomes less of a “late” concern because subset functionality is proven correct “early” in the process • Simpler / more thorough review – Structure provides orientation to peer reviewers – Encapsulation allows easier review – Peer input more likely to be useful 2005 MAPLD 67 Design Integrity Concepts

Why Partition? – Maintainability • Clearer documentation – Documentation has “smaller” parts to focus on – Structure of documentation grows from the design structure • Simpler maintenance – Changes affect only enhanced area – Interactions between changed area of design and remainder of original design is controlled by a formal structure • Enhanced reuse – Sub-circuits / functions usable in other applications as long as the interface structure is observed • Inherent capability of design better understood 2005 MAPLD 68 Design Integrity Concepts

Guidelines for Partitioning • Take advantage of organizational strengths – Expertise (analog, digital, software, etc. ) is seldom the same across organizations – Partition the design in accordance with organizational strengths according to primary functions – Divide auxiliary functions (those that can be assigned to multiple organizations) so that • Interfaces are simplified • Workload is equalized • Functions are easily tested without requiring all of the hardware 2005 MAPLD 69 Design Integrity Concepts

Guidelines for Partitioning (cont. ) • Example: Alice UV spectrograph electronics (Rosetta) • Core expertise – Sensor Sciences: Detector and front end electronics – Mobius Systems: Embedded software – Sw. RI® space systems: Digital, low speed data acquisition – Sw. RI® space science: LV and HV power supplies • Primary partition per core expertise 2005 MAPLD 70 Design Integrity Concepts

Guidelines for Partitioning (cont. ) • Alice example (work breakdown chosen) – Sensor Sciences: detector electronics through parallel digital output – simplified interface – Mobius: instrument control / protocol servicing (some error decoding partitioned to H/W) – Space systems: microcontroller; s/c interface; heater, motor, door digital control; analog high-voltage control – Space sciences: LVPS; HVPS; motor, door, and heater drive circuitry 2005 MAPLD 71 Design Integrity Concepts

Guidelines for Partitioning (cont. ) • Alice example – partitioning decisions – Auxiliary expertise split along sensible interfaces • C&DH to detector – analog or digital? (noise, ease of test) - digital • C&DH to HVPS – analog or digital? (space, noise susceptibility, ease of test) - analog • C&DH to S/W – protocol decoding? (performance margin, logic capability) – hardware error decoding, s/w protocol encoding 2005 MAPLD 72 Design Integrity Concepts

Guidelines for Partitioning (cont. ) • Use specific functionalities – Combine functionality with very similar characteristics • Low-level analog / discrete bi-level (comparator is difference) • Example: Spacelab flex interfaces – Cluster related functionalities • Shared data-flow direction, shared logical control • Examples – Constant level discretes / pulsed discretes – Low-level discretes / high-level discretes 2005 MAPLD 73 Design Integrity Concepts

Guidelines for Partitioning (cont. ) • Use specific functionalities (cont. ) – Isolate related functional sub-groups from other items • Ex. – analog data acquisition group (multiplexers, converters, signal processing, control logic) • Isolate – Through appropriate data/address buffering – Through separate programmable logic sub-sets – Exploit directionality • Write functions; read/write functions – appropriate buffering • Examples – Write: Analog output, digital output, telemetry – Read: Analog input, digital input, command – R/W: Memory, bi-directional discretes, GSE 2005 MAPLD 74 Design Integrity Concepts

Guidelines for Partitioning (cont. ) • Exploit operational considerations – Operational considerations often determine the specific configuration of a set of common components – Example: memory system • Components: memory, write control, read control, sequencing, buffering • Application: telemetry system – – Packetized / unpacketized? Asynchronous timing / TDM ? Science data / engineering data? Pushed / pulled ? • The type of telemetry system determines the partitioning 2005 MAPLD 75 Design Integrity Concepts

Example – Science Data Storage and Readback System • How should the logic be partitioned? – Is write logic part of science data process or memory process? – Is read logic part of telemetry system or memory process? – How complicated is the arbitration? • How it is partitioned depends on specific operational requirements 2005 MAPLD 76 Design Integrity Concepts

Example - New Horizons Alice Science Data • Alice Operation – – • Slow source accumulation relative to output speed Push interface initiated by instrument Alice Implementation (others likely in different circumstances) – – Block 1: handshake and latch data Block 2: Ingest data, process, write to memory Block 3: Read, serialize, send, blank memory Arbitration simplified to a switched double buffered memory access (no real-time arbitration) 2005 MAPLD 77 Design Integrity Concepts

Guidelines for Partitioning (cont. ) • Ensure encapsulation is reflected in the form of the engineering documents – Functions contain many types of operations – Example: Telecommand interface • De-serialization, decoding, error determination, re-packetization, temporary storage • Real time functionality [level 0], forwarding to software • Storage access / arbitration, status log maintenance, data-bus handshaking – Partitioning should ensure that all reasonable aspects of a function are in one locality (we are ordering data for understanding) • One (or a few) pages in a schematic • One module in HDL • One object in software – Benefits: readability, error determination, testability, maintainability 2005 MAPLD 78 Design Integrity Concepts

Ordering Example – Bus Structure • A logical bus structure – – – – Simplifies data flow Eases expansion / enhancement Identifies bottlenecks / opportunities for efficiency Ensures signal compatibility Reduces timing uncertainty (capacitive loading) Reduces power Simplifies control logic / arbitration Simplifies analysis 2005 MAPLD 79 Design Integrity Concepts

Ordering Ex. – Bus Structure (cont. ) Before: After: 2005 MAPLD 80 Design Integrity Concepts

Ordering Ex. – Bus Structure (cont. ) • Directional data flow is clustered • High-speed access devices (SRAM) not buffered • Exclusive functions clustered (PROM/ EEPROM) • Simplification of – Timing – Loading – Control 2005 MAPLD 81 Design Integrity Concepts

Why is Partitioning Not a Priority? • It is – at the S/C, sub-system, and box level (sometimes) • Why not at the board level? – Requires potentially significant planning effort (schedule / cost) – The tool syndrome (CAD / CAE) • Crush creativity by forcing an early start to design • Primarily a way to communicate between software packages rather than humans (schematic => PWB) – Too much junk being placed in too little space (simplification may not always be space efficient) – Lack of emphasis from senior engineers – Boards aren’t where the action is (FPGAs) – less effort placed on them 2005 MAPLD 82 Design Integrity Concepts

Why is Partitioning Not a Priority? (cont. ) • At the FPGA level – Lack of solid design methodology • Methodology must be tailored to a tool • VHDL / Verilog are functional descriptions • VHDL / Verilog don’t inherently enable data flow visualization or block oriented deconstruction • The synthesizer can understand non-partitioned design – Perception of expansive resources (no / few constraints) – The tool syndrome (“I must start coding”) – Lack of effective design quality gates prior to start of detailed design 2005 MAPLD 83 Design Integrity Concepts

Summary • A design is only as solid as its weakest part • Proper planning and partitioning of a design: – – – Ensures individual functions are logical and complete Ensures interconnects are ordered and efficient Provides for improved reliability / verifiability Allows easier modification and enhancement Enhances detailed understanding of how things work • Partitioning requires ordering the design by considering – – The capabilities of the team The functionality of the modular pieces How the design will operate The individual components of a particular function • Partitioning a design ensures that all parts are solid – resulting in a solid whole 2005 MAPLD 84 Design Integrity Concepts

You Mean We’re Still Working On It? Sustaining a Design 2005 MAPLD 85 Design Integrity Concepts

Agenda – Design Sustainability • Definitions – Sustainable – Capable of being supported (AH) – Sustainability – The characteristic of an item that allows it to be supported • Why is this important? • Suggestions for sustainability • Summary 2005 MAPLD 86 Design Integrity Concepts

Why Sustainability? • Missions may be multiple decades long – Flight systems may develop anomalous behavior – Ground equivalents may need repair – An understanding of the design is necessary to ensure mission success – Original designers will not be available for debugging • Other critical assignments • Working in telecon • Cruising the Pacific – Sustainable designs allow analysis and correction without the access to the original designers 2005 MAPLD 87 Design Integrity Concepts

Why Sustainability? (cont. ) • Many designs are derivative – Reuse of unmodified circuits essential for similar performance in modified designs – Acceptable modification depends on creative incorporation of what IS • Derivation may take many years – Example – Alice UVS • • Design 1 – Rosetta (1997 -2001) Design 2 – New Horizons (2002 -2005) Design 3 – LRO (2005 -2008) Design 4 – Juno (2006 - ? ? ? ) – Staffing will not be constant – Human memory will not be precise • Sustainability ensures an ability to efficiently build on past successes 2005 MAPLD 88 Design Integrity Concepts

Why Sustainability (cont. ) • You may not be the person who has to make it work – Staffing is dynamic • You may quit • You may get re-assigned • Somebody with more clout may be needed to satisfy the customer – Teams produce a product and share debugging • Test technician • S/W designer • I&T team – Self-interest and common courtesy • You don’t want 18 questions per day • Ethics – Do unto others – Example (ICB)* 2005 MAPLD 89 Design Integrity Concepts

Suggestions for Sustainability • Remember the dual nature of design input – The CAD perspective • Schematic => PCB layout package => circuit board • HDL => Synthesizer => Fuse file => Programmed FPGA – The human perspective • Schematic – Interrelationships (time, space, connection) – Debugging tool – Functionality description • HDL – Describes functions and interaction – Renders constraints understandable – Ensure Readability! 2005 MAPLD 90 Design Integrity Concepts

Suggestions for Sustainability (cont. ) • Record the design process – Keep a notebook (type not vital) – Describe everything of importance • Why? – Is this bus used for this function? – Is this function implemented like this? – Etc. • How? – – – Do these things talk to one another? Does this sequential logic work (state diagrams)? Is the address map decoded? Are errors handled? Etc. 2005 MAPLD 91 Design Integrity Concepts

Suggestions for Sustainability (cont. ) • Record the Design Process (cont. ) – Describe (cont. ) • What? – – – Signals are needed to perform this function? Do the waveforms look like? Timing do I expect to observe? Changes have I made? Etc. • When? – Record chronology – Provide a way to reproduce what was done – Make part of permanent project record – Example – Radarsat 1 Notebook 2005 MAPLD 92 Design Integrity Concepts

Suggestions for Sustainability (cont. ) • Schematics – Provide an overview of the design • Schematic table of contents • Block diagram (hierarchical design if available) – Provide consistent naming scheme • Descriptive of signal direction / function / polarity • Consistent across logic gates and within various blocks – – Cluster sub-circuits on contiguous pages Make connections between components explicit Add comments where necessary for clarification Remove unused circuitry (for FPGA schematics) 2005 MAPLD 93 Design Integrity Concepts

Suggestions for Sustainability (cont. ) • Don’t: 2005 MAPLD 94 Design Integrity Concepts

Suggestions for Sustainability (cont. ) • Do: 2005 MAPLD 95 Design Integrity Concepts

Suggestions for Sustainability (cont. ) • Help! 2005 MAPLD 96 Design Integrity Concepts

Suggestions for Sustainability (cont. ) • HDL (Must be done from beginning!) – Provide overall orientation to design – Provide top-level comments on • Level of use (top, intermediate, etc. ) • Overall purpose of function / block / module • Signal function and origination (external and internal) – Provide operational comments on • • State machine purpose and configuration (how? why? ) Transition logic (theory and reasoning) Function of particular sequences State to control signal translation – Clarify obscure references • Remove superseded code (don’t comment out) and explain uncommon structures – Improve readability • Create logical file names • Minimize file, logic block, function sizes • Include related functions together (error generation, data interface, basic function, etc. 2005 MAPLD 97 Design Integrity Concepts

Suggestions for Sustainability (cont. ) Comments? 2005 MAPLD 98 Design Integrity Concepts

Suggestions for Sustainability (cont. ) Header from same design! (After the fact documentation) 2005 MAPLD 99 Design Integrity Concepts

Suggestions for Sustainability (cont. ) • Post process documentation – Theory of Operation / Users Manual • Generate One • Include – – Design concept / features Operational Constraints Appropriate Uses Complete Engineering Documentation • Update, release and correct as necessary – Create design archive • Self-consistent and complete • Place under revision control • Control changes 2005 MAPLD 100 Design Integrity Concepts

Summary • Useful designs will be corrected, modified and evaluated – FOR A LONG TIME! – By people besides you • Sustainability measures must be implemented to make this happen efficiently • Sustainability requires – Adequate conceptual documentation and records – Clear and readable implementation records – Finalized and controlled configuration records • Ensuring sustainability will preserve your legacy 2005 MAPLD 101 Design Integrity Concepts

What’s the Exit Strategy? Verifying a Design 2005 MAPLD 102 Design Integrity Concepts

Agenda – Design Verification • • Concepts and implications The specification connection Verification before test Test thoroughness Tiered verification Concluding the process Summary 2005 MAPLD 103 Design Integrity Concepts

Verification Concepts • Verification – Confirmation, through the provision of objective evidence, that the specified requirements have been fulfilled (Q 9000 -2000) • Confirmation – the act of ratifying or corroborating (AH) 2005 MAPLD 104 Design Integrity Concepts

Verification Implications • The verification process is not intended to be a “craps shoot” – Verification is primarily intended to highlight correctness – Verification is NOT primarily intended to reveal incorrectness • The mindset is important! – Doubt that a design will work expresses a lack of confidence in the designer and the design process – Waiting until “verification” to guarantee correctness is an excuse for being sloppy • We should expect our designs to work! Verification simply puts the seal of confirmation on the expectation 2005 MAPLD 105 Design Integrity Concepts

Verification Implications (cont. ) • Verification addresses two processes – Design • Primarily a one-time, iterative process in which the developed concept is proven sound • Fundamental correctness can be proven analytically – Inspection confirms logical correctness in all circumstances » Peer reviews are a manual form of inspection » Functional simulation is an automated form of inspection » Inspections must be tailored to design – Analysis verifies correctness in the presence of variability » Reliability (WC, Derating, FMEA, etc. ) » Timing over environments and age 2005 MAPLD 106 Design Integrity Concepts

Verification Implications (cont. ) • Verification addresses two processes (cont. ) – Instantiation • Particular instance must be sound • Particular correctness must be demonstrated experimentally – Inspection confirms that instructions are followed » Correct components installed » Correct configuration selected » Correct processes performed – Test and demonstration confirms that operation matches expectations » Functionally and parametrically » Predicted by nominal analytical predictions 2005 MAPLD 107 Design Integrity Concepts

Verification Implications (cont. ) • Verification “after the fact” is too late – Analysis and test are NOT equivalent – Design analysis and inspection must proceed in lockstep with design processes – Implementation tests and inspections should simply confirm what is known • Design efforts fail because they occur in a vacuum – Creativity takes primacy over correctness – Functionality comes first with operability being “shoehorned” after the fact • The “monster” simulation syndrome • Conclusion: Verify as you proceed 2005 MAPLD 108 Design Integrity Concepts

Verification Implications (cont. ) • Correctness is a matter of personal integrity for the designer – Verification is not simply externally imposed task – Verification is something that the designer must want to do (a part of his/her ethos) – Verification is confirmation that the designer is “worth his/her salt” 2005 MAPLD 109 Design Integrity Concepts

The Specification Connection • “Requirements” are confirmed during verification – The designer does not have a free hand with respect to how a design is confirmed – Specification provides the constraint set under which verification is prosecuted – Therefore, verification must be defined prior to start of design • Not simply in a general manner • Specifically – Since specification is a customer and vendor document • Both customer and vendor must be involved in the verification process • “Trust me” is not a reasonable phrase when it comes to verification 2005 MAPLD 110 Design Integrity Concepts

The Specification Connection (cont. ) • Specification contains a complete [ideally] set of requirements for the design – Some requirements are very specific • “All discrete outputs shall have a source impedance of 2 k. Ohms. • An adequate test is fairly obvious – Some requirements are not very specific • “The unit shall provide 512 kbytes of SRAM” • Note the implied requirement – The SRAM shall function correctly • What is an adequate functional test for this requirement? (walking 1’s, 0’s addressing, etc? ) – Some requirements lend themselves to quantitative analysis – “The unit shall provide a. 1° C accuracy at end of life” – Some requirements lend themselves to simple inspection – “The unit shall provide 4 analog output channels” – When do we decide adequacy of the method of verification and the individual verification cases? • – Before we start designing 2005 MAPLD 111 Design Integrity Concepts

The Specification Connection (cont. ) • Therefore, verification planning must begin with specification creation – Each requirement must be assigned a method or methods – Each requirement must be assigned a test or analysis case • Must answer question – What is an adequate verification? • Must establish answer to question – When are we done? – Each requirement must be verified at one or more levels [FPGA, board, box, sub-system, system] – Customer must concur with decision • Note that modern verification databases make this relatively straightforward 2005 MAPLD 112 Design Integrity Concepts

The Specification Connection (cont. ) • Advantages to this type of verification planning – More verifiable / verified designs • Up-front focus on programmatically difficult verification can be instituted – Earlier analysis – Simplification of overly complex circuit designs • Guidance for lower-level verification efforts can be established – Sub-module vs. module level – Module vs. unit level • Integral self-test provisions can be included in design • Earlier simulation vector definition (FPGAs / logic) 2005 MAPLD 113 Design Integrity Concepts

The Specification Connection (cont. ) • Advantages (cont. ) – More robust test sets • Early knowledge regarding end-item function communicated • Capability of test set matched to need – Precision (timing, voltage, current, etc. ) – Speed – Test level (component, unit, system) • Test flow design considerations included in test set design – Knowing verification requirements early makes the entire development process more efficient 2005 MAPLD 114 Design Integrity Concepts

The Specification Connection (cont. ) • Defining requirements and test cases at the same time as the specification clarifies and simplifies the FPGA verification process – Allows early start to simulation vector design – Creates early knowledge of additional functional models (SRAM, peripherals, etc. ) needed – Clarifies decision regarding what can be functionally simulated and what must be simulated with backannotation – Defines levels at which simulations must be run 2005 MAPLD 115 Design Integrity Concepts

Verification Before Test • It is a relatively bad thing to discover a design flaw during test (better than nothing) – Late in the development cycle – Correction is more complicated / expensive – Personal stress is higher • Yet … a surprising lack of verification occurs early (before test) • Types of pre-test verification – Inspection - conformity evaluation by observation and judgment accompanied, as appropriate by measurement or testing (ISO) • Personal self-check • Peer Review • Basic functional verification (Signal audits, functional simulation, etc. ) 2005 MAPLD 116 Design Integrity Concepts

Verification Before Test (cont. ) • Types of pre-test verification (cont. ) – Analysis – Conformity evaluation of the predictions produced by realistic models of the system • Worst case analysis (voltage, current, frequency, time) using models that incorporate real world degradation of function • Derating analysis using models that reduce stress • Other • Note that pre-test verification is fundamentally conceptual – Not tool dependent – Not design dependent – Can be accomplished many ways 2005 MAPLD 117 Design Integrity Concepts

Verification Before Test (cont. ) • Two approaches to pre-test verification – Verify full design at once (one big peer review, code walkthrough, analysis, simulation) • Benefits – – Complete design reduces need to develop assumptions When proved correct, does not need to be repeated When design is solid, less work Reduces final documentation for verification • Drawbacks – Allows small design flaws to propagate for a long time – Complexity reduces visibility (verification more prone to mistakes and oversights) – When design is flawed, it increases work – Leads to corrections that are global (hard to test effectively and predict side effects) – May take longer to execute than an aggregate of smaller verification activities 2005 MAPLD 118 Design Integrity Concepts

Verification Before Test (cont. ) • Two approaches to pre-test verification (cont. ) – Verify incremental designs • Benefits – – Supports development of “known good” sub-designs Meshes with a partitioned design approach Facilitates visibility (fewer mistakes, clearer goals) Allows small flaws to be fixed quickly (minimizes downstream impact) • Drawbacks – Is more work in the ideal case (no/few flaws) – Increases documentation if formality is vital • Either approach can be made to work, but one or the other must be chosen and implemented 2005 MAPLD 119 Design Integrity Concepts

Test Thoroughness • Principles for test thoroughness – Every “shall” in the specification that meets the following criteria must be tested • Items that may be affected by the instantiation process (subject to fabrication error) • Items for which the only extant verification is model based • Items for which the testing does not pose unacceptable risk (radiation, overstress, etc. ) – Every instance of a “shall” must be tested • 18 interfaces require 18 specific tests – Requirements that have an exclusive character • Must be tested for correct operation • Should be tested at some level for abnormal operation • Example: If you write a 5 Ah to something to set it on, you should verify that writing other values does not set it on – Requirements must be tested over a representative range of conditions 2005 MAPLD 120 Design Integrity Concepts

Test Thoroughness (cont. ) • A thorough test is complicated and time consuming – Requires some level of automation to be comprehensive – Requires some level of compromise to be practical – Requires team agreement to be acceptable 2005 MAPLD 121 Design Integrity Concepts

Test Thoroughness (cont. ) • Ensuring thorough, yet practical, tests – Define which tests require manual interaction and which can be automated • Output impedance or analog fine precision may need to be manual • Signal level or analog coarse precision may be automatable – Define which tests must be performed over environmental conditions and which can be performed at room temperature only • Make decision based on character of measurement • Do not decide based purely on practicality – Define level of complexity for all measurement sets • Example: 24 analog inputs (3 eight channel muxes) • Test full range of values for 24, or limited range for 7 of 8 mux inputs and full range for remaining – Define need for repetition of measurement at higher level of integration (tiered verification) • Example: voltage level / signal quality / clock jitter at board level • Repeat at box level? 2005 MAPLD 122 Design Integrity Concepts

Test Thoroughness (cont. ) • Test thoroughness issues affect everything – Customer relationship – Test set design (board, box, system) – Schedule / cost • An adequate test program is not trivial – Cannot wait until system level – Must incorporate lowest level of design – Must be a constant background activity during design process • All programs must balance perfect and good enough • A good test program – – Will ensure that the specification is met Will verify everything that can be affected by fabrication Will build on the analysis and inspection effort Will maximize automation without sacrificing test accuracy 2005 MAPLD 123 Design Integrity Concepts

Tiered Verification • Tiered verification is the process of matching the correct type of verification to the appropriate level of integration • Tiered verification incorporates all types of verification (before and during testing) • Tiered verification applied to the test regime attempts to have thoroughness with practicality – Test a requirement at the lowest conceivable level – Determine which tests must be repeated at higher levels by • Establishing the purpose of a test at a particular level • Determining whether the next level has the potential to change previous measured quantities • Determining what test set capabilities are • Tiered verification cannot be retrofitted into the process – Must be planned up front – Must be implemented throughout the development process 2005 MAPLD 124 Design Integrity Concepts

Tiered Verification - Example • Need: A set of 16 high-level pulsed discretes to control latching relays used to change the spacecraft power configuration • Basic Requirements – – – Quantity: Level: Load: Duration of pulse: Rise / Fall time max: Command capability: – S/W requirements: 2005 MAPLD 16 +30 V +/- 2 V 2 k. Ohm, 4 m. H 50 ms 1 ms Unique bit pattern, only one at a time, arm first Various (beyond scope of example) 125 Design Integrity Concepts

Tiered Verification – Example (cont. ) • Tier 1 – Basic Verification (functional simulation) – Verify each channel fires for a specific bit pattern / address – Verify cannot be fired without an arm command – Verify that other complementary patterns don’t accidentally fire the pulse – Analyze drive capability of FPGA and input / output requirement / capability of translation chip – Evaluate glitch hardness of logic selected 2005 MAPLD 126 Design Integrity Concepts

Tiered Verification – Example (cont. ) • Tier 2 – Board level – Test to confirm signal level between FPGA and driver chips is compatible – Test rise, fall, duration, level with dummy load – Repeat test for all 16 outputs – Replace dummy load with relay, verify that the relay switches for all 16 outputs – Repeat over temperature if necessary 2005 MAPLD 127 Design Integrity Concepts

Tiered Verification – Example (cont. ) • Tier 3 – Box level – Attach GSE (verifies with relays) – Functionally pulse all outputs • Verify relay switches • Verify gross drive duration • Tier 4 – Box and operational software level – Verify functionality commanded through software – GSE verifies correct relay switch • Tier 5 – System – Verify gross functionality as needed for system operation 2005 MAPLD 128 Design Integrity Concepts

Tiered Verification - Notes • Most complex and detailed functionality is verified at lower levels – Performance – Error cases – Predictive of future operation • • Higher levels focus on functionality / operation Lower level verification requires more manual touch Higher level verification is more automated STE and test procedure developed as appropriate – Purpose is maximum test completeness – As well as “bang for the buck” 2005 MAPLD 129 Design Integrity Concepts

Concluding the Process • (“… through the provision of objective evidence…”) • The importance of traceability – Something is only verified (formally) when it is documented – The tie between verification item and specification must be made (verification matrix or database) • The importance of planning – Establish the links between requirement, method, level, test case early – Use the established structure to develop verification items • The importance of buy in – All responsible must complete verification and report results – Systems engineers must track and close – People must keep up with the process • Only personal commitment from all involved will ensure that the process is successful 2005 MAPLD 130 Design Integrity Concepts

Summary • Verification is to prove success, not avoid failure • Verification planning is an integral part of the project lifecycle – Must be initiated with specification (including customer buy-in) – Must be included in design effort – Must be used to develop documentation and appropriate test hardware • Verification is more than final test – Begins at lowest possible level – Incorporates analysis and inspection throughout design process – Develops proof of compliance at all levels of operation • A thorough test is complicated – Requires extensive work – Must trade perfect for practical (a tiered test approach can help) • Verification processes must be tracked and documented – The whole team must buy in 2005 MAPLD 131 Design Integrity Concepts

What Do You Mean It Doesn’t Do What We Thought? Validating a Design 2005 MAPLD 132 Design Integrity Concepts

Agenda – Design Validation • • • Concepts and implications Specification mitigations Design mitigations Test set mitigation Summary 2005 MAPLD 133 Design Integrity Concepts

Concepts • Validation – Confirmation, through the provision of objective evidence, that the requirements for a specified intended use or application have been fulfilled 2005 MAPLD 134 Design Integrity Concepts

Implications • The issues relating to validation encompass those of verification • Additional concerns with validation – Caused by the need to match the application to the product – Desired application has been translated to specification through the requirements process – Requirements process is by nature imperfect – Sometimes the specification does not satisfy the needs of the application • Result – a verified product might be invalid – May require significant rework to the product – May require accepting reduced functionality (waiver) • A goal of the development process is to minimize validation failures – Begins in review of the requirements process (hopefully primary point) – Mitigate by design activities – Reduce by robust test set design 2005 MAPLD 135 Design Integrity Concepts

The Implication Illustrated • Alice electronics (detector component and C&DH component) • Joint requirement: process > 10 k sustained events per second • Individual requirements defined for detector and C&DH processing – Both met individual requirements for processing – When combined only 6 -7 k sustained events per second • Verification of individual units led to invalid system • What went wrong? – The overall requirements were not broken down correctly – The C&DH and DE test sets were not high fidelity • Verified functionality, not performance • We got lucky that a waiver was acceptable 2005 MAPLD 136 Design Integrity Concepts

Specification Mitigation • Only list requirements, not desirements • State unambiguous performance requirements • Build in adequate margin – Not open-ended enhancement, but – Enough to accommodate performance shortfalls • Ruthlessly remove TBDs • Insist on definite test methods for mitigation • Remember – Unless application needs can be unambiguously specified, they won’t be met! 2005 MAPLD 137 Design Integrity Concepts

Design Mitigation • Implement specification exactly • Isolate various sub-sections – Minimizes “corner cases” and negative interactions – Allows correction with minimal impact when things don’t work right • Verify complex functions early, thoroughly, and completely – Allows early look at potential problems – Analysis / simulation / what ifs should be as realistic as possible • Insist on end-user review of implementation – Allows user community to comment – Minimizes misunderstandings upon delivery • Develop test plans that have high fidelity to the end application 2005 MAPLD 138 Design Integrity Concepts

Test Set Mitigation • Ensure interfaces are maximally flight-like – Precludes misunderstandings of characteristics – Provides early indication of problems – Don’t emulate only one characteristic of interface (R vs. R+L) • Make test set reasonably sophisticated – Sufficient complexity to reproduce operational timing – Adequate functionality for stress testing • Run all interfaces at maximum speed with margin • Don’t let the same group build the tested unit (design) and the unit tester (test bench) – Identical assumptions might go into both ends of an interface – Faithful reproduction is dependent on familiarity (if possible, test bench should be provided by end user) 2005 MAPLD 139 Design Integrity Concepts

Test Set Mitigation (cont. ) • Make the control interface as application like as possible – Forces correct command structures / types – Allows all test scripts to be reproduced at higher levels • If at all possible, incorporate early interface tests of real engineering hardware • Keep the test (or simulation) environment constant unless the flight system changes – Don’t change test equipment hardware configurations – Apples to apples comparisons during tests vital – Ensure that flight changes are reflected in test set design 2005 MAPLD 140 Design Integrity Concepts

Test Set Mitigation (cont. ) • Use the same controls for test set development as for flight unit development – Configuration management – Software development – Peer reviews • Build in diagnostics so that anomalies can be traced to test equipment or unit under test • Ensure that test results mean something – – Pass / fail criteria clear Allowable flight parameter variations included Reasonable displays (with significant information clearly shown) Ensure that test set accommodates calibration 2005 MAPLD 141 Design Integrity Concepts

Summary • Successful verification does not always guarantee successful validation • Techniques can be incorporated that improve the likelihood that validation will succeed – Careful specification development – Thorough and cautious design techniques – Extensive test set fidelity to flight requirements • Effective techniques for validation are extra effort – More time consuming – More expensive – But, definitely worth it. 2005 MAPLD 142 Design Integrity Concepts