Constellation Program Surveillance Strategy 28 May2008 Jim Schultz

Constellation Program Surveillance Strategy 28 -May-2008 Jim Schultz 28 -October-2008 1

Cx. P Surveillance Strategy • Traditional Surveillance Strategies consist primarily of Inspections and Audits. • Inspections and audits are necessary, however they are “after-the-fact” and should not be the sole surveillance approach. 28 -May-2008 2

Cx. P Surveillance Strategy (cont. ) • Constellation Projects are required to develop and document a surveillance strategy concurred with by the Constellation Program QA manager. – The level of risk and the impact of failure are major determinants in defining the type of surveillance required. If the impact of failure is minor and the risk low, only a minimal amount of insight driven surveillance is needed. Conversely, if the failure impact could be significant and the risk high, more extensive surveillance (including possible oversight) is necessary. – The focus should be on prevention rather than detection, i. e. , emphasizing controlled processes and methods of operation, as opposed to relying solely upon inspection to identify problems. 28 -May-2008 3

Cx. P Surveillance Strategy (cont. ) • Identify specific risks and surveillance methods used to help track and mitigate these risks. • Examples of Surveillance Methods – Metrics reviews (e. g. , process escapes, audits scheduled, audits completed, audit findings, nonconformances, corrective actions taken, cost of quality, delivery performance, quality performance, process performance, cycle time, labor performance (utilization, efficiency, productivity), lost workdays/recordable incidents, employee surveys, etc. ) – Document/quality record reviews – Audits 28 -May-2008 4

Cx. P Surveillance Strategy (cont. ) • Examples of Surveillance Methods (cont. ) – Management information (e. g. , management review minutes (w/ focus on internal audit and corrective action effectiveness), development projects status reports, board/meeting minutes, Contractor weekly activity reports, NASA/Constellation Project Contractor Management tag-ups, Special Issues meetings, Flight Readiness Review chart, Data Requirements Documents (DRD), etc. ) – Performance reviews (e. g. , witnessing the performance of the task being surveyed such as flight controller simulation and real-time operations, process audits, Flight Readiness Reviews (FRR), real-time inline product inspection, tests, etc. ) 28 -May-2008 5

Cx. P Surveillance Strategy (cont. ) • Examples of Surveillance Methods (cont. ) – Quality tool implementation (e. g. , Design of Experiments (DOE), Multiple Environment Overstress Testing (MEOST), Environmental Stress Screening (ESS), Quality Functional Deployment, Total Productivity Maintenance (TPM), Benchmarking, Self-Check Systems, Next Operation as Customer, Supply Management, Total Value Engineering, Cycle Time Reduction, Lean Six Sigma, Fault Tree Analysis, Process Failure Modes and Effects Analysis (PFMEA), Statistical Process Control (SPC), Probability Risk Analysis (PRA), PRACA, FMEA/CIL, etc. ) – Customer feedback - formal and informal (e. g. , board representatives, Program Offices, all levels of contract management, customer surveys, field data, etc. ) 28 -May-2008 6

Cx. P Surveillance Strategy (cont. ) • Examples of Surveillance Methods (cont. ) – Self-Check Systems • Provide sensors (electrical, mechanical, or visual) that warn an operator that a mistake has been or is about to be made. Keep the solutions simple. – In-Process Monitoring – address condition before it becomes a problem (e. g. , Statistical Process Control (SPC), Chemical Fingerprinting, cycle time, precursor analysis, etc. ) 28 -May-2008 7

Cx. P Surveillance Strategy (cont. ) • Examples of Surveillance Methods (cont. ) – Supplier Performance • Quality performance • Delivery performance • Total recordable incident rate (TRIR) • Lost work incident rate (LWIR) • Earnings Before Interest, Taxes, Depreciation, and Amortization • Mergers or changes in management • Mergers or changes in major subcontractor management 28 -May-2008 8

Cx. P Surveillance Strategy (cont. ) • The surveillance strategy employed must not drive technical requirements. • The surveillance strategy employed must not drive design; instead design, manufacturing, and assembly must drive surveillance strategy employed. • The surveillance strategy should change over the life cycle of the item (vehicle, system, etc. ). As the item ages and more risk information is identified, changes may be necessary or beneficial to reflect either an increase or decrease in risk. • The surveillance strategy should be reviewed and adjusted at least annually. 28 -May-2008 9

Cx. P Surveillance Strategy (cont. ) • Use a risk management approach and apply Penetration Levels based on level of risk of each area: – High Risk Areas = Higher Penetration – Low Risk Areas = Lower Penetration • Minimum acceptable Penetration Level is contingent on ensuring: – Safety for the public. – Safety for astronauts and pilots. – Safety for NASA workforce. – Safety for high-value equipment and property. 28 -May-2008 10

Cx. P Surveillance Strategy (cont. ) • Feedback should be used to adjust level of penetration – Changes in risk – NASA and Contractor metrics – Customer evaluations – Quality performance / delivery performance – Cost of Quality (rework, repair, scrap, warranty, concessions, recalls/GIDEPs/NASA Alerts, returns, etc. ) – Inspection / test results – Audit / survey / assessment results – Schedule aggressiveness – Number and scope of requirements and design changes 28 -May-2008 11

Cx. P Surveillance Strategy (cont. ) Typical Penetration Mapped on Risk Chart L i k e l i h o o d of w s vie esse e R oc Pr e th o ? y d ngs e i h d t ht th i D ig r 28 -May-2008 of ion w ie ntat v Re eme pl do ? y Im t e th righ d Di ings th QA & R g. S ion t a etr n Pe sin ea cr In Risk H = High M = Medium L = Low Penetration Level 5 Level 4 Level 3 Level 2 Level 1 Consequence of Failure 12

Cx. P Surveillance Strategy (cont. ) • Acceptable risk is the risk that is understood and agreed to by the appropriate parties sufficient to achieve the defined success criteria within the approved level of resources. • Characterization of a primary risk as "acceptable" must be supported by the rationale, with the concurrence of the senior management, that all reasonable mitigation options (within cost, schedule, and technical constraints) have been instituted. 28 -May-2008 13

Surveillance Activity must Add Value • The surveillance strategy must add value to the output of the process. – Identify and correct problems (or potential problems) as early as possible in the process. – Do not add unnecessary surveillance activities in the process. • The surveillance strategy should be streamlined into the manufacturing/assembly process – Steps performed within process during operation. – Avoid/rethink surveillance that inspects batches of parts. Avoid surveillance bottle necks. 28 -May-2008 14

Surveillance Activity must Add Value Yes Is it (part of): *adjusting? Activity Does it convert input to output? Are we doing this for customer (vs. (Is it required to for us)? Does the end customer care? *approving? *doing work over? (Is the end customer *expediting? willing to pay *inspecting? for it? ) *moving? *reviewing? *storing? No *troubleshooting *waiting Yes produce output? ) No No No Yes NVA Yes VA 28 -May-2008 15

Cx. P Surveillance Strategy (cont. ) • Summary: – The level of surveillance is determined by the level of risk and the impact of failure. – There are many different surveillance methods; not just inspection. – Put an emphasis on surveillance methods that uncover conditions before they become problems (e. g. , precursor, self-check, SPC, business/financial changes, etc. ) – Characterize the acceptable risk with rationale and concurrence from Senior Management. – Surveillance methods must add value and be 28 -May-2008 16 streamlined into processes.

Questions? 28 -May-2008 17

Backup 28 -May-2008 18

DR AF T Assurance Penetration Levels Program / Project Level The premise of risk-based surveillance is simply this: The higher the risk in terms of safety, hardware criticality, or contractor’s stability or capability, the deeper the surveillance penetration. Penetration Level 5 4 Risk Factors (Typical) High risk to operational safety High risk to system safety Hardware is critical (Mission Critical and/or Criticality 1 / 2) Contractor stability is poor Contractor capability is poor NASA has in-line involvement. Performs a complete and independent evaluation of processes and products that satisfy SR&QA requirements. Approves all SR&QA processes and products. Approves all processes, including implementing instructions, that satisfy SR&QA requirements. Approves/disapproves or recommends approval/disapproval of request to depart from or permit noncompliance with SR&QA requirements. Mandatory NASA involvement, contractor cannot proceed without NASA “Go”. High risk to operational safety High risk to system safety Hardware is critical (Mission Critical and/or NASA has in-line involvement with processes and products associated with program accepted risks to SR&QA. Performs a complete and independent evaluation, and approval of these processes and products. Reviews and approves all implementing work instructions. Approves/disapproves or recommends approval/disapproval of request to depart from or permit noncompliance with SR&QA requirements. Mandatory NASA involvement in accomplishment of processes and products associated with program accepted risks to SR&QA. Monitors, during occurrence, all other processes and products that satisfy SR&QA requirements. Contractor cannot proceed without NASA “Go”. Criticality 1 / 2) Contractor stability is good Contractor capability is good 3 2 1 Assurance Implementation (Typical) Medium risk to operational safety Medium risk to system safety Hardware is critical (Mission Essential and/or Criticality 2) Contractor stability is very good Contractor capability is very good NASA has in-line involvement with processes and products associated with program accepted risks to SR&QA. Reviews and approves the contractor completed evaluation of these processes and products, including implementing work instructions. Approves/disapproves or recommends approval/disapproval of request to depart from or permit noncompliance with SR&QA requirements. Mandatory NASA involvement in accomplishment of processes and products associated with program accepted risks to SR&QA, contractor cannot proceed with these tasks without NASA “Go”. Monitors all other processes and products that satisfy SR&QA requirements, contractor proceeds unless NASA says “No Go”. Some risk to operational safety Some risk to system safety Hardware is critical (Mission Essential) Contractor stability is excellent Contractor capability is excellent NASA has in-line involvement with the review and approval of processes and products associated with program accepted risks to SR&QA. Mandatory NASA involvement in the accomplishment of these tasks, contractor cannot proceed with these tasks without NASA “Go”. Monitors all other processes and products that satisfy SR&QA requirements, contractor proceeds unless NASA says “No Go”. Approves/disapproves or recommends approval/disapproval of request to depart from or permit noncompliance with SR&QA requirements. Low risk to operational safety Low risk to system safety Hardware is not critical Contractor stability is exceptional Contractor capability is exceptional NASA has in-line involvement with the review and approval of products associated with program accepted risks to SR&QA (i. e. , Hazard reports, CILs, etc. ). Mandatory NASA involvement may be required in the accomplishment of these tasks (GMIPs). Evaluates contractor data associated with all other processes and products to assure satisfaction of SR&QA requirements, contractor proceeds unless NASA says “No Go”. Approves/disapproves or recommends approval/disapproval of request to depart from or permit noncompliance with SR&QA requirements. 28 -May-2008 19

T DR AF 28 -May-2008 Assurance Penetration Levels Process / Task Level 20

el v e L ct tion e oj tra r / P ene am A P r og &Q r P SR 28 -May-2008 P E N E T R A T I O N 21

el v Le ion k at s a r / T net ss Pe e oc QA r P R& S 28 -May-2008 P E N E T R A T I O N 22

T AF DR Assurance Levels - Quality 28 -May-2008 23