Standardizing Product Development Process for NonStandard Products Karen

Standardizing Product Development Process for Non‐Standard Products Karen Bolen

Agenda • • • Background Motivation History Objectives Certification Strategy QMS Architecture Features Components Additional ISO Required Features Lessons Learned Measures of Effectiveness

Background Steve Jobs: “The Man Who Always Seems to Know…WHAT’S NEXT” • Innovation is thought to be important to economic growth • Research and Development (R&D) organizations are generally the incubators of innovation • R&D customers are frequently government customers •

Background (Continued) • Research – waste of taxpayer dollars?

Background (Continued) • R&D products are often proof‐of‐concepts or prototypes • R&D projects are often short‐term and work is innovative without repetition • Cost can outweigh benefit of standardizing processes for innovative products

Background (Continued) ISO 9001 quality management standard bases requirements on process‐driven best practices • ISO successes • • Early quality standards focused on reducing fatalities in British bomb factories in WWII ISO 9001 certified organizations often realize better business performance and increased profits than firms without a certified QMS are able to achieve (Chow‐chua, Goh & Wan, 2003) Cost of poor quality

Motivation • • • RFPs currently call for compliance with ISO/CMMI or other quality standards Quality certifications allows organization to qualify for major procurements Commercial or industrial customers often require ISO 9001 certification enables organization to competitively bid on very large multi‐contracting vehicles Process identifies focus areas for operational efficiency improvements

History • 1996: Organization made it part of the corporate strategy to implement quality management and formal process improvement programs to achieve progressively higher quality and control • 2003: Organization was evaluated under the Software Engineering Institute’s (SEI) Capability Maturity Model (CMM) as Level 3 compliant • 2016: Organization achieved ISO 9001: 2008 certification of its quality management system (QMS) • 2017: Organization successfully completed a surveillance audit in 2017 • 2018: Organization refined the project and product development processes to transition to certification under the ISO 9001: 2015 standard

History –Obstacles to Certification • After attaining Level 3 compliance for CMM in 2003, management decided ISO Certification should be pursued • • Many senior engineers were not receptive to ISO 9001‐aligned process • • • Hundred of pieces of equipment needed calibration brought up‐to‐date Mix of projects was varied and included some project work that did not fit the ISO framework • • They were reluctant to abandon their own devised processes Measuring equipment calibrations had expired • • Took several years to get resources in place ISO aligned process was difficult to define for OSHA inspections, training classes, and some aircraft legacy systems work 99% of customer base could care less about ISO certification and did not want to pay for any ISO related expenses • No quality management requirements in most funding vehicles or contracts ISO 9001 certification is not common in applied research and development ; more common in manufacturing environment • ISO auditors have difficulty auditing applied research and development projects

Objectives Define flexible yet standard processes and procedures for the development of low volume, one‐of‐a‐kind products • Create a maintainable QMS that can be certified and easily updated for future versions of the ISO standard • Provide a framework that can coordinate the audits of many concurrent projects • Develop continuous improvement initiatives to respond effectively to problems and issues with project activities •

Certification Strategy Build QMS on CMM process document foundation • Utilize current practices when possible • Write process documentation at high level to address environment • • No manufacturing or production work, mostly prototypes Work is set up as research projects or programs with informal and formal deliverables Delivered work products are variable; software, hardware, software/hardware hybrids, complex system integration, system engineering support, reports, training, human systems engineering Project Managers (PMs) act as CEOs of their project activities Projects solve unique problems with unknown or vague requirements No common document repository, few standardized forms and work instructions, large transient work force Assign definition of processes and procedures to researchers, not QA

QMS Architecture

QMS Architecture Features Initiated at proposal stage • Guided by organization Project Management Manual at high level • Focuses on additional ISO‐ 9001 required project planning and work product development activities •

Components ‐ Initiation • • Starts at project notification (unique ID) Requires scheduling and conducting Initiation Meeting • • • Agenda for meeting is completed Initiation Form Ensures project contractual, financial, technical, high‐level project planning, and pertinent management information is briefed Addresses project risks, potential issues, and mitigation strategies

Components ‐ Planning Generates project and work product planning documentation • Describes plans for ISO‐ specific requirements • • • Verification, validation, configuration management, calibration, and subcontractors Supplemented by Execution Checklist for Project Management • Incorporates QMS processes and best practices for design and development of work products

Components ‐ Execution • • • Kicked‐off by internal and sponsor meetings Develops work product per project plan Includes periodic status meetings and formal reviews • • Delineated further into QMS activities • • Initiates corrective/preventive actions when appropriate Work Product Analysis/Design Work Product Development/Integration Work Product Evaluation and Delivery Completed when all work products delivered

Components – Execution: Work Product Analysis/Design Documents requirements • Defines design or technical approach • Includes formal and informal reviews • Conducted iteratively as needed •

Components – Execution: Work Product Development and Integration • Performs all development and integration activities • • Coding Documenting Fabrication Integration Procurement Calibration Requires formal and informal reviews Conducted iteratively as needed

Components – Execution: Work Review/Test and Delivery • • • Inspects, reviews, and tests work products Identifies and controls nonconforming work products as needed Executes corrective/preventive actions when appropriate Incorporates planned verification and validation activities Requires formal and informal reviews, including subcontractor work Conducted iteratively as needed

Components – Monitor and Control • Oversees project execution • • Monthly project status reports generated and reviewed by management for every active project Occurs throughout life of project Ensures risks and issues are reviewed and mitigated Focuses on successful completion of project objectives and delivery of contractual deliverables

Components – Closeout • Finalizes project activities of development and delivery Ties to project initiation and closes loop of development • Manages work product releases • Solicits sponsor satisfaction • Evaluates supplier performance •

Additional ISO Requirements • • • • Perform internal audits to ensure compliance Conduct management review meetings to review QMS continuing adequacy, effectiveness and strategic objective alignment Record organizational external and internal issues Define QMS scope for boundaries and applicability Document process inputs, outputs, sequence, interactions, measurements/performance indicators, resources, responsibilities and authorities, and risk and opportunities Document quality policy; documenting quality manual is advisable Establish quality objectives and measurements of effectiveness Determine process for documented information and change management approach Provide resources for infrastructure and environmental support Address workforce competency requirements Implement continuous improvement processes and activities Track and protect sponsor property Develop sponsor communication approach and plans for organizational knowledge transfer

Additional ISO Requirements ‐ Examples • Record organizational external and internal issues https: //www. liveplan. com/blog/what‐is‐a‐swot‐analysis‐and‐how‐to‐do‐it‐ right‐with‐examples/

Additional ISO Requirements ‐ Examples Document process format: • Inputs, outputs, sequence, interactions, effectiveness evaluation, resources, and risk and opportunities • • Turtle Diagram

Additional ISO Requirements ‐ Examples Limited mandatory forms to a few checklists • Checklists walk PMs through QMS process and serve as templates for next project •

Additional ISO Requirements – Examples – Documented Information Configuration Management Documentation 1. ESTABLISH CM LIBRARY • • • Version control toolset or method Project storage location(s) Access control and Backup 2. IDENTIFY CHANGE MANAGEMENT APPROACH • • • Timing of formal control Change control approach and toolset or method Change Control Board (CCB) Record method for change requests (CR) Methodology for review and disposition of change requests 3. ESTABLISH NAMING/ NUMBERING CONVENTION • • Unique naming conventions for deliverables and project items Unique naming/numbering conventions for versions and baselines 4. BASELINE IDENTIFICATION • • Item types to be included in baselines Mechanism for recording differences between baselines 5. BUILD INSTRUCTIONS • • Build environment description Required scripts or instructions Operational/runtime environment QA Audit of software build if applicable 6. WORK PRODUCT RELEASE • • Version description documentation Version description review method

Lessons Learned • • Define value‐added processes rather than ISO‐required processes Assign process documentation tasks to the end users or stakeholders Minimize standard form creation if data is available elsewhere Pilot processes before establishing at organizational level View any employee feedback as continuous improvement opportunities Investigate causes and improve usability of processes that have low compliance history Utilize current practices and tweak if needed for ISO compliance

Measures of Effectiveness ‐ Indicators • Document process measurements‐ performance indicators Indicators related to: Return of quality management system investment Internal failure cost to total cost ratio External failure cost to total cost ratio Total quality related cost to total cost ratio Value of key customers Average profit from one quality improvement project § Nenadál, Jaroslav. (2016). Adequacy, Suitability, Effectiveness and Efficiency of Quality Management Systems: How to perceive and assess them? . Quality Innovation Prosperity. Level of employees training effectiveness Overall equipment effectiveness, etc.

Measures of Effectiveness Successfully attained ISO 9001: 2015 certification in organization’s biggest lab (500+ researchers) with zero non‐conformances • On‐time delivery of contractual work products rose from 97% to 99% on‐time while project workload almost doubled in one year • Unbillable expense rate to total revenue lowest among all of organization labs • • Unbillable expenses indirectly related to running over budget and schedule • Recent establishment of user‐focus groups to determine best practices for software development and improved usability of organizational tools and resources • Helps standardize internal processes that comply with ISO requirements Certification enables lab to successfully bid on very large multi‐contracting vehicles Internal audit findings significantly reduced New management template for monthly project status reporting reinforces QMS compliance and alignment with ISO requirements • Organizational culture reflects increased focus on quality • • •

Questions

References Amador GJ, Mao W, De. Mercurio P, Montero C, Clewis J, Alexeev A, Hu DL. 2015 Eyelashes divert airflow to protect the eye. J. R. Soc. Interface 12: 20141294. http: //dx. doi. org/10. 1098/rsif. 2014. 1294 • Clare Chow‐Chua, Mark Goh, Tan Boon Wan, (2003) "Does ISO 9000 certification improve business performance? ", International Journal of Quality & Reliability Management, Vol. 20 Issue: 8, pp. 936‐ 953, https: //doi. org/10. 1108/02656710310493643 • http: //www. news. gatech. edu/2016/09/22/three‐isye‐researchers‐receive‐ 2016‐golden‐ goose‐award‐honey‐bee‐algorithm • Hamidreza Marvi et al. , “Sidewinding with Minimal slip: Snake and Robot Ascent of Sandy Slopes, ” Science (2014). • Images: • Honeybees: https: //www. google. com/search? rlz=1 C 2 NWPB_en. US 508 US 531&biw=1366&bih=662&noj=1&t bm=isch&sa=1&btn. G=Search&q=honey+bee+images • Mudskipper: http: //www. rh. gatech. edu/news/550391/robot‐helps‐study‐how‐first‐land‐animals‐moved‐ 360‐ million‐years‐ago • Sidewinder snake: https: //www. treehugger. com/gadgets/robots‐reveal‐secrets‐behind‐how‐sidewinder‐snakes‐ move. html