Staffless SQA An Effective SQA Alternative Unmesh Gundewar
- Slides: 41
Staffless SQA An Effective SQA Alternative Unmesh Gundewar E-Mail: gundewar_unmesh@emc. com February 25, 2003 EMC and the EMC logo are registered trademarks and "where information lives" is a trademark of EMC Corporation. 1
Agenda n Introduction n SQA Purpose, Goals, Roles and Responsibilities n Typical SQA Approach and Issues n EMC’s Strategic Approach for Staffless SQA n SQA Coverage (Level 2 and Level 3) n Challenges n Lessons Learned n Summary December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved. 2
Introduction To meet the needs of today’s tumultuous marketplace, delivery of products and services with superior quality is the key to corporate success and survival. As a result, to improve quality, many corporations are using CMM framework for process improvement. “Software Quality Assurance is the least frequently satisfied Level 2 Key Process Area” — August 2002, Process Maturity Profile published by Software Engineering Institute (SEI) December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved. 3
SEI Capability Maturity Model Level Focus 5 Optimizing Continuous process improvement Defect prevention Technology innovation Process change management 4 Managed Product and process quality Process measurement and analysis Quality management Engineering process Organization process focus Organization process definition Peer reviews Training program Intergroup coordination Software product engineering Integrated software management Project planning Project tracking Subcontract management Quality assurance Configuration management Requirements management 3 Defined 2 Repeatable 1 Initial Result Key Process Areas Productivity & Quality Heroes Risk Carnegie Mellon University - Software Engineering Institute December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved. 4
The CMM Structure Maturity Levels Process Capability ate c i d in contain Key Process Areas organized by ve Goals ie ach Common Features Implementation or Institutionalization contain ss re add Key Practices be Infrastructure or Activities December 9, 2002 cri s e d Copyright © 2002 EMC Corporation. All Rights Reserved. 5
Expanding CMM Structure 5 CMM maturity levels 0 -7 KPAs per maturity level = 18 total KPAs 2 -4 goals per KPA = 52 goals = 316 total key practices 5 common features per KPA with 1 -n practices each December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved. 6
Software Quality Assurance Goals Purpose Provide management with appropriate visibility into the processes being used by the project, and of the products being built. n Goal 1 Quality Assurance activities are planned. n Goal 2 Adherence of products and activities to the applicable standards, procedures, and requirements is verified objectively. n Goal 3 Affected groups and individuals are informed of quality assurance activities and results. n Goal 4 Noncompliance issues that cannot be resolved within the project are addressed by senior management. December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved. 7
SQA Roles and Responsibilities n Ensure that the organizational standards, practices, and procedures have been selected and followed. n Identify all deviations from the agreed upon processes as early as possible n Assist project managers in developing quality assurance Software Quality Assurance Plan n Promote best practices n Conduct audits and check all software lifecycle work products n Escalate noncompliance and non-resolvable issues n Periodic reporting to affected groups December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved. 8
Typical SQA Approach and Issues Approach: n Establish Group to Focus on SQA Issues: n Low Morale n Funding Issues n Policing/Enforcement n Lack of Respect December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved. 9
EMC’s Strategic Approach to Staffless SQA Program managers lead cross-functional teams and use corporate processes to drive consistency, product quality, business readiness, and speed time to market for product and service solutions. December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved. 10
EMC’s Strategic Approach to Staffless SQA Role of a Program Manager n Leads one or more cross-functional teams to drive end-to-end product and service solutions for EMC customers. n Defines scope, develops cross-functional program plans, drives program to completion, and ensures that the product is delivered on time and meets the EMC quality goals. n Leads the cross-functional team to resolve program issues. Escalates issues to the appropriate senior management team when needed. December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved. 11
EMC’s Strategic Approach Implementation n Organizational Software Development (SDP) Process – Corporate Lifecycle (LC 300) – Software Development Lifecycle Software Development Process and Checkpoints Professional Development Process and Checkpoints – – n Readiness Management Team (RMT) Process n Escalation Process n Compliance Report n Quality Metrics December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved. 12
EMC’s Product Lifecycle n A Corporately-Accepted Process n Guideline for: - Product Creation - Product Testing - Delivery and Release - Field Support - Services - Replacement/End-Of. Life n A Flexible Model for Program Management Teams (PMT’s) December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved. 13
Software Development Process Activities and Checkpoints 0 1 2 3 4 Release Specific PRD Time 5 6 EFS 7 GA PSA SRS Kickoff SPPp SPPc SDS PDS STP Prototype Waterfall Example 0 1 2 3 Kickoff Preliminary Requirements Review Critical Requirements Review Preliminary Planning Review December 9, 2002 PEL Test Cases Coding Unit Testing 4 5 6 7 QA Tracking Int QA Testing Critical Planning Review Functionally Complete Beta Ready Release to Operations Copyright © 2002 EMC Corporation. All Rights Reserved. 14
Service Offering Development Overlay and Checkpoints 1 Release Specific PRD 2 N P C 0 3 Positioning & Bundling 4 Beta Ready Product PDS 5 Publish! 6 PEL t 0 SDR PS Positioning Impact RTO Time SCR t 1 Develop Build Validate LCR Validate Monitor Engagements t 2 LC 300 Phase 0 LC 300 Phase 1 1 (TSG) PS Positioning & Impact Review 2 Service -Dev Validation of SDR/SCR 3 Final Approval to build Offering 4 Field Readiness Review December 9, 2002 LC 300 Phase 3 5 Publication Review 6 Monitoring initial release of Service 7 t – Times will vary depending upon size of product and type of Service Copyright © 2002 EMC Corporation. All Rights Reserved. 15
Readiness Management Process {Full Product Name V#} Readiness Dashboard {Date} Current LC 300 Phase(s): {Phase #(s)} Program Manager: {Name} Date Various Milestones December 9, 2002 Date Beta Starts Date SVT Date GA Post GA Steady State Copyright © 2002 EMC Corporation. All Rights Reserved. 16
Readiness Management Process (Continued) {Product Name V#} Schedule Delta {Date} Phase 0 Milestone Phase 1 Tentative Dates PDS Dates {date} Phase 2 Phase 3 {date} Estimate of Next Phase PRD Sign off NPC Phase 0 Presentation PDS Sign off BRT Product Overview Beta Start NPC Go to Market Presentation PS Offering Marketing Collateral Field, Help Desk, Sales trained SVT begins SVT With MASTER begins RTM GA Quality, Service, Unit Tracking begins Explanation: Impact: § {What are the factors that contributed} § {What does this mean} § {How has the plan been modified? } December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved. 17
Readiness Management Process (Continued) {Product Name V#} Program Overview {Date} Highlights: Risks/Issues: § {what are the items to be excited about} § {Facts - Not a value judgment} § {what is going very well} § {Acknowledgement of the Risks – Future items} § {nine bullets max} § {These may map directly to an ASK § {nine bullets max} Lowlights: Ask: § {Where are we NOT performing up to our capability – past events} § {what does the Cross Functional Team Require from Management? } § {These may map directly to an ASK} § {nine bullets max} December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved. 18
Leadership Meetings Workflow Escalation Process Cross-Functional Representatives Corp Mktg G/S Customer Services Markets & Products Corp Mktg Alliance & Channel Partners G/S Professional Services Corporate Quality PMT Product Manager December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved. Tech Pubs 19
Compliance Metrics SW Development Checkpoint Status – MM/DD/YY Total: 34 Product Releases n Check. Point Breakdown: n Exec Summary Reds: (Products Below with Arrow) n 7 Programs with Missing Checkpoint Items: • 2 Program Schedules • 5 Not Scheduled for Reviews n 5 Programs have a ‘Red’ Checkpoint Item: Ø Program A • Program B • Program C Ø Program D • Program E December 9, 2002 Program Managers’ RMT Dashboards Copyright © 2002 EMC Corporation. All Rights Reserved. 20
0 5 -Sep 6 -Sep 7 -Sep 8 -Sep 9 -Sep 10 -Sep 11 -Sep 12 -Sep 13 -Sep 14 -Sep 15 -Sep 16 -Sep 17 -Sep 18 -Sep 19 -Sep 20 -Sep 21 -Sep 22 -Sep 23 -Sep 24 -Sep 25 -Sep 26 -Sep 27 -Sep 28 -Sep 29 -Sep 30 -Sep 1 -Oct 2 -Oct 3 -Oct 4 -Oct 5 -Oct 6 -Oct 7 -Oct 8 -Oct 9 -Oct 10 -Oct 11 -Oct 12 -Oct 13 -Oct 14 -Oct 15 -Oct 16 -Oct 17 -Oct 18 -Oct 19 -Oct 20 -Oct 21 -Oct Daily Triage for Product A GA Issues 300 200 147 100 86 48 12 New December 9, 2002 Resolved 12 A. 1 Open Copyright © 2002 EMC Corporation. All Rights Reserved. 0 0 A. 0 Open 21
Product B Open Problem Tracking (OPT) and Backlog Severity 1 Severity 2 & 3 New Sev 1 (Moving average) 1300 1200 1100 1000 900 800 700 600 500 400 300 200 December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved. 28 -Oct 21 -Oct 14 -Oct 7 -Oct 30 -Sep 23 -Sep 16 -Sep 9 -Sep 26 -Aug 19 -Aug 12 -Aug 5 -Aug 29 -Jul 22 -Jul 15 -Jul 8 -Jul 1 -Jul 24 -Jun 14 -Jun 0 7 -Jun 100 22
Product A. 1 Test Trends vs A. 0 Test Coverage % Tests Passed 100% 86% 81% 75% 83% 79% 78% 70% 71% 70% 63% 60% 50% 49% 45% 42% 40% 44% 30% 27% 25% 20% 22% December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved. 28 -Oct 21 -Oct 14 -Oct 7 -Oct 30 -Sep 23 -Sep 16 -Sep 9 -Sep 26 -Aug 19 -Aug 12 -Aug 5 -Aug 29 -Jul 22 -Jul 15 -Jul 8 -Jul 1 -Jul 24 -Jun 18% 7 -Jun 10% A. 0 % Tests Passed 100% 98% 99% 98% 98% 99% 94% 95% 93% 92% 92% 93% 89% 88% 90% 80% A. 0 Test Coverage 23
Software Quality Assurance Coverage Requirements Management Key Process Area KPA Verification 3 Compliance RM The activities and work Product requirements product reviews and/or (PRD) are approved and Ve 3 audits are conducted to signed by affected groups manage requirements and (Checkpoint 1). results are reported. Changes to requirements are approved by affected groups (Requirements Management Process). Commitments are negotiated and agreed to by the affected groups. December 9, 2002 Ø Italic compliance is in progress 24 Copyright © 2002 EMC Corporation. All Rights Reserved.
Software Quality Assurance Coverage Project Planning Key Process Area KPA Verification 3 SPP The activities and work product reviews and/or Ve 3 audits are conducted for project planning and results are reported. Compliance Preliminary project plan is approved (Checkpoint 3). Standards are used for preparing the project plan. The content of the software development plan is reviewed. December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved. 25
Software Quality Assurance Coverage Project Tracking and Oversight Key Process Area KPA Verification 3 PTO The activities and work product reviews and/or audits are conducted for project tracking and oversight and results are reported. Compliance Revised project plan is approved [Critical Planning Review] (Checkpoint 4). The activities for revising the software development plan is reviewed with affected groups. The content of the revised software development plan is reviewed. December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved. 26
Software Quality Assurance Coverage Configuration Management Key Process Area KPA Verification 4 SCM The activities and work product reviews and/or Ve 4 audits are conducted for configuration management and results are reported. Compliance CM responsibilities are assigned and approved. A build plan exists. SCM standards are applied for changes to the software. Different versions of the software exist. December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved. 27
Software Quality Assurance coverage Software Quality Assurance Key Process Area No. Commitment Co 1 The project follows organizational policy. Compliance Quality assurance function is in place. Program manager, engineering group, and other affected groups have a reporting channel to senior management. Senior management reviews the compliance activities weekly. December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved. 28
Software Quality Assurance Coverage Software Quality Assurance Key Process Area No. Ability AB 1 SQA Group exists. Compliance Program manager owns SQA function. AB 2 Adequate resources and Reviews are an integral part funding are provided to of the engineering conduct reviews. organization’s processes. Program managers are chartered to ensure that checkpoint reviews are planned and conducted. AB 3 Individuals are trained to Program managers are perform quality assurance trained as well as coached activities. through RMT process. December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved. 29
Software Quality Assurance Coverage Software Quality Assurance Key Process Area No. Activities Compliance Act 1 SQA activities are planned. Checkpoint reviews are planned in the software project plan. Act 2 SQA activities are performed in accordance with the plan. Act 3 Software Development Plan is reviewed. Checkpoints reviews are conducted and affected groups are involved for the reviews. Program manager works closely with the software development project manager. Ø Italic compliance is in progress December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved. 30
Software Quality Assurance Coverage Software Quality Assurance Key Process Area No. Activities Act 4 Software Engineering activities are reviewed for compliance. Compliance Covered in the verification of other Key Process Areas. Act 5 Software work products verified for compliance. Covered in the compliance metrics. Act 6 The results of review Communicated weekly in activities are periodically the leadership reported to affected groups. management meetings. December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved. 31
Software Quality Assurance Coverage Software Quality Assurance Key Process Area No. Measurement Me 1 Measurements are made and used to determine the cost and schedule status of the SQA activities. December 9, 2002 Compliance No measurements are made because the roles and responsibilities of program managers include SQA. Copyright © 2002 EMC Corporation. All Rights Reserved. 32
Software Quality Assurance Coverage Software Quality Assurance Key Process Area No. Verification Compliance Ve 1 SQA activities are Weekly leadership reviewed periodically with meetings cover senior management. compliance activities. Ve 2 SQA activities are reviewed with the project manager. December 9, 2002 Program manager reviews work products with the project manager. Copyright © 2002 EMC Corporation. All Rights Reserved. 33
Software Quality Assurance Coverage Software Product Engineering Key Process Area KPA Verification 3 SPE The activities and work product reviews and/or Ve 3 audits are conducted for software product engineering and results are reported. December 9, 2002 Compliance Readiness and completion criteria for each software engineering task are satisfied. Required testing is performed. Tests satisfy acceptance criteria, as documented in the software test plan. Copyright © 2002 EMC Corporation. All Rights Reserved. 34
Software Quality Assurance Coverage Software Product Engineering Key Process Area KPA Verification 3 SPE The activities and work product reviews and/or Ve 3 audits are conducted for software product engineering and results are reported. Compliance Problems and defects detected are documented, tracked, and addressed. Tracing of the allocated requirements through the software requirements, design, code, and test cases is performed. Ø Italic compliance is in progress December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved. 35
Software Quality Assurance Coverage Integrated Software Management Key Process Area KPA Verification 3 ISM The activities and work product reviews and/or Ve 3 audits are conducted for integrated software management and results are reported. Compliance Software development plan is developed using organizational standards. Organizational process database to support for planning, estimating, and tracking is not available. Ø Italic compliance is in progress December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved. 36
Software Quality Assurance Coverage Peer Review Key Process Area KPA Verification 1 PR The activities and work products reviews and/or Ve 3 audits are conducted for Peer Reviews, and results are reported. Compliance Peer review process is developed and peer reviews are planned. Peer review process is being implemented. Ø Italic compliance is in progress December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved. 37
Challenges n Resistance to change n Need for individual group processes n Compliance of individual group processes n Tracking and oversight n Too many processes n Impact on business n Improving processes December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved. 38
Lessons Learned n Program managers are key part of the solution n Executive forums helped program managers to achieve EMC goals n Faster escalation, resolve issues, assign resources n Best practices shared among program managers n Quicker feedback to program management n Sponsorship and coaching December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved. 39
Summary “Using organizational and program management standards, program managers drive end-to-end product and service solutions by breaking down organizational barriers, anticipating problems before they occur, and speeding the delivery of products and service solutions. ” — Peter Conway, V. P. Software Product Management, EMC December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved. 40
“The best way to predict the future is to create it. ” Closing Slide Peter Drucker December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved. 41