CTIS 494 Software Quality Assurance by Dr Ozlem

CTIS 494 Software Quality Assurance by Dr. Ozlem Albayrak 2007 Fall by Ozlem Albayrak

menu CTIS 494 Outline (1) Week Subject 17. 09 Introduction to Software Quality Assurance 24. 09 Software Inspections and Testing 01. 10 Capability Maturity Model 08. 10 ISO 9000 and SPICE (1 hour missing) 15. 10 Software Quality Metrics 22. 10 Seven Basic (B 7) Tools of Quality 29. 10 The Seven MP Tools 05. 11 MIDTERM (30%) by Ozlem Albayrak

CTIS 494 Outline (2) 12. 11 5 S for Intelligent Housekeeping in Software Development 19. 11 Software QFD and Voice of the Customer 26. 11 Risk Assessment and Failure Modes and Effects Analysis 03. 12 Quality Measures and Statistical Methods for Trustworthy Software 10. 12 Verification and Validation (PROJECT REPORTs Submission 15%) 17. 12 Testing and Evaluation (1 hour missing) 24. 12 Integration, Extension and Maintenance for Trustworthiness (PROJECT PRESENTATIONs 20%) 31. 12 Review (2 hours missing) 7. 12 FINAL (35 %) by Ozlem Albayrak

menu Today. . . Week Subject 17. 09 Introduction to Software Quality Assurance to Software 24. 09 Software Inspections and Testing 01. 10 Capability Maturity Model 08. 10 ISO 9000 and SPICE (1 hour missing) 15. 10 Software Quality Metrics 22. 10 Seven Basic (B 7) Tools of Quality 29. 10 The Seven MP Tools 05. 11 MIDTERM (30%) by Ozlem Albayrak

Review l l l Introduction to course elements What is quality? Differences between service and product by Ozlem Albayrak

CTIS 494 PROJECT DETAILS l l l 35% of the total grade is from the projects (same weight as final) The course project is a group project each of which will be developed by 4 students. Project subjects will be announced by the instructor. Project requires groups to submit a report and deliver a in-class presentation. Reports will be submitted 2 weeks before the presentations. by Ozlem Albayrak

Project Report specifications l l Project reports will be prepared as web pages that can be accessed by all the students and the instructor. The groups are free to design their web pages format, but the generated pages should be readable and understandable. Navigation among the pages should be carefully organized. The quality of the web pages will be determined by is content, format as well as its timely distribution. Late submissions will be heavily penalized. by Ozlem Albayrak

Project Presentations l l l Project study will be presented in classroom. Following the presentations there will be a question-answer session during which the instructor and the other students are allowed to ask questions. Those students who prepare good questions will be awarded by the instructor. Fluency of the presentation is as important as its content, thus the groups are urged to prepare their presentations in advance to deliver a high quality presentation. Duration of the presentations will be announced by the instructor. Using time effectively is an important element. by Ozlem Albayrak

Project Evaluation l l 20% of the project grade is allocated for project presentation and 15% is for the project report. 15% includes evaluation of the following items: – – – l Content Format and ease and fluency of navigation Timely delivery 20% includes evaluation of the following items: – – Content Fluency of the presentation Obedience to time limits Answers to questions by Ozlem Albayrak

Quality exercise l l l Define high quality toilette paper? Single (5 min) In groups of two (5 min) Reviews. . . . Hw # : form a group of four One address for the class and registration to the group. . Do not wait for invitation register by Ozlem Albayrak

What is quality? by Ozlem Albayrak

Different Perspectives on Quality l There are many different definitions and dimensions of Quality – – – Transcendent Product based User based Manufacturing based Value based by Ozlem Albayrak

Garvin’s 8 quality dimensions Performance Durability Serviceability Features Reliability Aesthetics Conformance Percieved quality by Ozlem Albayrak

Performance l Efficiency with which a product achieves its intended purpose by Ozlem Albayrak

Features l Attributes of a product that supplement the product basic performance by Ozlem Albayrak

Reliability l Propensity (eğilim) for a product to perform consistently over its useful design life by Ozlem Albayrak

Conformance l Is a particular dimension of a product is within the allowable range of tolerance of the specification, the product conforms. by Ozlem Albayrak

Durability l The degree to which a product tolerates stress or trauma without failing by Ozlem Albayrak

Serviceability l Ease of repair for a product by Ozlem Albayrak

Aesthetics l Subjective sensory characteristics such as taste, feel, sound, look and smell. by Ozlem Albayrak

Perceived Quality l As the customer perceives the quality. QUALITY by Ozlem Albayrak

Service Quality dimensions l l More difficult to define than to define product quality Tangibles Service reliability Assurance by Ozlem Albayrak

Differing Functional Perspectives on Quality l l l Engineering Operations Strategic Management Marketing Financial Human Resources by Ozlem Albayrak

Engineering l Engineering is an applied science. – – l Product Design Engineering Concurrent Engineering (simultaneous performance) Statistics + technical orientation by Ozlem Albayrak

Design Life Cycle Idea Generation Preliminary Design Iterations Prototype Development Final Definition Product Design Evaluation Implementation by Ozlem Albayrak

Operations l l Focus was initially on statistical process control Uses system view underlying modern quality management thinking by Ozlem Albayrak

Strategic Management l l l Refers to the planning processes Long term orientation Goal: to achieve sustainable competititve advantage by Ozlem Albayrak

Generic Strategic Planning Process (1) Firm Mission and Goals External analysis Strategic Options Business level strategy Corporate level strategy Operational Subplans by Ozlem Albayrak Internal analysis

Generic Strategic Planning Process (2) Operational Subplans Organizational Design Conflict Politics and Change Strategic Alignment between Structure and Goals by Ozlem Albayrak Organizational Reward System

Marketing l Customer satisfaction by Ozlem Albayrak

Financial l Risk of investment vs. Potential rewards by Ozlem Albayrak

Deming Value Chain Improve quality Cost decrease Productivity Improves Capture the market Stay in business Provide Jobs and more jobs by Ozlem Albayrak

Human Resources l l l Employee empowerment Job analysis Total quality human resources management by Ozlem Albayrak

The Three Spheres of Quality Management Quality Assurance Quality Control by Ozlem Albayrak

Quality Control l Quality Control Scientific method includes: – – – Analysis ( a process is broken into fundamental pieces) Relation (relationships between parts) Generalization (how interrelationships apply to the larger phenomenon of quality being studied) by Ozlem Albayrak

Quality Control Activities l l l Monitoring process capability and stability Measuring process performance Reducing process variability Optimizing processes to nominal measures Performing acceptance sampling Developing and maintaining control charts by Ozlem Albayrak

Quality Assurance l Activities associated with guaranteering the quality of a product or service. by Ozlem Albayrak

Quality Assurance Activities l l l l Failure mode and effect analysis Concurrent Engineering Experimental Design Process improvement Design team formation management Off-line experimentation Reliability/durability product testing by Ozlem Albayrak

Quality Management l Quality Management processes that overarch and tie toegther the control and assurance activities make up quality management. by Ozlem Albayrak

Quality Management Activities l l l Planning for quality improvement Creating a quality organizational culture Providing leadership and support Providing training and re-training Designing a system to reinforce quality ideals Facilitating organizational communication by Ozlem Albayrak

Other perspectives l l Value Added Perspective Cultural Perspectives by Ozlem Albayrak

Contingency Theory l There is no theory or method for operating a business that can be applied in all instances. This provides flexibility to managers in pursuing quality. by Ozlem Albayrak

Summary l l Different perspectives on quality management Quality sphere is composed of: – – – Quality Control Quality Assurance Quality Management by Ozlem Albayrak

Quality Example l l l Define high quality toilette paper? Single (5 min) In groups of two (5 min) Reviews. . . . One address for the class and registration to the group. . Do not wait for invitation register by Ozlem Albayrak

CTIS 494 Software Quality Assurance by Dr. Ozlem Albayrak 2007 Fall by Ozlem Albayrak

A Practical Approach to Software Quality Book by G. O’Regan Slides by Dr. Ozlem Albayrak 2006 Spring by Ozlem Albayrak

Outline l l l Software Engineering Challenge History of Software Failures Background to Software Quality History of Quality Software Engineering Modern Software Quality Management by Ozlem Albayrak

Next Week Assignment. . . l Your group will – Write a C/C++ code to do addition. . – Please refer to your first assignments by Ozlem Albayrak

Software Engineering Challenge l Challenge is to deliver: High quality software on time to the customer l Focus on customer satisfaction by Ozlem Albayrak

Software Engineering Challenge l l Overestimation How to estimate cost and schedule accurately is a big challenge Risk management is key Example: Taurus project at London Stock Exchange was 13, 200% late and costed a lot, and abondened by Ozlem Albayrak

Software Engineering Challenge l l l Changing requirements In civil engineering there also problems Software eng. less mature than civil eng. by Ozlem Albayrak

History of Software Failures l Y 2 K problem – – – l Find analyze all code with a Y 2 K impact Plan and make necessary changes Cost to UK is $32 billion!!! Software quality needs to be a key driving factor by Ozlem Albayrak

Introduction to Software Quality(2) l History of Quality – – – Shewart Deming Juran Crosby Other Quality Gurus by Ozlem Albayrak

Background to Software Quality l l l Customers have high expectations In the market there are many high quality companies Late delivery leads to extra cost by Ozlem Albayrak

Background to Software Quality (2) l What is software quality – “Conformance to the requirements” Crosby – “Fitness for use” Juran – ISO 9126 standard for information technology by Ozlem Albayrak

ISO 9126 – Quality Characteristics l l l Functionality: extent to which functions are avaliable Reliability: extent to which software reliable Usability: users say easy to use Efficiency Maintainability: easy to maintain Portability: easy to transfer do different environments by Ozlem Albayrak

Total Quality Management l Modern approach to quality management is TQM – – – Customer focus Process improvement Development of quality culture by Ozlem Albayrak

Software Quality Control l l Extensive inspections and testing Formal reviews Aim is to identify defects White box and black box tests Unit, function, subsystem, performance, integration, acceptance tests by Ozlem Albayrak

History of Quality l Shewhart – – Statistian at AT&T, 1920 s PDCA Model l l Plan Do Check Act by Ozlem Albayrak

History of Quality (2) l Deming – – – 1945 s, Japan 14 step programme Consistent purpose and direction in the organization is more important then everyone doing his best alone by Ozlem Albayrak

History of Quality (3) l Juran – – – l Another giant in the quality movement Fitness for use 10 step programme Crosby – Has influenced CMM (Capability Maturity Model) by Ozlem Albayrak

Other Quality Gurus l l l Taguchi Ishikawa Feigenbaum by Ozlem Albayrak

Introduction to Software Quality(3) l Software Engineering – – – Requirements Analysis Specification Design l l – Architectural Functional Object-oriented reuse Verification of Design Implementation by Ozlem Albayrak

Introduction to Software Quality(4) – Testing l l l – Unit Integration Sub-system System Performance Acceptance Maintenance by Ozlem Albayrak

Maintenance l l 1 st Malcom Baldridge Quality award to Motorola Steps of Six Sigma: – – – Identify product or service Identify customer and requirements Identify needs to satisfy customer Define process of doing work Eliminate waste Ensure continious improvement by: l Measuring, analyzing and controlling improved process by Ozlem Albayrak

Modern Software Quality Management l l Software Inspections Software Testing Software Quality Assurance Problem Solving – – – Fishbone diagrams Histograms Pareto Chart - Trend Graph - Scatter Graphs - Failure Mode Effect Analysis by Ozlem Albayrak

Modern Software Quality Management(2) l l l Modeling Cost of Quality Metrics Customer Satisfation Assessments Total Quality Management by Ozlem Albayrak

Metrics l Measurement is important part of science and engineering l Goal Question Metric by Basili l Data, Charts, Analysis, Trends, Action by Ozlem Albayrak

Customer Satisfaction Customer Saticfaction Surveys Customer Saticfaction Measurement Improvement Planning Execute Plan by Ozlem Albayrak

Formal Methods l Advantage: – – Utilization of mathematical models are being used Lead to more robust(gürbüz) systems by Ozlem Albayrak

Modern Software Quality Management l l Future of Software Quality Organizational Culture and Change Formal Methods Quality and the Web by Ozlem Albayrak

Quality and the WEB Accesss from everywhere in the world l May different browsers l Usability, look and feel important l Performance as key concept l Security l Many transactions at any time l Avaliability (24 x 365) l Relaibility JAD/RAD with spiral development suggested l by Ozlem Albayrak

End of Musical notes Check this web site: http: //sw-assurance. gsfc. nasa. gov by Ozlem Albayrak

TO DO by Next Week: l l Review Next class: Search web for software quality assurance Come to class to discuss by Ozlem Albayrak

Software Quality Assurance S. Thomas Foster, Jr. Boise State University Power. Point prepared by Dave Magee University of Kentucky Lexington Community College Stephen Mc. Ilroy BT Retail Senior Systems Architect modified by O. A. by Ozlem Albayrak

Software Quality Assurance l What is it? l Why do we need it? l How do we achieve it? by Ozlem Albayrak

What is Quality Assurance ? l Quality = – – – l l . . . meeting the customer’s requirements, . . . at the agreed cost, . . . within the agreed timescales. Quality = “Fitness for purpose” Quality = Customer satisfaction ! Quality Assurance is what we do to achieve this by Ozlem Albayrak

“Success” v “Failure” Quality Assurance helps us avoid failure ! On Content Failure by Ozlem Albayrak et Success dg bu tim e n thi Wi Failure

Software failures some examples l Taurus (London Stock Exchange) – l London Ambulance Service – l faults led to reversion to manual processes Ariane 5 – l 11 years late & 13, 200% over budget coding defect in module reused from Ariane 4 AT&T switching system – coding defect in three-line “fix” by Ozlem Albayrak

Software failures the statistics by Ozlem Albayrak Source: The Standish Group

Software failures cost overruns l Average cost for “failed” projects is 189% of the original estimate, i. e. almost double. by Ozlem Albayrak Source: The Standish Group

Software failures time overruns l Average time for “failed” projects is 222% of original estimate, i. e. more than double. by Ozlem Albayrak Source: The Standish Group

Software failures content deficiencies l On average “challenged” projects deliver 61% of specified functions by Ozlem Albayrak Source: The Standish Group

Software failures are things getting any better? l Only 41% of IT managers think there are fewer failures today than 5 years ago by Ozlem Albayrak Source: The Standish Group

Why is it so hard to succeed? by Ozlem Albayrak

Why is it so hard to succeed? Software engineering is difficult – – – problems are complex & difficult to define solutions are complex & difficult to specify high degree of precision often required progress is difficult to monitor rapid pace of technological and business change by Ozlem Albayrak

Why is it so important to succeed? l l l Huge demand for software Shortage of skilled people Limited financial resources The need for correctness Limited windows of opportunity We can’t afford the cost of failure by Ozlem Albayrak

How do we achieve QA? l l By adopting a disciplined, professional approach - engineering is predictable and repeatable Quality Management System – – l Documented processes & Best Practice ISO 9001 & Tick. IT registration Continuous Improvement – – New methods (e. g. DSDM, UML) & tools Better & more focussed Training by Ozlem Albayrak

The Quality Management System l Defines the way we work – – – l Procedures and guidelines – l focuses on what rather than how flexible broadly applicable mandatory elements and recommendations Mix of best practice & common sense by Ozlem Albayrak

Project control l l l Project initiation Project closure Estimation Risks & issues Project & Quality plans Schedule plans Progress monitoring & reporting by Ozlem Albayrak

Quality control l l Traceability Quality Control Reviews Approval process Validation, Verification & Testing by Ozlem Albayrak

Summary l l What is Quality Assurance? Why is it so difficult to succeed? Why is it so important to succeed? How do we achieve QA? by Ozlem Albayrak

l The goal of software inspection and test is to reduce the expected cost of software failure over the life of a product. by Ozlem Albayrak

Popular Definitions of Software Quality l l l l Software that is developed right the first time Minimization of fixes between releases Measured loss imported to society after delivery User satisfaction Fitness for use Performance/ expectations Measurable properties of software Conformance to requirements by Ozlem Albayrak

Quantitative Definitions l l l Availability Reliability Defect rate Problems per month Mean time between failure Loss to society (Taguchi) by Ozlem Albayrak

What is a good software? l l l Written according to a coding standard Design must be high quality Succint and useful documentation Comprehensive suite of regression tests must accompany the software Ref: H. Erdogmus, IEEE Software, 2007 by Ozlem Albayrak

Software Inspections and Testing l l l Inspection plays a key role in building software Testing plays a key role in verifiying software There are different ways to software inspections: – – Formal Informal by Ozlem Albayrak

Formal Inspections l l Fagan inspection methodology Gilb methodology Roles Author Inspector Tester Moderator by Ozlem Albayrak

Informal Inspection l l Walkthrough of document or code by individual other than author Deliverable is reviewed informally by Ozlem Albayrak

Fagan Inspections l l l l Planning Overview Preparation Inspection Process improvement Re-work Follow-up by Ozlem Albayrak

Economic Benefits of Sofware Inspections l Positive impacts on: – Productivity Quality Time to market Customer satisfaction – IBM Space Shuttle %85 inspections, %15 testing – – – by Ozlem Albayrak

Benefits of Software Inspections l l l Good for educating people about standards and procedures Cost of defect increases as software is being developed Cost of quality estimation varies between 20%40% by Ozlem Albayrak

Software Inspection Methodology l l Fagan, Gilb Typical Roles Author – Inspector Also – Moderator – Tester – Reader – by Ozlem Albayrak

E-mail/Fax reviews l l No physical meeting reviews Depending on the organization this type of reviews may work by Ozlem Albayrak

Semi Formal Review Meeting l l l Actual review meeting If review leader is other than author may work well Phases are: – – – Planning Distribution Informal meeting (optional) Preparation Review meeting Post review (corrections are verified) by Ozlem Albayrak

Meeting Minutes Item Under Review_____ Review Reference_____ Date__________ Document Name _____ Version No________ Author_________ No of Previous Reviews____ Reviewers_________________________ NOTES________________________ by Ozlem Albayrak

Actions from the review Issue No Raised by Page/ DSection/Line by Ozlem Albayrak Description Action

Summary block in inspections l l l # major defects discovered # minor defects discovered Author estimate of rework time # hours preparation Amount of reviewed (pages/LOC) by Ozlem Albayrak

Fagan Inspection Methodology l Objective of inspection is – – to identify as many defects as possible and to confirm the correctness of a particular deliverable. by Ozlem Albayrak

Fagan Inspection Process l l l l Moderator – Planning/overview Moderator, author – Overview Inspectiors – Preparation Inspectors – Inspection Inspectors – Analysis, improvement Author – Rework Moderator, author – Follow-up by Ozlem Albayrak

Fagan Inspection Guidelines Activity Area Amount/hr Max/hr Preparation Requirements Design Code Test Plans 4 pages 100 LOC 4 pages 6 pages 125 LOC 6 pages Inspection time Requirements Design Code Test Plans 4 pages 100 LOC 4 pages 6 pages 125 LOC 6 pages by Ozlem Albayrak

Relaxed Fagan Inspection Guidelines Activity Area Amount/hr Max/hr Preparation Requirements Design Code Test Plans 4 -8 pages 100 -130 LOC 4 -8 pages 6 pages 125 LOC 6 pages Inspection time Requirements Design Code Test Plans by Ozlem Albayrak

Fagan Inspection Methodology l l l Roles Entry Criteria Preparation Inspection Meeting Exit Criteria by Ozlem Albayrak

Inspection Exit Criteria (1) Inspection Type Mandatory/ Optional Exit Criteria Requirements M 1. Requirements satisfy the customers needs 2. All requirements defects are corrected Design Ins. M 1. Design satisfies the requirements 2. All defects in the inspection are corrected 3. Design satisfies the design standards Code Ins. M 1. Code satisfies design and requirements 2. Code follows coding standards 3. Code compiles correctly 4. All defects from inspection removed by Ozlem Albayrak

Inspection Exit Criteria (2) Inspection Type Mandatory/ Optional Exit Criteria Test Plan M 1. Test plan sufficient to test requirements 2. Test plan follows test standards 3. All defects from inspection removed by Ozlem Albayrak

Issue Severity l l Major Minor Process Improvement Investigate by Ozlem Albayrak

Defect Types Code Design Requirements Logic Design Requirements Maintainability Interface Data Usage Performance Standards Code Comments Usability Requirements Logic System Interface Portability Reliability Maintainability Error handling Other Product objectives Documentation Hardware interface Competition Analysis Function Software Interface Performance Reliability Spelling by Ozlem Albayrak

Software Testing l Objective is – – l to verify the correctness of the software system and to identify defects that are present. Testing effort is often complicated by real world issues, such as late delivery of software. by Ozlem Albayrak

Test Planning l l l Identify scope of testing to be done Estimates of time, resources, people, hardware, software and tools Provide resources needed Provide test environment Assign people to tasks Define schedule Identify risks and contingency plan Track progress and take corrective action Provide regular test status of passed, blocked, failed tests Re-plan if scope of the project changes Conduct post mortem to learn any lessons by Ozlem Albayrak

Sample Test Plan Activity Resource Name Start End Comments Review requirements Test Team 15. 02. 2007 05. 03. 2007 Complete by Ozlem Albayrak

Steps in Test Plan l l l Review requirements Overall test plan and review System test plan and review Performance test plan and review Usability test plan and review Regression plan and review Test environment setup System testing and re-test defects Performance testing and re-test defects Usability testing Regression testing Test reporting by Ozlem Albayrak

Test Process l The quality of testing is dependent on the maturity of the test process and a good test process include: • Test planning and risk management • Dedicated test environment and test tools • Test case definition • Test automation • Formality in handover to test department by Ozlem Albayrak • Test execution • Test result analysis • Test reporting • Measurements of test effectiveness • Post mortem and test process improvement

Test Case l A Test Case should include: – – – Purpose of test case Setup required to execute the test case Inputs to test case The test procedure Expected outputs or results by Ozlem Albayrak

Test Management Tools Open source Test Management Tools: l http: //www. opensourcetesting. org/testmgt. php l http: //www. technologyevaluation. com/request/t esttool_softeval. asp? se=google_testtools&gcli d=CJvgv 8 Ca 3 o 4 CFQxa. Xgod. Iy-FPA by Ozlem Albayrak

Next week: l Bring your first assignment to class – l Demo’s will be done Come to class prepared: – – What is CMM/CMMI? List of Turkish software companies having CMM(I) by Ozlem Albayrak