Agile Software Development Mark Woyna October 2006 Agile

Agile Software Development Mark Woyna October 2006 Agile Software Development

Overview l l Why Agile? What is Agile? Agile Manifesto Overview of Scrum and XP Agile Software Development 2

Business View l What has the business always wanted? – – – – Better quality Predicable deliveries High customer satisfaction Higher development productivity Lower costs Mind-reading High ROI “Why do I outsource? So I can get the same wrong software for less. ” – Overheard at an outsourcing seminar Agile Software Development 3

Standish Chaos Report l 1994 – 16% of projects on time, on budget – 31% cancelled before completion – 53% challenged (avg. 180% over budget) l l > 25% complete with < 50% original functionality 2004 – 29% of projects on time, on budget – 18% cancelled before completion – 53% challenged (avg. 45% over budget) Agile Software Development 4

Project Complexity Far from Agreement Requirements Anarchy Close to Agreement Complex C om pl ic at ed Simple Close to Certainty Technology Far from Certainty Source: Strategic Management and Organizational Dynamics by Ralph Stacey in Agile Software Development with Scrum by Ken Schwaber and Mike Beedle. Agile Software Development 5

Cost of Change B. Boehm (Software Engineering Economics – 1981) Cost of Change l Time Agile Software Development 6

Ramifications of Cost of Change l Because the cost of discovering errors late is high, we are willing to make a very large investment in upfront in analysis and design – Big Design Up Front (BDUF) l Thus, waterfall and BDUF are the conventional wisdom Agile Software Development 7

Has the Cost of Changed? l l l It doesn’t take a day to compile a program Computers are orders of magnitude faster and cheaper We have CASE tools, DBMSs, CM tools, IDEs Object-oriented languages and principles make software much easier to change Test-driven development and continuous integration allow us to catch errors quickly Agile Software Development 8

Cost of Change - Flattened K. Beck (Extreme Programming Explained – 1999) Cost of Change l Time Agile Software Development 9

Flattening the Cost of Change Curve l If the cost of change curve can be flattened – Upfront work become a liability l We pay for speculative work, some of which will be wrong – Ambiguity or volatility is a reason to delay l Don’t plan for something that never happens – It is cost effective to delay all decisions until the last possible moment l We only pay for what we use Agile Software Development 10

Cone of Uncertainty l S. Mc. Connell - Software Project Survival Guide (1998) 4. 0 x Estimate 2. 0 x Final Actual x 0. 5 x 0. 25 x Time Agile Software Development 11

Time and Uncertainty l l If you implement a feature today, and it turns out to be not valuable, you lose money and opportunity If you can wait, then the risk is reduced over time Time answers questions and removes uncertainty We need a process that exploits a flat changecost curve, and reduces uncertainty Agile Software Development 12

Why we use Processes l We are afraid that – – – The project will produce the wrong product The project will produce a product with low quality The project will be late We’ll have to work 80 hour weeks We’ll have to break commitments We won’t be having fun Agile Software Development 13

Project Management in a Nutshell l l The delivery date is frozen The specification is never frozen Agile Software Development 14

Software Project Management l Predictive (Defined, Plan/Forecast-driven) – – – l Define scope Develop Work Breakdown Structure (WBS) Manage cost and schedule Control scope Deploy when finished Adaptive (Empirical) – – Define minimum useful feature set Develop, integrate, and test Deploy and learn Repeat Agile Software Development 15

Predictive/Sequential Requirements Design Implementation Test Maintenance Agile Software Development 16

Business Value (Predictive) Time Agile Software Development 17

Perspective “In the history of science…simplistic but inferior ideas first hold the dominant position, even in the absence of results” Craig Larman “The definition of insanity is doing the same thing over and expecting different results. ” Benjamin Franklin Agile Software Development 18

History of the Waterfall Process l l l “Managing the Development of Large Software Systems”, Dr. Winston Royce, Proceedings, IEEE WESCON, August 1970 Outlined the basic phases of analysis, design, coding, testing, and operations Recommended that 2 passes be performed if the product under development was new – Iterative development? Agile Software Development 19

Waterfall Process - Epilogue “He was always a proponent of iterative, incremental, evolutionary development. His paper described the waterfall as the simplest description, but that it would not work for all but the most straightforward projects. The rest of his paper describes [iterative practices] within the context of 60 s/70 s government-contracting models. ” Walker Royce, son of Winston Royce Agile Software Development 20

Adaptive/Iterative Highest Priority Features 30 days Potentially Shippable Product Increment Agile Software Development 21

Business Value (Adaptive) Time Agile Software Development 22

Taylorism l “The Principles of Scientific Management” (1911) – – – – Hierarchical leadership Fixed, not fluid Divide manufacturing and office work Work became specialized Suited for unskilled labor Product focused, not customer focused Manufacturing-like repeatable process Agile Software Development 23

Terminology l “Traditional” methodologies – Traditions change l “Plan-driven” – Agile methods employ continuous planning – Forecast-driven may be a better term l “Heavyweight” – Implies agile is “lightweight” l “Rigorous” – Implies agile is not rigorous Agile Software Development 24

Project Variables l Scope – Features ill-defined, or changing l Cost – Cost overruns are the norm – People are almost always non-linear l Time – “Adding people to a project just makes it later” [Brooks] l Quality – Code-and-fix, throw it over the wall – Low internal quality kills productivity Agile Software Development 25

Software Development vs. Engineering l Engineering methodologies are inspired by disciplines such as civil or mechanical engineering – Emphasis on planning and design before construction – Design requires creative people, whereas construction can use people with lower skills – Bridge construction (10 % design, 90% construction) l The engineering model may be the wrong model – Software construction (90% design, 10% construction) – Software development is closer to new product development Agile Software Development 26

Engineering Project Management l l The software field likes to believe that the engineering field has perfected plan-based PM The engineering field has plenty of examples of big cost and schedule overruns – Boston’s “Big Dig” ($1 B+) – Navy’s Virginia class submarine ($1 B+ per submarine for first 4 delivered) – Chicago’s O’Hare Expansion ($400 M+) – Google “construction delays cost overruns” l Reasons for overruns – – Weather Rework due to faulty/changed design New technology, new product development Unexpected discoveries Agile Software Development 27

Software Development vs. Manufacturing l Engineering methodologies attempt to treat development as an Tayloristic assembly line – Specialization – Repeatability l l Custom software usually results in a single, unique system, not multiple identical systems What is software construction/manufacturing? – Coding? (UML is the design specification) l What about Model Driven Architecture (MDA)? – Compilation? (Source code is the design specification) – Media replication and packaging? l The manufacturing model may be the wrong model Agile Software Development 28

Software - Design and Construction l l Software construction is cheap Software design is expensive, and requires creative and talented people – Never ending search for the magic-bullet CASE tool that will make everyone a software developer l l Creative processes are not easily planned, and predictability may be an impossible target We should be wary of the traditional engineering/manufacturing metaphor for building software, as it’s a different kind of activity, and requires a different process Agile Software Development 29

Lean Production l Taiichi Ohno – The Toyota Production System: Beyond Large Scale Production (1978) l James Womack and Daniel Jones – The Machine That Change the World: The Story of Lean Production (1990) l Mary and Tom Poppendieck – Lean Software Development: An Agile Toolkit (2003) Agile Software Development 30

Lean Principles l l l l Eliminate Waste Amplify learning Decide as late as possible Deliver as fast as possible Empower the team Build in integrity See the whole Agile Software Development 31

Seven Wastes of Manufacturing l l l l Overproduction Inventory Extra processing steps Motion Defects Waiting Transportation Agile Software Development 32

Seven Wastes of Software Development l Overproduction = Extra features – Develop only for today’s features l Inventory = Requirements – Requirements are only detailed for the current iteration l Extra processing steps = Extra steps – Code directly from stories, get verbal clarification from customer l Motion = Finding information – Have everyone in the same room, including the customer Agile Software Development 33

Seven Wastes of Software Development (2) l Defects = Defects not caught by tests – Test first, both customer (acceptance) and develop (unit) tests l Waiting = Waiting, including customers – Deliver in small increments l Transportation = Handoffs – Developers work directly with customers, and perform all necessary steps Agile Software Development 34

Eliminating Software Feature “Waste” l The biggest cost of predictive software development is extra features (Standish Group) – – – 7% always used 13% often used 16% sometimes used 19% rarely used 45% never used Agile Software Development 35

Scope Creep How-to l Ask customers what they want – They may not know, or think they know l Reward them for thinking of everything – BUFR l Punish them for changing their minds later – Change requests l 80% of the value typically comes from 20% of the features Agile Software Development 36

Managing Scope l Write less code! – Develop and deploy the features that provide the highest business value, and thus maximize ROI – Customer decides what to develop first – Customer can change their mind at any point – Work is stopped when l l l Out of time Out of money Customer is satisfied with current business value Agile Software Development 37

Agile Manifesto – www. agilemanifesto. org l We are uncovering better ways of developing software by doing it and helping others doing it. Through this we have come to value: – Individuals and interactions over processes and tools – Working software over comprehensive documentation – Customer collaboration over contract negotiation – Responding to change over following a plan That is, while there is value in the items on the right, we value items on the left more. Agile Software Development 38

Agile Principles l l l Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. Agile Software Development 39

Agile Principles (2) l l l Business people and developers must work together daily throughout the project. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. Agile Software Development 40

Agile Principles (3) l l l Working software is the primary measure of progress. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely. Continuous attention to technical excellence and good design enhances agility. Agile Software Development 41

Agile Principles (4) l l l Simplicity--the art of maximizing the amount of work not done--is essential. The best architectures, requirements, and designs emerge from self-organizing teams. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly. Agile Software Development 42

Agile Core Strategies l l Iterative and incremental development Adaptive project management – Continuous planning l l Collaborative, “whole team approach” Common shared vision, goals, and principles Agile Software Development 43

Agile Practices l Knowledge sharing – Verbal “documentation” – Written documentation when appropriate l Fast feedback – – l Onsite customer Continuous integration Short iterations Retrospectives Emphasis on testing – Test-driven development (TDD) – Tests express unambiguous requirements, and provide exit criteria Agile Software Development 44

Agile Practices (2) l Emerging design and refactoring – KISS principle – YAGNI l Pairing – Continuous review and knowledge sharing l Continuous process improvement – Retrospectives reflect on what’s working, and what’s not working Agile Software Development 45

Practices are Tools, not Laws l Pairing helps, so we do it. But we don’t have to do it all the time Test first helps, so we do it Collective ownership helps, so we do it l But we are not constrained l l Agile Software Development 46

Agile vs. Predictive Methods l Agile – – – – Human centric Tacit knowledge Code-centric Face-to-face Generalists Plan and correct Customer focused l Predictive – – – – Process centric Explicit knowledge Documentation-centric Over-the-wall Specialists Plan and control Contract focused Bureaucratic Agile Software Development 47

Axioms l Agile – We are creators – Reality is perceived – Change is natural, and encouraged l Predictive – Humans are resources – Reality can be legislated – Change is bad Agile Software Development 48

Disciples l Agile – Empower the team – Amplify learning – Eliminate waste l Predictive – Enforce the process – Avoid experimentation – Eliminate variance Agile Software Development 49

Agile Development as a Management Policy l l Agile development is more a management policy, than a set of development techniques Use of incremental development, access to user expertise, periodic delivery, location of staff, etc. are all management policies Agile Software Development 50

Agile is Disruptive “. . . there is nothing more difficult to carry out, nor more doubtful of success, nor more dangerous to handle, than to initiate a new order of things. For the reformer has enemies in all those who profit by the old order, and only lukewarm defenders in all those who would profit by the new order, this lukewarmness arising partly from fear of their adversaries. . . and partly from the incredulity of mankind, who do not truly believe in anything new until they have had actual experience of it. ” Niccolo Machiavelli, The Prince Agile Software Development 51

Agile will take Time "A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it. ” Max Planck Agile Software Development 52

Planning l Agile – Only rough plans beyond a few months – Low precision – Long term plans are expected to change – Detailed plans in short horizons – 2 weeks to 2 months l Predictive – Figure out everything that needs to be done before starting – Figure out best way to do everything – Long planning horizon of a year or more – Deviations from the plan are a problem Agile Software Development 53

The Planning Trap l A plan is not a prediction of the future – Things will change – “Plan-driven” vs. “Forecast-driven” l Don’t use plans to measure virtue – – People want to say things are going well Will hide early signs of trouble Plan turns into an illusion Deviations from the plan are not errors “In preparing for battle I have always found that plans are useless, but planning is indispensable. ” - Dwight Eisenhower Agile Software Development 54

Balance of Power l Business people make business decisions – Dates – Scope – Priority l Technical people make technical decisions – Estimates – Risk assessment Agile Software Development 55

Customer Bill of Rights l l l You have the right to an overall plan, to know what can be accomplished, when, and at what cost. You have the right to get the most possible value out of every programming week. You have the right to see progress in a running system, proven to work by passing repeatable tests that you specify. You have the right to change your mind, to substitute functionality, and to change priorities without paying exorbitant costs. You have the right to be informed of schedule changes, in time to choose how to reduce scope to restore the original date. You cancel at any time and be left with a useful working system reflecting investment to date. Agile Software Development 56

Developer Bill of Rights l l l You have the right to know what is needed, with clear declarations of priority. You have the right to produce quality work at all times. You have the right to ask for and receive help from peers, superiors, and customers. You have the right to make, and update your own estimates. You have the right to accept your responsibilities instead of having them assigned to you Agile Software Development 57

Agile Estimation l Estimation is often political – Estimates are selected to meet arbitrary end dates – Same bad estimates are accepted over and over l Yesterday’s weather – How much can we get done in this iteration? l l As much as we got done last iteration How big is this task – Find a similar task you’ve done l It’ll take that long Agile Software Development 58

Consequences of Yesterday’s Weather l l Won’t habitually over-estimate Encourage people to finish some tasks, rather than half-finish all tasks Time to recover from bad iterations Easy to explain Agile Software Development 59

The Agile Craftsman l l l Contribution over authorship Collaboration over cleverness Learning over knowing Delivering over process Simplicity over generality Cleanliness over being prolific Agile Software Development 60

Agile Methodologies l l l l Scrum Extreme Programming (XP) Crystal (Clear, Yellow, Orange, etc. ) Lean Development Dynamic Systems Development Model (DSDM) Feature-Driven Development (FDD) Adaptive Software Development Agile Software Development 61

Scrum l “The New Product Development Game” in Harvard Business Review, 1986. – “The… ‘relay race’ approach to product development…may conflict with the goals of maximum speed and flexibility. Instead a holistic or ‘rugby’ approach—where a team tries to go the distance as a unit, passing the ball back and forth— may better serve today’s competitive requirements. ” l Wicked Problems, Righteous Solutions by De. Grace and Stahl, 1990. – First mention of Scrum in a software context Agile Software Development 62

Scrum Origins l Jeff Sutherland – Initial Scrums at Easel Corp in 1993 – IDX and nearly 600 people doing Scrum – Not just for trivial projects l Ken Schwaber – ADM – Initial definitions of Scrum at OOPSLA 96 with Sutherland l Mike Beedle – Scrum patterns in PLOPD 4 Agile Software Development 63

Scrum has been used in… l l l Independent Software Vendors (ISVs) Fortune 100 companies Small startups Internal development Contract development Non-software development – Marketing campaigns Agile Software Development 64

Scrum has been used for… l l l FDA-approved, life-critical software for x-rays and MRIs Enterprise workflow systems Financial payment applications Biotech Call center systems Tunable laser subsystems for fiber optic networks Application development environments 24 x 7 with 99. 99999% uptime requirements Multi-terabyte database applications Media-neutral magazine products Web news products Agile Software Development 65

Scrum Characteristics l l l One of the “agile processes” Self-organizing teams Product progresses in a series of month-long “sprints” Requirements are captured as items in a list of “product backlog” No specific engineering practices prescribed – Often paired with practices embraced by XP l Uses generative rules to create an agile environment for delivering projects Agile Software Development 66

Scrum Overview Daily Scrum Meeting Sprint Backlog 24 hours Backlog tasks expanded by team Product Backlog As prioritized by Product Owner 30 days Potentially Shippable Product Increment Source: Adapted from Agile Software Development with Scrum by Ken Schwaber and Mike Beedle. Agile Software Development 67

How Scrum Works l l l l Develop highest value functionality Only fund for 30 days Self-organize to maximize productivity Inspect every day Remove impediments daily Decide what to do next based on inspection of working functionality Only build what is needed Agile Software Development 68

The Scrum Master l l Represents management to the project Typically filled by a Project Manager or Team Leader Responsible for enacting Scrum values and practices Main job is to remove impediments Agile Software Development 69

The Scrum Team l l Typically 5 -10 people Cross-functional – QA, Programmers, UI Designers, etc. l Members should be full-time – May be exceptions (e. g. , System Admin, etc. ) l Teams are self-organizing – Ideally, no titles but rarely a possibility – What to do if a team self-organizes someone off the team? ? l Membership can change only between sprints Agile Software Development 70

Sprints l Scrum projects make progress in a series of “sprints” – Analogous to XP iterations l Target duration is one month (+/- a week or two) – A constant duration leads to a better rhythm l Product is designed, coded, and tested during the sprint Agile Software Development 71

Sequential vs. Overlapping Development Source: “The New Product Development Game”, Hirotaka Takeuchi and Ikujiro Nonaka, Harvard Business Review, January 1986. Agile Software Development 72

No changes during the sprint Change Inputs Sprint Tested Code Plan sprint durations around how long you can commit to keeping change out of the sprint Agile Software Development 73

Product Backlog l A list of all desired work on the project – Usually a combination of l l Task-based work (“improve exception handling”) Story-based work (“let user search and replace”) – Each task/story/feature is assigned an initial highlevel estimate l List is prioritized by the Product Owner – Typically a Product Manager, Marketing, Internal Customer, etc. – Reprioritized every sprint Agile Software Development 74

Sample Product Backlog Agile Software Development 75

t en ag an M Cu s to m em er s am Te m ru Sc Pr od uc t. O wn e r Sprint Planning Meeting Product Backlog Team Capabilities Business Conditions Technology Sprint Planning Meeting Sprint Goal Sprint Backlog Current Product Agile Software Development 76

The Sprint Goal l A short “theme” for the sprint: Life Sciences “Support features necessary for population genetics studies. ” Database Application “Make the application run on SQL Server in addition to Oracle. ” Financial Services “Support more technical indicators than company ABC with real-time, streaming data. ” Agile Software Development 77

From Sprint Goal to Sprint Backlog l Scrum team takes the Sprint Goal and decides what tasks are necessary – Velocity (i. e. yesterday’s weather) l Target Team self-organizes around how they’ll meet the Sprint Goal – Team members define tasks, estimates, and sign-up for tasks – Manager doesn’t assign tasks to individuals l l Managers don’t make decisions for the team Sprint Backlog is created Agile Software Development 78

Sample Sprint Backlog Agile Software Development 79

Sprint Planning as Shopping l Items – Features, stories, etc. l Prices – Estimates l Budget – How much can be done in a sprint – Velocity You can only buy what you can afford Agile Software Development 80

Sprint Backlog during the Sprint l Changes – Team adds new tasks whenever they need to in order to meet the Sprint Goal – Team can remove unnecessary tasks – But: Sprint Backlog can only be updated by the team l Estimates are updated whenever there’s new information Agile Software Development 81

Sprint Burndown l l Primary visual measure of progress Accurate estimates in the face of constant change Manage expectations Visibility and truth Agile Software Development 82

Sprint Burndown Graph Agile Software Development 83

Daily Scrum Meetings l Parameters – – l Three questions: 1. 2. 3. l What did you do yesterday What will you do today? What obstacles are in your way? Chickens and pigs are invited – l Daily 15 -minutes Stand-up Not for problem solving Help avoid other unnecessary meetings Only pigs can talk Agile Software Development 84

Questions about Scrum meetings? l Why daily? – “How does a project get to be a year late? ” l l “One day at a time. ” Fred Brooks, The Mythical Man-Month. Can Scrum meetings be replaced by emailed status reports? – Entire team sees the whole picture every day – Create peer pressure to do what you say you’ll do Agile Software Development 85

Constraints l A complete list of constraints put on the team during a Sprint: </end of list> Agile Software Development 86

Sprint Review Meeting l Team presents what it accomplished during the sprint – Typically takes the form of a demo of new features or underlying architecture l Informal – 2 hour prep time rule l Participants – – l Customers Management Product Owner Other engineers Reflect on product and process improvement Agile Software Development 87

Stabilization (Release) Sprints l During “regular” sprints target friendly first use – Beta customers and similar can use immediately after sprint l During “stabilization sprints” – Team prepares a product for release – Useful during: l l active beta periods when transitioning a team to Scrum if quality isn’t quite where it should be on an initial release Not a part of standard Scrum, just something some have found useful Agile Software Development 88

Stabilization Sprints Sprint 1 Sprint 2 Sprint 3 Sprint 4 Sprint 1 Sprint 2 Sprint 3 Stabilization Sprint Agile Software Development 89

Scalability of Scrum l l l Typical Scrum team is 5 -10 people Sutherland used Scrum in groups of 600+ I’ve used in groups 80+ (20+ teams) Agile Software Development 90

Scrum of Scrums / Meta-Scrum Agile Software Development 91

How Scrum Delivers Value l l l l Increases user involvement Allows the user to create or change requirements as the project progresses Ensure that the most important features are built first Ensure that only functionality that the user wants is built Deliver business value every 30 days Project can be cancelled at any time, and still deliver value Continuous improvement (product and process) Agile Software Development 92

Extreme Programming (XP) l Developed by Kent Beck, Ward Cunningham, and Ron Jeffries – C 3 Project l l 4 core principles 12 core practices Typically use 1 or 2 week iterations Requirements using User Stories – Notebook cards – JIT requirements Agile Software Development 93

XP Values l l l Communication Feedback Simplicity Courage Respect Agile Software Development 94

XP Principles l l Feedback Assuming Simplicity Incremental changes Embracing change Agile Software Development 95

XP Primary Practices l l l On-site Customer Planning game Small releases Collective ownership Coding standards Sustainable pace (40 hour week) l l l Metaphor Continuous integration Pair programming Refactoring Test first programming Simple design Agile Software Development 96

Summary l l l l Empirical Business value driven Continuous product and process improvement Barely sufficient process Self organizing Collaborative Disciplined Honesty and visibility Agile Software Development 97

Where to go next? l l l www. agilealliance. com www. controlchaos. com scrumdevelopment@yahoogroups. com Agile Software Development 98

References and Thanks l Mary Poppendieck – Scrum 101, March 2004 – Principles of Lean Thinking, 2002 l Ken Schwaber – Let Common Sense Be Your Guide to Agile Processes, Cutter Consortium, Vol. 4, No, 3, 2003 l Mike Cohen – An Introduction to Scrum, April 2003 l Martin Fowler – Planning Agile Projects, 2000 l Craig Larman and Victor Basili – Iterative and Incremental Development: A Brief History, IEEE Computer, June 2003 l Robert Martin – Agile Software Development: Principles, Patterns, and Practices, 2004 – XP Overview, 2004 Agile Software Development 99

References and Thanks (2) l Frank Mauer – Agile Software Engineering, January 2006 l Karl Scotland – Agile Development, 2005 l Mishkin Berteig – Scumdevelopment Yahoo. Group posting #16238, September 2006 Agile Software Development 100

Copyright Notice l This work is licensed under the Creative Commons Attribution. Non. Commercial-Share. Alike License. To view a copy of this license, visit http: //creativecommons. org/licenses/by-nc-sa/1. 0/ or send a letter to Creative Commons, 559 Nathan Abbott Way, Stanford, California 94305, USA. Agile Software Development 101