CERN European Organization for Nuclear Research GS Department

CERN – European Organization for Nuclear Research GS Department – Administrative Information Services Agile Software Development using Scrum Derek Mathieson Group Leader Advanced Information System CERN – Geneva, Switzerland

Agenda l Traditional Software Development l What is Agile? l The Agile Manifesto l Agile Methods l SCRUM @ CERN GS-AIS

Traditional Development CERN GS-AIS

Waterfall Model Requirements Design Implementation Verification Maintenance Time CERN GS-AIS

Waterfall in Practice Safeguard - Ballistic Missile Defence System ‘By the time the 6 -year anti-missile system project was completed, the new missiles were faster than the anti-missiles’ CERN GS-AIS

Spiral Model Cumulative cost 2. Identify and Progress 1. Determine resolve risks objectives Rev iew Requirements plan Concept of operation Prototype 1 Concept of requirements Development plan Verification & Validation Test plan Verification & Validation Prototype 2 Requirements Draft Operational Prototype Detailed design Code Integration Test Implementation CERN GS-AIS 4. Plan the next iteration Rele ase 3. Development and Test

Why Software Is Different? l Software is Complex l Technology Changes Rapidly l Requirements are Incomplete l Change is Considered Easy l Software Design is Research l Construction is Actually Design l Change is Inevitable CERN GS-AIS

Software is Complex “Computer programs are the most intricate, delicately balanced and finely interwoven of all the products of human industry to date” [James Gleick 1992] CERN GS-AIS

Software is Complex CERN GS-AIS

Software is Complex CERN GS-AIS

Software is Complex CERN GS-AIS

Software is Complex CERN GS-AIS

Why Software Is Different? l Software is Complex l Technology Changes Rapidly l Requirements are Incomplete l Change is Considered Easy l Software Design is Research l Construction is Actually Design l Change is Inevitable CERN GS-AIS

Technology Changes Rapidly CERN GS-AIS

Requirements Are Incomplete CERN GS-AIS

An Experiment CERN GS-AIS

Project Description ‘Personal Transport Device’ l Usable outside – Weather proof. l Reasonably Strong – Usable for several years. l Stable, relatively easy to use – Probably 4 wheels? CERN GS-AIS

Implementation CERN GS-AIS ?

“Many projects fail because their developers fail to build the right thing” —Grady Booch CERN GS-AIS

Why Software Is Different? l Software is Complex l Technology Changes Rapidly l Requirements are Incomplete l Change is Considered Easy l Software Design is Research l Construction is Actually Design l Change is Inevitable CERN GS-AIS

Cost of Change Curve CERN GS-AIS

Why Software Is Different? l Software is Complex l Technology Changes Rapidly l Requirements are Incomplete l Change is Considered Easy l Software Design is Research l Construction is Actually Design l Change is Inevitable CERN GS-AIS

Introducing Agile Development CERN GS-AIS

What is Agile? CERN GS-AIS

What is Agile? Agile: – Having the faculty of quick motion; nimble, active, ready. (Oxford English Dictionary) l Agile software development: – A group of software development methodologies based on iterative and incremental development, where requirements and solutions evolve through collaboration between self-organizing, cross -functional teams. CERN GS-AIS (Wikipedia)

Why Agile? CERN GS-AIS

The Iron Triangle Traditional Development Scope Pick Two Quality CERN GS-AIS Schedule

Software Delivery Success 16% Over Cost 53% CERN GS-AIS Canceled 31% The Chaos Report Standish Group (1995)

The Agile Manifesto (2001) Early and continuous delivery of valuable software 2. Welcome Change 3. Deliver Often 4. Customers and developers must work together 5. Best possible people, tools and workplace 6. Emphasis on face-to-face communication 7. Working software is the best measure of progress 8. Constant sustainable progress 9. Focus on technical excellence and good design 10. Simplicity 11. Self-organizing teams 12. Regular reflection on improvements 1. CERN GS-AIS

The 4 Agile Values l Individuals and interactions over processes and tools l Working software over comprehensive documentation l Customer collaboration over contract negotiation l Responding to change over following a plan CERN GS-AIS

The Iron Triangle Traditional Development Scope Pick Two Quality CERN GS-AIS Agile Development Schedule

Iterative Development l Regular releases to customer – ‘Time-boxing’ – Normally 2 – 6 weeks l Adjust design as the project progresses Requirements Analysis & Design Implementation Initial Planning Deployment Evaluation CERN GS-AIS Testing

Agile Methods l Scrum l Feature Driven Development (FDD) l Lean l Extreme l RUP l Kanban CERN GS-AIS Programming (XP)

SCRUM CERN GS-AIS

What is SCRUM? CERN GS-AIS

What is SCRUM? l Scrum is a framework for iterative, incremental development using crossfunctional, self-managing teams. It is built on industry best practices, lean thinking, and empirical process control. Ken Schwaber, 2006 co-creator of SCRUM CERN GS-AIS

Method Comparison Waterfall Spiral Iterative Scrum Required Planning & Closure only Final product Determined during planning Fixed during planning Set during project Project cost Determined during planning Partially variable Set during project Completion date Determined during planning Partially variable Set during project Planning only Planning primarily At end of each iteration Throughout Defined processes Responsiveness to environment Team flexibility, creativity Limited - cookbook approach Unlimited during iterations Training prior to project Teamwork during project Knowledge transfer Probability of success Low Medium low Medium High Jeff Sutherland, ‘The Scrum Papers’ 2010 co-creator of SCRUM CERN GS-AIS

SCRUM in Pictures CERN GS-AIS

SCRUM in Practice CERN GS-AIS

EDH Statistics l 14, 500 active users l 25 k Documents/month l 60 k Signatures/month 3. 00 25, 000 60, 000 Documents per month 20, 000 Distinct Users per month 50, 000 Signatures per month Ratio Signatures/Document 2. 50 2. 00 15, 000 40, 000 1. 50 30, 000 10, 000 20, 000 1. 00 5, 000 10, 000 - CERN GS-AIS - 0. 50 0. 00

EDH Development Team l 4 Staff l 3 Project Associates l 1 Fellows l 1 or 2 Students (9 month contract) l 1. 8 million lines of code l ~1000 3 rd line support calls/year CERN GS-AIS

EDH Development B. C. l Before B. C. SCRUM – Constant Developer Interruptions • Low efficiency – Delivery was often late • Poor estimation – many unknowns – Scope Creep • Specification constantly changing • Everything is Free • Some features never used CERN GS-AIS

SCRUM Vocabulary l Product Owner l Product Backlog l Scrum Team l Sprint Planning l Scrum Master l Daily Scrum l Sprint Backlog l Sprint Review Meeting CERN GS-AIS

Chickens and Pigs. . . Involved CERN GS-AIS Committed

The Product Owner l Typically a Product Manager, Internal Customer, etc. l Responsible for: – Providing and maintain a prioritised “Product Backlog” – Responsive to questions during a sprint CERN GS-AIS

The Product Backlog l. A list of all desired work on the project – Usually a combination of • story-based work: “let user search and replace” • task-based work: “improve exception handling” l Prioritised by the Product Owner – Priority should be (ideally) based on “Business Value” l “Cost” CERN GS-AIS assigned by the Scrum Team

The Scrum Team l Teams are self-organising l Cross-functional – QA, Programmers, UI Designers, Technical Writers, etc. l Assign Cost to each Item on the Product Backlog l Commit CERN GS-AIS to the “Sprint Goal”

Agile Estimation l Planning CERN GS-AIS Poker

The Scrum Master Responsible for enacting Scrum values and practices (The Process) l Main job is to remove obstacles which affect the team l Typical obstacles could be: l – – – CERN GS-AIS My ____ broke and I need a new one. I still haven't got the software I ordered. I need help debugging a problem with ____. I'm struggling to learn ____ and would like help. The GL has asked me to work on something else "for a day or two. "

The Sprint Planning Meeting l Attended by: • Product Owner, Scrum Master, Scrum Team, and any interested and appropriate management or customer representatives. l Product Owner describes the highest priority features to the team. l Collectively the Scrum Team and the Product Owner define a “Sprint Goal” CERN GS-AIS

The Sprint Goal l. A short “theme” for the sprint: “Create Reports. ” “Create Working Form. ” “Implement Workflow. ” “Implement Bulk Emailing. ” l The CERN GS-AIS SCRUM Team commit to this goal.

The “Sprint” l Fixed “Time-Box” (we chose 2 weeks) l Product is designed, coded, and tested during the sprint l Daily Scrum Meetings l Produce demonstratable, working, new functionality. CERN GS-AIS

The Daily Scrum l Anyone Invited l Led by Scrum Master l 15 minutes, every day l Not for problem solving l Three questions: 1. What did you do yesterday 2. What will you do today? 3. What obstacles are in your way? CERN GS-AIS

Process repeats. . . 2 Weeks Pass… CERN GS-AIS

The Sprint Review Meeting l Team presents what it accomplished during the sprint l Typically takes the form of a demo of new features or underlying architecture l Participants – Management – Product Owner – Other engineers –… CERN GS-AIS

Release Sprint 1 l GS-AIS Sprint 3 Release Sprint Concentrate on preparing for production: – – – CERN Sprint 2 No new features Last minute bugs, typos, layout issues, etc. Translation (if not done already) Desktop Icons Communication, Bulletin Articles, etc.

Scrum– value driven not plan driven Empower lean teams to deliver more software earlier with higher quality. l Demonstrate working features to the customer early and often so the customer can inspect progress and prioritize change. l Deliver exactly what the client wants by directly involving the customer in the development process. l Provide maximum business value to the customer by responding to changing priorities in real time. l CERN GS-AIS Jeff Sutherland, 2007 co-creator of SCRUM

SCRUM in Industry l The most profitable software product ever created (Google Adwords) is powered by Scrum. l The most productive large project with over a million lines of code (Sirsi. Dynix) used a. . . Scrum implementation. Jeff Sutherland, 2010 co-creator of SCRUM CERN GS-AIS

SCRUM in Industry No, 31% Organizations using Agile methods Yes, 69% Agile Adoption Survey, March 2008 No, 24% CERN GS-AIS Yes, 76% State of the IT Union Survey, July 2009

SCRUM in Industry Scrum 10. 9% Agile Modeling 6. 0% Feature-driven development (FDD) 3. 8% Test-driven development (TDD) 3. 4% 2. 9% e. Xtreme Programming (XP) Lean development 2. 1% Agile, 35% Microsoft Solutions Framework (MSF) for Agile 1. 8% Agile Data Method 1. 6% Adaptive Software Development (ASD) 1. 3% Six Sigma 0. 9% Crystal 0. 3% Behavior-driven development (BDD) 0. 2% Dynamic Systems Development Method (DSDM) 0. 2% Do not use a formal process methodology 30. 6% Iterative development 16. 3% Iterative, 21% Rational Unified Process (RUP) 2. 7% Spiral 1. 6% 8. 4% Waterfall, 13% 2. 5% Capability Maturity Model Integration (CMMI) ISO 9000 2. 5% CERN GS-AIS Base: 1, 298 IT professionals Source: Forrester Research, Inc. 2009

Visible benefits of SCRUM l Time-Boxed: – Maximum investment known up-front l Tackle most valuable features first l Focus on working, tested, documented product features CERN GS-AIS

Conclusions l Product Owner: – Active Participant – Can “see” product evolve – Know the cost of each feature – Good Product Owners can be hard to find CERN GS-AIS

Conclusions l Team: – Work closely with Product Owner – Know the “Value” of each Feature – Known Start and End of Project – Efficient, highly focused development – Strong Team Spirit CERN GS-AIS

Why SCRUM? l What I wanted: – Manage Product Requirements – Provide Visibility to Clients – Better manage developer time – A more repeatable development process l What developers wanted: – Something ‘light’ – Task management – Communication CERN GS-AIS

What did we adapt? l 2 week Sprint l Release Sprint l Not everyone ‘SCRUMs’ – Full time support staff – Technology l (Almost) Everyone does support too l Some people have several roles CERN GS-AIS

Implementation Barriers l Some clients insist all features must be in final product Scope l Daily Sprint meetings = Interruption l Poor Product Owner – Not final decision maker – Doesn’t want to be. Pick involved Two – More than one (that don’t agree!) Quality CERN GS-AIS Schedule

Lessons Learned l Be careful of the choice of Product Owner l Use tools to simplify admin – Excel, whiteboards, Scrum. Works, JIRA, … CERN GS-AIS

Product Backlog Window CERN GS-AIS

Sprint Detail Window CERN GS-AIS

Web Client CERN GS-AIS

JIRA + Green. Hopper CERN GS-AIS

Jira IDE Integration (IDEA) CERN GS-AIS

Does it Increase Productivity? l Probably… l Did it make development work easier? l Yes… – Communication is better – Estimates are better – Planning is easier – Customers are happier CERN GS-AIS

Thank You CERN GS-AIS

Yes… but… l “I like writing software, but I don’t like doing the other development stuff which we are not forced to do here. ” l SCRUM lets you: – Focus on valuable development l Use CERN GS-AIS tools to minimise admin

Yes… but… l “It might help, but we have multiple projects person. ” l So do we… – It’s simpler to have only one, but sometime schedules don’t allow… – Time-boxing helps to reduce parallel activities. CERN GS-AIS

Yes… but… l “Management l SCRUM won’t agree” offers: – Better Planning – Deadlines met – Minimise unnecessary development – Happy Clients CERN GS-AIS

Yes… but… “Our clients won’t agree” l Tricky one… – SCRUM needs Client commitment – SCRUM exposes the cost of features – SCRUM makes the client choose l In return they get: – Transparency – License to change their minds – Met deadlines CERN GS-AIS

Yes… but… l “I like X from Scrum, but not Y, I might try X. ” l Do CERN GS-AIS X!

Yes… but… l “You are trying to get us to work more for less! No way!” l SCRUM lets you: – Focus on useful work CERN GS-AIS

Yes… but… l “Our project X is special and not industry so we don’t need a process. ” CERN GS-AIS

Thank You CERN GS-AIS