Does Agile Work A quantitative analysis of agile

Does Agile Work? – A quantitative analysis of agile project success by Pedro Serrador and Jeffrey K. Pinto A review by Michael A. Gurwell February 8, 2016

Organization • • • Introduction Agile Methods in more detail Project Success Research Methods Results Conclusions 2

Introduction Authors • Pedro Serrador, PMP, ITIL, P. Eng. , MBA, Ph. D has worked as a project/program management consultant for some of the largest corporations in Canada. • Jeffrey K. Pinto, Ph. D. , University of Pittsburgh (1986): Research interests include project management, information system implementation, power and political processes in organizations, and the diffusion of innovations. He teaches in the areas of project management, organizational behavior, and organization theory. • Published in International Journal of Project Management Vol. 35 Issue 5 Pages 1040 -1051 3

Introduction Background • Projects are everywhere In our society involving trillions of dollars every year • The Standish Group’s 2011 CHAOS reports indicate that Failure rates remain high despite importance placed on project performance • New methods of project management were needed. Agile was one such approach that has gained popularity. 4

Introduction Traditional Project Management • Early specification and Design freezing • Sequencing of deliverables / review sessions to track progress and schedule • After requirements gathering, little customer interaction • Static Assumptions 5

Introduction Traditional Results • • Lots of Rework Not very flexible Customer dissatisfaction May not fit into current ever changing business environment Would this be a success even if it met cost and schedule? 6

Introduction The Agile method • • Less Up front planning Continuous design Flexible scope Finalizing design as late as possible Customer interactions Cross disciplinary project teams Iterative and incremental 7

Introduction Do Agile methods work? • Gaining popularity outside of strictly software development projects. • Research has been anecdotal • Based on small sample sizes or limited A comprehensive quantitative study on Agile project success was needed 8

Agile Methods in more detail History • Good mangers never used the “Damn the torpedoes, full speed ahead” approach • Good project management was always aware of possible deviations • Rolling Wave and other iterative processes had been in use before the birth of Agile • The answer to deviations from the plan was not more planning, but more flexibility 9

Agile Methods in more detail The Agile Manifesto • Published in 2001 by 17 software developers • Individuals and interactions over processes and tools • Working software over comprehensive documentation • Customer collaboration over contract negotiation • Responding to change over following a plan 10

Agile Methods in more detail Where is flexibility needed most? • Technology projects exist in a more dynamic environment and customers are less familiar with technology • Projects where the Goals may Change • Projects where resources may change • Projects that have dependencies or other relationships with similar projects 11

Agile Methods in more detail Front end planning – traditional methods • • • Customers only get one shot at Requirements can be overly complex Gold Plating is common Requirements are for a specific point in time Not dynamic Boehm, B. , 1996 Anchoring the Software process IEEE Software. 13 (4) p. 73 -82 12

Agile Methods in more detail Front end planning – Agile methods • Agile methods still have front end planning • First iteration requires large amount of requirements gathering and planning • Agile methods may actually have more planning overall • Planning spread out across the entire development cycle 13

Agile Methods in more detail Factors that call for up-front Planning • • • Size Safety / High (Risk Impact * Risk Probability) Look ahead requirements already known Need a balance Success Find the “Sweet Spot” Wasted effort and Plan On inverted U- curve Rework Failure Up Front Planning 14

Project Success Traditional View Functionality In Budget On Time Data available at project completion Project manager can announce success at project wrap up party • No consideration for customer experience • • • 15

Project Success Definitions for this Analysis Three different Success factors: 1. Project efficiency = meeting cost, time and scope goals. 2. Stakeholder success = satisfying the expectations of project stakeholder who are the best judges of overall success 3. Overall Success = The general perception of a project 16

Project Success Possible Factors that Influence Success • Methodology used • Alignment with corporate strategic goals • Complexity – Variation of interrelated tasks – Complicated tasks • Experience level of project team 17

Project Success Model of Research to be performed Predictor Outcomes 1. Overall Project Success 2. Project Efficiency 3. Stakeholder satisfaction Degree of effort in Agile planning 1. Quality of the vision/Goals 2. Project Complexity 3. Team experience Moderators 18

Research Methods Procedures • • • Wanted as large and diverse data set possible Questionnaire Linkedin Project management groups PMI on-line communities Notices in Mailings Used multiple communities to avoid mononsource bias 19

Research Methods Respondents • Were asked to respond on one “more successful” project and one “less successful” • Targeted project managers, but was not restricted to them • 865 started the survey, but only 859 completed the successful project section. • After removing outliers and bad data they were left with 1386 projects to Analyze 20

Research Methods Participant Breakdown Country Role United States 499 - 36% Project Manager 304 India 96 – 6. 9% Sr. Project manager 141 Canada 93 - 6. 7% Program Manager 72 Australia 31 – 2. 2% Project Coordinator 66 Spain 24 – 1. 7% Team Member 58 Brazil 18 – 1. 3% Senior Manager 36 Singapore 18 – 1. 3% Portfolio Manager 22 Germany 14 – 1% C Level Mgmt 14 85% had more than 5 years of experience 21

Research Methods Survey Questions • Numerical range questions used a 7 point Likert scale • Subjective questions used a 5 point Likert scale • Some questions required a numerical response 22

Research Methods Likert Scales • • • Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree 23

Research Methods Survey Questions • 3 questions measure project efficiency • 4 questions measured project stakeholder satisfaction • One question measured overall Project success • Respondents were also asked about the potential modifiers and most importantly what percentage of the project was Agile 24

Research Methods Efficiency Factor Efficiency factor = mean of the following three questions 1. How did the project do in meeting budget goals? 2. How did the project do in meeting project time goals? 3. How successful was the project in meeting project scope and requirement goals? 25

Research Methods Stakeholder Success Factor= mean of the following three questions 1. How did the project sponsors and stakeholders rate the success of the project 2. How do you rate project team’s satisfaction with the project? 3. How do you rate the client’s satisfaction with the project’s results? 4. How do you rate the end users’ satisfaction with the project’s results? 26

Research Methods Overall Success Factor The Overall Success Factor was based on just one question • How do you rate the overall success of the project? 27

Research Methods Planning Effort Index To help Analyze the planning effort 2 indexes were calculated. 1. Planning effort Index = (Total effort expended on the planning phase in person days)/ (Total project Effort in person days) 2. Agile planning effort Index = (Total effort expended on planning after the planning phase)/ (Total project Effort in person days) 28

Research Methods Bad data removal • • Wanted to focus on projects that had a substantial execution phase and were not canceled prior to completion. Removed projects with planning indexes >. 6 (2 standard deviations over the mean) Agile Planning question had issues with respondents understanding question. Remaining projects were divided into Group A (412) and Group B (590) - Only affected Planning effort 29

Results • Agile is becoming more popular across multiple industries • 6% of the projects were more than 80% composed of Agile methods • 65% of projects used at least some Agile methods • The more a project used Agile methods, the more likely it was to be successful 30

Results Percent of Project was Agile Percentage Count Percent 80 -100 % 80 5. 8 % 60 -79 % 232 11. 0 % 40 -59 % 347 25. 0 % 20 -39 % 162 11. 7 % 1 -19 % 194 14. 0 % 0% 451 32. 5 % Totals 1386 100. 0 % 31

Results ANOVA Analysis of success based on Agile Percent Agile Upfront Planning Agile Planning Efficiency Factor Stakeholder Valid N Success 80 -100 % 0. 161 0. 149 4. 821 3. 638 80 60 -79 % 0. 147 0. 138 4. 664 3. 567 152 40 -59 % 0. 164 0. 132 4. 793 3. 544 347 20 -39 % 0. 135 0. 101 4. 638 3. 414 162 1 -19 % 0. 150 0. 091 4. 460 3. 179 194 0% 0. 154 0. 048 4. 582 3. 208 451 All Groups 0. 153 0. 105 4. 647 3. 376 1386 F 1. 492 18. 37 1. 93 7. 73 p(F) 0. 173 0. 000 0. 087 0. 000 32

Results Correlation Analysis • Correlation between methodology type and success factors Overall Project Efficiency Stakeholder Success Rating Factor Success Factor -. 1721 -. 0617 -. 1570 P=. 000 P=. 022 P=. 000 1 = 80 -100% agile and 6 = fully waterfall 33

Results Regression Analysis of Project Success • Using just methodology and overall success rating R 2=. 030 p level = 0. 000 • Using combined methodology with Agile planning index and overall success rating R 2=. 019 p level = 0. 0001 34

Results -Agile Success by Industry Methodology. Efficiency Stakeholder N p value Government 3. 647 4. 304 3. 169 34 0. 336 Education 3. 8 4. 867 3. 15 10 0. 084 Other 4. 245 4. 484 3. 193 53 0. 0002 Utilities 4. 261 4. 435 3. 413 23 0. 584 Professional Services 4. 273 4. 53 3. 318 22 0. 034 Retail 4. 313 4. 563 3. 234 16 0. 722 Telecommunications 4. 343 5. 105 3. 707 35 0. 57 Health care 4. 5 5. 042 3. 521 24 0. 017 High Technology 4. 526 4. 69 3. 427 57 0. 035 Construction 4. 739 4. 536 3. 652 23 0. 184 Financial Services 4. 781 4. 621 3. 267 73 0. 747 Manufacturing 4. 81 4. 437 3. 304 42 0. 726 All Groups 4. 432 4. 617 3. 356 412 0. 007 35

Results Moderators • Quality of Vision/Goals – Only a marginal significant moderator effect was found • Complexity – No moderator effect found • Team Experience – No moderator effect found 36

Conclusions Authors’ comments • The use of Agile methods does have a statistically significant impact on all 3 measures of project success (efficiency, Stakeholder success, and perceived overall success) • Best results for Agile methods are obtained in the Technology, Personal services and health care. • Quality of goals/vision was the only moderator to have an effect on success and it was only marginal. Complexity and experience had no statistical effect. 37

Conclusions Next Steps • The role of complexity as a moderator • Results may have been statistically significant, but had a small effect in the amount of variance explained. Further research should be conducted with additional modifiers. • Research into dynamic vs. all environments • Research into hybrid Agile vs. pure Agile • Research into total planning amount in Agile 38

Conclusions My comments • Use broad survey, but use projects that are in progress instead of relying on memory of past projects • The project planning data respondent errors • Does the Government, Education, and Utility industries really have a higher percentage use of Agile methods? 39

Conclusions Questions • Thank you for your time • Questions and feedback are welcome 40