Agile for Infrastructure Applying Agile Software Development Practices

Agile for Infrastructure Applying Agile Software Development Practices to other IT Domains Terry Hamilton IASSIST Computing Services Inc. http: //www. iassist. ca October 22, 2007 Agile for Infrastructure

Abstract Presentation Title: Agile for Infrastructure : Applying Agile Software Development Practices to other IT Domains Speaker: Terry Hamilton, President of IASSIST Computing Service Inc. Abstract: Agile Practices have shown their value for countless Software Development projects across many fields and specialties. However Software Development is not the only IT endeavor that can benefit from Agile for Infrastructure is a demonstration of how the principles of Agile can be applied by Systems teams to deliver Agile Infrastructure to large scale projects. October 22, 2007 Agile for Infrastructure 2

Not SOFTWARE DEVELOPMENT • Agile as discussed in this presentation: – NO CODE > Pure Server and Middleware rollout – NO DEVELOPERS > Sys. Admin types • Agile is applied to Infrastructure in many places – Boeing, Citibank, IBM, more : internal and engagements • RFP’s and Contracts starting to require Agile • Young Agile was SWAT Teams, October 22, 2007 Agile for Infrastructure Skunk. Works, etc. 3

With repeatable action and tangible metrics grip! October 22, 2007 Agile for Infrastructure 4

Agile for Infrastructure – Case Study • AGILE (Scrum) for large cutting edge SOA build – IBM z. Series, z. OS, 200 -400+ Linux on z. Series, VMWARE – 100’s of Web Services, 10 k’s of users – 40+ Middleware latest versions across z. OS and Linux on Z • Web. Sphere (application layer), Tivoli (sys mgmt), Rational (development) • Application Environments for SOA worth tens M$ October 22, 2007 Agile for Infrastructure – Will replace 30 yrs of COBOL, 5 yrs for application 5

Obstacles to Agile Adoption Obstacles are Opportunities October 22, 2007 Agile for Infrastructure

Biggest Obstacle - Chicken & Egg • How to build an infrastructure to run applications that have not been designed yet? • IT Team: “Tell us what you need!” • Dev Team: “Tell us what we’ll get!” – Large hardware buy-in costs = unwillingness to fire the starting shot! – Months of repeated estimating and planning and debate • This alone sold Agile! – We promised to deliver tangible, working product, month by month, and still be flexible to changing requirements from the application teams! October 22, 2007 Agile for Infrastructure 7

Obstacle – New Skills & Legacy Staff • New Infrastructure skills = becoming good with new Programming Lang. – Skills with this middleware stack are rare – Original Project plan had 25+ infrastructure person years! – Reality = Start Now! Deliver DEV environment in 6 months or less • Legacy shop – years of cookie cutter solutions (CICS, DB 2, z. OS) – Same processes, people and applications for years and years • “Lets start with 10 people for 1 month and see where we get. ” – Hand picked 8 go-getters from the staff plus two contractors – Only Scrum Master had any Agile experience or training • Had to break from the existing culture October 2007 a Post-it Note Agile Infrastructure – 22, “Here’s – goforset your own deliverables!!!” – Biggest fight = dedicated team members 8

Obstacles – Existing Processes • Process is an overloaded word – Many times a Process is really • Desk Procedures only known to few individuals or a single team • Habit instead of structure – Officially following CMMI Level 3 processes – But existing CMM supported Development, not infrastructure • Years since anything but upgrades have been formally required • You cannot bypass or ignore existing processes. – But we can work in a way that doesn’t engage them – We isolated the team and the work • removing dependencies implicitly removed ill-fitting processes October 22, 2007 Agile for Infrastructure – Don’t avoid processes just to be independent or 9

Obstacles – Management Doubt • Agile is so new it is scary – “Self Managed teams” is a political phrase – use it wisely. – Scrum Master had to accept risk of Ownership • Team had Ownership but additionally SM was the face for that risk – Agile was being evangelized by Middle layer (Architects) • Sold it UP to management and DOWN to technical staff • Evangelize ruthlessly and bring in Experts for the outside view – Scott Ambler could sell Agile Snow at the North Pole • “Had to allow oversight” (Agile demands it! We wanted it!) – Management updated at weekly Mgmt meetings • Invited but yet to actually attend a Scrum or Planning (but that’s another issue) • Existing processes eventually drove estimates that were scarier than Agile October 22, 2007 Agile for Infrastructure – Risk 22 days on Agile OR risk delivering late or never. 10

Implementing Agile Our customized techniques “Localized Process Improvements” October 22, 2007 Agile for Infrastructure

Daily Scrum Meeting • Implemented AS IS! • Typical difficulties: – Get folks to agree to another meeting each day – Become Tech chats instead of 1 minute Updates • Typical advantages: – Identify dependencies – Face time for Team Members – Early warning indicators – Sense of dependency between team members – Sense of urgency to the deadline October 22, 2007 Agile for Infrastructure 12

Deliver Working Systems • Every 22 days delivered working middleware / servers – Maybe not “complete” but installed, running and usable • Included Install docs, Standards, Hardened • Following iterations added tasks – Configured & Performance tuned, automation – Document procedures, integrate with XYZ, ABC and MNO products • “DONE” clearly defined to the team – Reviewed and approved by the Infrastructure Architect / Product Owner – Available to the team so available to review as required October 22, 2007 Agile for Infrastructure 13

Release & Iteration Planning • Agile for Infrastructure approach: – Middleware & servers prioritized with Stakeholders before iterations – Evolve the Operational Model as we learn • Leveraged Iteration 0 – hardware & software predecided • Iteration 0 design a cut/paste from “Patterns for e. Business” • Refined by learning what was wanted, needed and could do • Surprisingly little re-work required – Products not up to the marketing glossy but worked well 22, 2007 Agile for Infrastructure • October Stories = Middleware & Servers (the Release 14

User Stories started with none. • We had none! – Application couldn’t to tell us what their requirements were! – Devised our own: A piece of Middleware or a Server = a Story • We took holistic approach – Build it to Install & Planning Guides • Redbooks are great resource for IBM products • Whitepapers – but not just any whitepapers • Best Practices – as documented or consult experts – Case-by-case asks for Application Architect input • This input changed constantly as app designs floated • Decisions / Assumptions must favor the generic – Choices should prefer open-ended solution to allow October 22, 2007 Agile for Infrastructure changes later 15

Continuous Testing • This was Tough! • How to do this with Middleware and Servers? – A completely different concept in QA • Decided on two checkpoints: – IVT – Install Verification Tests • Use “sample” apps that come with the products – Peer Reviews • Architect (Product Owner) walkthru of docs and install • Integration between products a defacto Peer Review • Openly acknowledged that we couldn’t be perfect – Some changes likely. Agile required when the apps arrived for Infrastructure October 22, 2007 16

Burndown Charts • • Implemented (mostly) AS IS! Tracked interruptions as a separate work task – 100% Dedication is not the same as 100% focused • Focus and mind share required, not just time spent • Staff don’t acknowledge time spent on “small favours” • Skilled people like interruptions – reflects on their abilities October 22, • 2007 Go-To Person. Agile for Infrastructure syndrome – “Jim always has the answer. ” 17

Burndown Charts Panic! 5 days in and already 40 hrs By tracking Interruptions we behind! reveal several problems: 1. Not a dedicated team 2. Staff absorb extra work without revealing it 3. Skills resided in specific individuals 4. People didn’t feel empowered to say No. October 22, 2007 Agile for Infrastructure 18

Self Managed Teams • Often a tough sell – Even after you put politics and entrenched practices aside… – The hardest part of getting to the team to form • You can’t form a team only a group; they do the rest with encouragement! • Three sided: – Management needs to know the Agile Experiment is in good hands • A 30 day iteration wasted is still waste even if short – Team needs to learn to trust itself and its members • One loose cannon will put everyone on the defensive – Trust in the PO and SM to assist (“lead”) and protect team • Old habits – these positions of authority are deferred to in October 22, 2007 crisis Agile for Infrastructure 19

Information Radiator / Kanban • Go Big or Go Home – use a Task Room, not Board – All 4 walls of our meeting room covered in Post-it Notes • TODO Wall, WIP Wall, DONE Wall, • ACCEPTED Wall and ISSUES Wall (yes, that is 5 walls) – Information Blanket not Radiator – Can’t be missed! • Post-it® Notes – The Best Tool Ever! – Interactive! Team creates, moves through states, signed when done – Usual boring kickoff became impassioned, engaged planning session • Management stood at the door amazed : the usually docile staff debating, cajoling, negotiating, juggling the to-do list • Wiki as a team room & repository – Collaborative, version control, attachments, easy access October 22, 2007 for Infrastructure – Easy to learn, easy Agile to maintain, looks good too 20

Information Radiator - Task Room “Above the Line” uncommitted Iteration Backlog (products & servers todo) TODO Wall WIP Wall Tasks / Stories committed Done Wall October 22, 2007 Agile for Infrastructure 21

Accomplishments Our Successes and Failures October 22, 2007 Agile for Infrastructure

Successes • Other projects starting to evaluate Agile – Imitation of the project because it was a success • 1 st Agile project and 1 st Iteration by 1 st Scrum Team – only 5% off estimated hours (after “interruptions” accounted for) • 200+ Linux on z. Series servers including Middleware – On time : Estimated in January, on time in October – On budget: actually reduced dedicated team size over time – On quality: 2 months of use and 2 only mis-configurations October(bugs) 22, 2007 so far Agile for Infrastructure 23

Successes • Demonstrate ability to identify and manage risks and issues – The Burndown is Information • Project plans, Gantts, %Complete don’t reveal information, only document facts – Obstacles tracked daily via Scrum – Risks, deferred workload, assumptions on Wiki as they were found • Sr. Mgmt presented to Executives on our success – Made Agile more visible throughout organization and beyond • Users are satisfied, Team feels successful, Mgmt October 22, 2007 Agile for Infrastructure 24 relieved

Failures • Agile – That’s when you skip the documentation, right? – Highly visible but not highly understood • Not intended as a training exercise but could leverage better • Depth of understanding limited to “the guys with the post-it project plan” – Suddenly seeing “Iteration” instead of “Phase” on Waterfall plans • Iteration is obviously the cool new thing • “The server guys did it so we can do it” • Training and Education – One Team now has experience with Agile and Retrospectives were positive but need to grow the # of October 22, 2007 & individuals Agile Infrastructure teams withforknowledge. 25

Failures • Generalists – Cross-over knowledge gained but not enough to support • Middleware products require too much depth • Skills, techniques, dependencies are specific – Department structures hard to break • Structure separated skills, most returned to their traditional groups • Transition to Business As Usual – Still negotiating to show that Agile works on BAU support • Works for “one. Agile time” build cycle but unproven on October 22, 2007 for Infrastructure 26 support day-to-day

Lessons • Sponsor Absolutely required. – 2 nd Level management or higher – Person authorized to dictate allocation of staff across departments • Evangelists – Bring someone from outside to sell Agile – Scott Ambler works. – Have someone inside to promote and start/run it • Start with Agile/Scrum Skeleton and customize to fit – RESIST the urge to fall back to traditional approaches • Yes, Excel is nice. No, it is not a time tracking tool! – Don’t mess with the basics when you are starting out • Iterations of fixed length < 30 days, Daily Standup October 22, 2007 Agile for Infrastructure 27 meeting, Dedicated resources

Lessons • Burn Down chart is your absolute best friend – Simplest tool but hard for uninitiated to understand – take the time to teach • Form the team carefully – 1 st Agile team are “picked volunteers”, not pure volunteers, not assigned – Seed the team with potential: tech skills aren’t everything a team needs • Load the dice – On Project 1, Iteration 1, be conservative with estimates, limit committed deliverables, allow time for teaching, do training in advance (iteration 0) – Management willing to “risk” agile will usually understand the value on the first iteration October 22, 2007 of motivating the Agileteam for Infrastructure 28 – Do not cheat but make Iteration 1 a success by preparing it

Q&A October 22, 2007 Agile for Infrastructure 29