Agile software development in an earned value world

Agile software development in an earned value world: a survival guide AS 16 - AS 108 -73 Jeffrey Kantor LSST Data Management Project Manager Kevin Long LSST Project Controls Specialist and the LSST Data Management Technical/Control Account Managers Jeffrey Kantor, LSST (USA); Kevin Long, BCF Solutions (USA); Jacek Becla, SLAC National Accelerator Lab. (USA); Frossie Economou, LSST (USA); Margaret Gelman, Univ. of Illinois at Urbana-Champaign (USA); Mario Juric, Univ. of Washington (USA); Ron Lambert, LSST (Chile); K. Simon Krughoff, Univ. of Washington (USA); John D. Swinbank, Princeton Univ. (USA); Xiuqin Wu, California Institute of Technology (USA) AMCL • July 17, 2015 1

Motivation − ”Big science” astronomy and astro-physics projects are no longer treating the data management aspects as an afterthought to be done once the telescope is in place − Data management combines large-scale, multi-year equipment, communications, and services procurements, offthe-shelf and customized middleware, and custom scientific, algorithm-intense software, integrated into a complex system − Managing the entire effort requires more formal mechanisms, and if funded by International and/or Federal entities, that increasingly includes the requirement to perform Earned Value Management (EVM) − Software development best practice is “agile”, which employs iterations of short, intensive “sprints” to develop the software incrementally, and to address emerging requirements and iterative design AMCL • July 17, 2015 2

The essential problem − Earned Value Management requires significant up-front planning of the entire effort, to create a resource-loaded, scheduled baseline plan, against which progress is tracked analyzed in terms of cost and schedule − EVM also discourages purely “level of effort” planning, where the plan only indicates a team working for some time period and deliverables are not specified − Agile process does not encourage long-term up front planning, rather it focuses on the near-term, with the longerterm aspects left “fuzzy” or level of effort against a prioritized backlog − How to marry these two in such a way as to get the benefit of both? AMCL • July 17, 2015 3

Data Management System Architecture and WBS Application Layer (LDM-151) • Scientific Layer • Pipelines constructed from reusable, standard “parts”, i. e. Application Framework • Data Products representations standardized • Metadata extendable without schema change • Object-oriented, python, C++ Custom Software Middleware Layer (LDM-152) • Portability to clusters, grid, other • Provide standard services so applications behave consistently (e. g. provenance) • Preserve performance (<1% overhead) • Custom Software on top of Open Source, Offthe-shelf Software Infrastructure Layer (LDM-129) • Distributed Platform • Different sites specialized for real-time alerting, data release production, petascale data access • Off-the-shelf, Commercial Hardware & Software, Custom Integration 02 C. 05 Science User Interface and Analysis Tools 02 C. 01. 02 - 03 SDQA and Science Pipeline Toolkits 02 C. 06. 01 Science Data Archive (Images, Alerts, Catalogs) 02 C. 01. 02. 01, 02 C. 02. 01. 04, 02 C. 03, 02 C. 04 Alert, SDQA, Calibration, Data Release Productions/Pipelines 02 C. 03. 05, 02 C. 04. 07 Application Framework 02 C. 06. 02 Data Access Services 02 C. 07. 01, 02 C. 06. 03 Processing Middleware 02 C. 07. 02 Infrastructure Services (System Administration, Operations, Security) 02 C. 07. 04. 01 Archive Site 02 C. 07. 04. 02 Base Site 02 C. 08. 03 Long-Haul Communications Physical Plant (included in above) Data Management System Design (LDM-148) AMCL • July 17, 2015 4

Data Management EVM Planning Process/Levels LSST System Requirements and Operations Plans Project Risks & Milestones • DM Roadmap (LDM-240) • High-level picture • Granularity: 6 months DM System Requirements and Roadmap PMCS Release Agile Plan • • Institutional-level resource assignments Budgeting Earned-value management Basis of monthly reports Granularity: 3 - 6 months 2 x 6 -month development/release cycle per year 10 -20 activities for each cycle • • • Detailed 6 -month plan Individual resource assignments ~100 activities per cycle Granularity of activities: 1 – 3 months Granularity of steps within activities: 2 – 20 days AMCL • July 17, 2015 5

LSST Software Development Roadmap (LDM-240) • T/CAMs now tracking ~385 milestones • Major axes are 6 -month cycles x WBS • Cells represent milestones/major results • Cells to left are predecessors to cells to the right Feature Implementation AMCL • July 17, 2015 Performance Improvement 6

LDM-240 DM Key Performance Metrics Tracking 42 Key Performance Metrics Major axes are 6 -month cycles x metric Cells represent planned achievement of metric Progressively increasing performance from left to right • SRD minimum spec is met first, then design spec, then (if applicable) stretch goals • • AMCL • July 17, 2015 7

Organization DM PMCS Prior to 6 -month rolling wave plan − Estimated effort for future work exists in the time phased plan as planning packages along with estimated milestone completion dates − Each package has a clearly defined scope, deliverable(s), resource cost, and an end date − Planning Packages are converted to detailed work as planned in Jira in the rolling wave process • Planning Packages are in baseline plan as 6 month cycles. • Estimated individually via UMLbased specifications • DLP milestones are identified that will be completed at the end of a cycle AMCL • July 17, 2015 8

Planning, Scheduling, & Budgeting DM 6 -month rolling wave Release Plan • Activities (epics) are 1 – 3 months development • Estimated individually via developers, Agile process • Each one is further detailed in steps (stories) of 2 – 20 days AMCL • July 17, 2015 9

Planning, Scheduling, & Budgeting Agile Software Process using JIRA Agile − The development team T/CAMs plan each 6 -month cycle in terms of Epics, Stories, Bugs, etc. in JIRA − The T/CAMs and individual developers only work within JIRA using the agile process during the cycle. This allows them real -time access to record progress while data input into the PMCS is controlled by project controls (monthly for status and as needed for change control). − Performance metrics can be generated from PMCS and Jira tools − Resource estimation and assignment to Epics is done outside of Jira (in Excel) and is integrated into PMCS AMCL • July 17, 2015 10

Resource Loading to set Baseline • T/CAMs complete spreadsheet to assign resources and hours to a specific Epic/Activity AMCL • July 17, 2015 11

JIRA – PMCS Integration − At least one month prior to the start of the release cycle the Project Controls Specialist exports the cycle plan to the PMCS − Every month, the Project Controls Specialist exports plan updates and status information from JIRA to our PMCS − Every month, EV is analyzed in the PMCS, and variances are calculated − Every month, the T/CAMs are required to provide overall status and variance explanation narratives AMCL • July 17, 2015 12

PMCS – EVM - JIRA Agile Mapping Schedule Planning Object EVM Object JIRA Agile Object Work Breakdown Structure Cost Account Level 4 “WBS” Custom Field “Cobra WP” Activity Code Field Work Package/Planning Package Cycle, Fix Version Activity N/A – Many to one relationship between activities and Work Packages Epic Activity Step Sum of weights used to calculate % Complete Story, Bug, Improvement Milestone (KPM) Key Performance Metric Assignee Resource AMCL • July 17, 2015 Assignee 13

Stories Quantifiably Measure Performance − Epics are resource-loaded Activities in PMCS, Stories are Activity Steps − Steps are sub-activities that in general can be done in any order and represent portions of the total activity effort for the resource(s) assigned to that activity − All Stories in an Epic have Story Points (~ 4 hours of work/SP), setting their weight of each Step to measure total progress on an activity − A Story is either Not Done or Done, no partial completions − Completing a Story (Step) contributes to the % Complete of the Activity − As Stories are completed and marked Done in JIRA, the status is pulled into PMCS, on a monthly basis − When these Stories and any additional that may be added during a cycle areall marked Done and the status is imported to PMCS, the Epic will show 100% complete − It is likely that a subset of stories have been planned for an Epic. In that case a step is added to Primavera called “Remaining Stories” which is the delta of the estimated stories on the epic, and the actual planned stories. AMCL • July 17, 2015 14

Completed Epic Example DM-1107 JIRA • Each Jira story is assigned as an activity step in Primavera. • Weight in Primavera matches the Story Point in JIRA. • % Complete in Primavera calculates as 100% when all stories are complete. • In this case all Stories in Epic are completed so the epic and activity are marked as complete. Primavera AMCL • July 17, 2015 15

Work Package Integration Primavera to Cobra Primavera Activities in WP KLM 20401 A. PROC Screenshot from Cobra for WP KLM 20401 A. PROC • Note BAC, Earned and % complete match between Primavera and Cobra AMCL • July 17, 2015 16

Tracking Progress with JIRA − Earned Value is very effective over the long term, but it has “lags” – 1 week to 1 month lag between updates in JIRA and in PMCS – 1+ month lag between PMCS updates and monthly reports – EV is “skewed” by late invoicing, requiring estimated actuals − We have the ability to assess earlier, looking at trends in JIRA, based on Epics and Stories directly – Estimated Story Points vs. Planned Story Points – Planned Story Points vs. Completed Story Points − We use this to get more intra-cycle looks at progress AMCL • July 17, 2015 17

Process flow in LSST PMCS tools AMCL • July 17, 2015 18

e. CAM • e. CAM provides an interface for CAMs to drill down from their EVMS performance at the CA/WP level to the associated activities and provide variance narratives when appropriate AMCL • July 17, 2015 19

Summer 2015 Completed vs Planned Epics (3) − Typical Analysis: – We consider overall work rate to be relatively constant over a 6 -month cycle, since we do not have large staffing changes during the cycle – Progress should be roughly linear with time (but see reporting comment below) – The SP Variance % of Completed/Planned is 37% which is ~4% slow versus expectation of 33% at 2/3 date, i. e. we are behind by 4% of total planned work – Our experience is that we under-report progress until the last month, due to “conservative” assessments of completion, and having only done unit tests which don’t expose all problems – We will both add SP in the last 2 months and complete them at a higher rate – Based on the above, I expect us to slip 15% of S 15 SP AMCL • July 17, 2015 20