David Lorge Parnas Major contributor to information hiding

David Lorge Parnas Major contributor to information hiding and modularization Advocate of software development as an engineering discipline “A sign that the Software Engineering profession has matured will be that Including good we lose our preoccupation documentation! with the first release and Opponent of “star wars” focus on the long term health of our products. ” Fellow of ACM, IEEE

Textbook view: Maintenance Transfer Testing Construction Design Analysis Req's Software Lifecycle

Req's Analysis Design Construction Testing Transfer Textbook view: Maintenance Transfer Testing Construction Design Analysis Req's Software Lifecycle More realistic view: EVOLUTION, MAINTENANCE, AND ADDITIONAL RELEASES

Maintenance Transfer Testing Construction Design Analysis Textbook view: Req's Software Lifecycle Req's Analysis Design Construction Testing Transfer e h t f o 5% 7 “ a More realistic view: n i t r o eff t c e j o r p e r a softw ent on EVOLUTION, MAINTENANCE, is sp ” e c n a n AND ADDITIONAL i. RELEASES e t n a m

Meir (Manny) Lehman On team Building first computers in Israel in 1950 s Worked at IBM studying OS/360 Professor at Imperial College London Received Harlan Mills award Passed away 29. 12. 10 in Jerusalem “Father of software evolution” Defined “E-type” systems that are ingrained with their environment Lehman's 8 Laws describe how such systems evolve

Lehman's Laws 1) Continuing change (adaptation) 2) Increasing complexity (unless refactored) 3) Self regulation (of rate of change) 4) Invariant work rate (inertia) 5) Conservation of familiarity (of users and developers) 6) Continuing growth (more features) 7) Declining quality (unless maintained) 8) Feedback system (at multiple levels)

All projects Terminating • Project defined by its goals • Requirements map to acceptance test • Develop-delivermaintain trajectory Evolutionary • Goals are uncovered as project progresses • Continue indefinitely as long as useful • Multiple releases along the way

All projects Terminating Evolutionary Staged model • Repeated cycles • Each one is planned with well-defined goals Perpetual development • Plan as you go • Concurrent sub -projects • Fuzzy longterm goals

All projects Terminating Evolutionary Single delivery Pos sib Staged model Perpetual development Multiple releases Continuous deployment le u pda tes

Timescales Waterfall or unified process – Once Evolutionary development – Months Agile development Facebook Continuous deployment – deliver } Release – One day } Deploy – < Hour – Weeks

Staged Model • Each version passes conventional phases • E. g. full cycles of Unified Process • New versions branch in parallel to maintenance Rajlich & Bennett, Computer 2000

Evolution and Agile All software projects • Agile can be terminating • Agile implies a process Evolutionary/ perpetual may be processless e. g. Linux Terminating Evolutionary Perpetual Agile Staged

“Perpetual” Terminology Maintenance Evolution Delivery Release Requirements Feature requests

size Perpetual Development Lifecycle l nt a i t e ini pm lo e v de time

size Perpetual Development Lifecycle rel ea l nt a i t e ini pm lo e v de se Production use & maintenance time

size Perpetual Development Lifecycle rel ea l nt a i t e ini pm lo e v de se d t e nu en i t n pm o c elo feedback v e d & requests Production use & maintenance time

Perpetual Development Lifecycle rel size ea rel ea l nt a i t e ini pm lo e v de se se d t e Production use & maintenance nu en i t n pm o users c elo feedback v upgrade de & requests Production use & maintenance time

Perpetual Development Lifecycle rel size ea rel ea l nt a i t e ini pm lo e v de se se d t e nu en i t n pm o c elo feedback v e d & requests d t e Production use & maintenance nu en i t n pm o users c elo feedback v upgrade de & requests Production use & maintenance time

Perpetual Development Lifecycle rel ea rel size ea rel ea l nt a i t e ini pm lo e v de se se se d t e Production & maint. u en n i nt pm o users c elo feedback v upgrade de & requests d t e Production use & maintenance nu en i t n pm o users c elo feedback v upgrade de & requests Production use & maintenance time

Perpetual Development Lifecycle rel ea rel size ea rel ea l nt a i t e ini pm lo e v de se se se ed nt u tin me n co elop v e d d t e Production & maint. u en n i nt pm o users c elo feedback v upgrade de & requests d t e Production use & maintenance nu en i t n pm o users c elo feedback v upgrade de & requests Production use & maintenance time

Linux Example

“Linux is evolution, not intelligent design” -- Linus Torvalds

Eric Raymond: The Cathedral and the Bazaar Planning and management Evolution

Raymond’s Main Insights • Open-source developers are “scratching a personal itch”, so they know the requirements • Having real users leads to real feedback • “with enough eyeballs, all bugs are shallow” (Linus’s Law) • Users are co-developers, may come up with good ideas • “Release early, release often” leads to the most rapid progress

Linux Old Release Model

Linux Old Release Model • Distinction between versions • Production is stable • Development is current • Distinction between releases • Minor production release for bug/security fixes • Minor development release every few days • A major release is a major operation taking months • Interval between major releases is long • Two years of delaying new features

Linux New Release Model (2. 6)

Linux New Release Model (2. 6) • Only production versions are released • New version every 2 -3 months • • Development done individually Merge window of 2 weeks ~2 months of stabilization and preparation Few minor releases as needed • Maintain some long-term versions for stability

Software Growth Lehman and Turski: A system's complexity grows with time It is harder to modify a more complex system Assuming a constant effort per release, This leads to a diminished growth rate:

Software Growth Godfrey and Tu: Linux (and other systems) is growing at a super-linear rate

More Data 1994 to 2011 3541 l/d 78 x growth (29% annual) 1385 l/d Best models: Quad. -exponential Piecewise linear No theory 660 l/d

Growing Growth Rate • Possible explanation: Positive feedback • Successful project attracts more developers • More developers produce more code • Suggests exponential growth • Contradicts Brooks’s Law (“adding manpower to a late project makes it later”) • System is modular so don’t need so much communication • Good information dissemination via mailing lists etc. • New developers are typically already experts, not novices

Perpetual Development Benefits Lead time to first working version is short, and a working version is always available No danger of the project coming to nothing Real users doing real work are effectively brought into the development cycle Helps to test system functionality and find problems Used to prioritize further development according to what is really needed Ability to use new technology as it becomes available

Continuous Deployment Variant New software released in timescales of minutes, not days or weeks Each developer immediately deploys whatever he works on Requires strong framework to control releases and roll them back if needed Makes the whole notion of a “version” meaningless Popular mainly in web-based companies and applications

Implications for Development No fixed goal that has to be reached Goal is to continually improve the system and maintain is usefulness Monitor system usage to identify inadequacies Prioritize Don't according to user needs plan too far ahead (YAGNI protection) Use contracts that take longevity into account Support Access for continued evolution to code is company becomes insolvent

Implications for Architecture Can't decide on architecture based on analysis of all the requirement Need architecture that accommodates change Two tiers: stable core and evolving libraries Open system like e-commerce site based on web services Use refactoring May need to abandon project eventually But may still salvage parts for a followup project

Conservation of Familiarity One of Lehman's laws of software evolution Limits the rate of progress that can be sustained Need specialized tools to help new team members to become familiar with the system Newbies are at a disadvantage because they didn't see how the system developed Need to capture the history of design decisions

Explaining Monumental Failures caused by "feature creep" Developers made elaborate and beautiful plans But these plans were obsolete by the time they were completed Do exactly what is most needed at each instant Failures caused by successful maintenance Delivered system was good for a very long time But when it is to be replaced, an attempt is made to do too much at once Make improvements continuously all the time

Experimental Evidence Conducted by the World Wide Consortium for the Grid (W 2 COG) The goal: develop a secure service-oriented architecture system Traditional approach: standard government acquisition process Alternative: use a "Limited Technology Experiment" based on evolutionary methods Both start with same government supplied software baseline

18 Months Later. . . Traditional: Evolutionary: A concept document with no functional architecture Cost $1. 5 M No concrete deployment plan or timeline Delivered open architecture prototype addressing 80% of requirements Cost of $100 K Plan to complete in 6 months Denning, Gunderson, & Hayes-Roth, CACM 12/2008

Bottom Line Expect to see many more projects using evolutionary and agile methods Especially in environments challenged by rapid technological progress and rapid change These ideas are actually not new ― However, not articulated well till recently ― Contradict traditional engineering approach ― Nevertheless work well in practice