NASA OCIO IT Summit NASA Long Term Experience

NASA OCIO IT Summit NASA Long Term Experience with CMM and CMMI Aug 17, 2010 John C. Kelly, Sally Godfrey & Tim Crumbley Office of Chief Engineer

Involvement with CMM & CMMI • Dr. John Kelly (NASA HQ, Office of the Chief Engineer) – – – Program Executive for Software Engineering within the NASA Headquarters Office of Chief Engineer. Certified Project Management Professional Chair of the NASA Software Working Group Member of the CMMI Steering Group (co-sponsored by Do. D and NDIA). Principal Engineer at NASA’s Jet Propulsion Laboratory (JPL) in Pasadena, CA and led Agency-wide initiatives in Formal Methods for Computer Systems, and Software Formal Inspections. – Prior to joining NASA-JPL, Dr. Kelly was a professor of Computer Science at Furman University in Greenville, SC, and a Mathematics professor at Darton College in Albany, GA. • Sally Godfrey (GSFC) – – – • Worked with CMM concepts in Software Engineering Laboratory during 80’s and 90’s Led GSFC Flight Software Branch to CMMI Level 2 in 2006 Led GSFC Class A and B projects to CMMI Maturity Level 2 in 2008 Trained as Candidate SCAMPI Lead Appraiser On SEI’s CCB for CMMI V 1. 3 update Tim Crumbley (MSFC) – Co-Chair of the NASA Software Working Group – Led the MSFC flight software organization to achieved a CMM Maturity Level 2 rating in December 2000 (first NASA center to achieve CMM ML 2) – Led the MSFC flight software organization to achieved a Flight Software organization achieved CMM Maturity Level 3 rating in May 2003 – Led the MSFC flight software organization to achieved a Flight Software organization achieved CMMI Maturity Level 2 rating in May 2005 – Led the MSFC flight software organization to achieved a Flight Software organization achieved CMMI Maturity Level 3 rating in August 2007 (first NASA center to achieve CMMI ML 3) 2

Timeline 2000 – 2010 NASA Software Engineering Initiative – CMM and CMMI Activities MSFC JPL La. RC* (FSSB) La. RC* (SDAB) GSFC La. RC* SDAB La. RC* ASDC La. RC (FSSB) JSC KSC La. RC CMMI Level 3 CMMI Level 2 MSFC CMM Level 3 CMM Level 2 MSFC 2000 JSC 2001 MSFC ARC* JSC La. RC JPL ARC GRC JPL 2002 2003 JPL JSC 2004 2005 Started: NASA SW Eng Improvement Initiative GRC** MSFC (SIL) ARC *- Implemented part of the model 2006 Signed: Software 5 Centers had Procedural experience Requirement using the model NPR 7150. 2 Software Working Group Charter MSFC (FSW**) JPL** 2007 2008 2009 2010 Signed: Software 8 Centers had Procedural experience Requirement using the model NPR 7150. 2 A NPR 7150. 2 (2004) NPR 7150. 2 A (2009) Class A and B - CMMI Level 2 or CMM Level 3 Class C - Center Decision Class A - CMMI Level 3 Class B - CMMI Level 2 Class C - Center Decision CMM/CMMI Appraisals & Consultations at NASA Centers Software Engineering Training and SEI Training at NASA Centers ** appraisals scheduled in late FY 10 for these CMMI targeted ratings 3

OCE NASA-wide Software Classification* Class A Space Flight Human Rated Software Systems Class B Non-Human Space Rated Software Systems Class C Mission Support Software & Facilities Class D Analysis and Distribution Software Class E Development Support Software (e. g. , Class A – C is mostly software developed or acquired for Highly Specialized IT systems) Class F General Purpose Computing Software CIO (Multi-Center or Multi-Program/Project) Class G General Purpose Computing Software (Single Center or Project) Class H General Purpose Desktop Software Notes 1. “It is not uncommon for a project to contain multiple systems and subsystems having different software classes” (P. 2. 1) 2. Whether software is safety critical is an independent determination based on NASA-STD 8719. 13 * Established by NPR 7150. 2 A 4

NASA CMMI Summary Completed CMMI Appraisals from FY 07 -FY 10 Center Rating Date La. RC- ASDC PP(CL 3), CM(CL 1) Nov-06 MSFC ML 3 JPL # Project s Type Organizatio nal size Software Classes Assessed 1 Data Center Support 85 Class C Apr-07 3 Development 63 Class A, B and C ML 3 Sep-07 7 Dev & Maintenance 1000 Class A, B and C GSFC ML 2 + RSKM(2) May-08 4 Dev & Maintenance 600 Class A, B and C La. RC- FSSB ML 2 + CL 3 Oct-08 3 Services 5 Class B & C La. RC- SDAB PP(CL 3), REQM(CL 3), CM(CL 3), MA(CL 3) Mar-09 4 Development 21 Class B & C JSC ML 2 Apr-09 4 Development 90 Class A, B, C and D KSC ML 2 Sep-09 1 Development 225 Class A, B and C MSFC – SIL ML 2 + CL 3 May 2010 1 Development 50 Class C ARC –ISD (Code TI) ML 2 May 6 Appraisals Development 63 Class B & C Scheduled CMMI in FY 10 2010 SCAMPI A B C Center Month MSFC June JPL October GSFC February April GRC August GSFC June JSC 5

CMM/CMMI Lessons Learned • • • • • Established sponsorship across departments – Management Steering Group – Was difficult to get mid-level managers to “own” improvement program Established early on a relationship with the Lead Appraiser PIID development and artifact collection – PIIDs and artifacts were maintained on a server for ease of access and review Importance of interview preparation and training Develop an extensive set of “tools” (i. e. , templates, spreadsheets) to help projects with CMMI practices and artifacts – Use of toolset helped projects reach compliance much faster Mentors can help get Project tool use started and help Projects tailor the artifacts Use the workshops to review the processes in depth and reinforced the tool sets Tracking Progress, determine a method for projects to report progress Many of our projects need basic project management and configuration management training CMMI process helped Centers establish a baseline of where they are Process provided projects feedback and helped identify areas for improvement Despite initial reluctance, pre-appraisal was a positive experience for our projects - laid a good foundation for future involvement Projects appreciated systematic and analytical feedback on what they are doing Measurement and analysis is a big challenge Improved quality and review of management plan early in the life cycle and reuse of the plans for new projects Preparing for an appraisal is where you get the measurable process improvement Resource planning and tracking below the account level specifically at the individual process area level provided little additional benefit to the individual projects The many smaller projects need to have light-weight processes to avoid being smothered (especially for a one person task) 6

Key Impacts • Reduces risk of software failure - Increases mission safety – Improvement processes based on best practices in Industry and Government – Risk management much improved on software subsystems--Previously there was little monitoring of risks • • More predictable software cost estimates and delivery schedules Smarter buyer of contracted out software – Educate the NASA workforce on best practices in Software Engineering • • More defects found and removed earlier Reduces duplication of efforts between projects Increases ability to meet the challenges of evolving software technology Software development planning has been improved across the Agency – – • There is a growing consensus among the practitioners and software managers that working to a defined process has substantial benefits. Vast improvement in planning of software projects and in monitoring progress NASA’s contractor community has heard the word that the bar has been raised with respect to software engineering and is responding appropriately 7

Key Impacts • A solid foundation and structure is now in place for developing software in a disciplined manner – – – • More uniformity in management plans, reviews, test plans, status reporting Risk management much improved on software subsystems--Previously there was little monitoring of risks Data management and configuration management has improved Improve the working relationships between Engineering and Safety and Mission Assurance in respect to software engineering The Agency is better prepared for major programs and projects than it was 8 years ago – • Software teams and software quality engineers are working together to assure compliance to standards, to improve quality The knowledge and skills of the NASA software engineering community has significantly improved – • • We have seen significant cultural changes. Extensive mentoring program established to improve software practices Our projects are now better managed –particularly in the area of progress tracking – Now we know exactly where we are in the project and how long it’s likely to take to finish 8