Contents 1 Description of Initiative 2 Description of

Contents 1 Description of Initiative 2 Description of Lessons Learned Study 3 Results: Characterization 4 Results: Lessons Learned 5 Implications & Next Steps Software Inspections at NASA: Lessons Learned Report on State of the Practice Forrest Shull, Fraunhofer Center for Experimental Software Engineering - Maryland Judith Bachman, Computer Sciences Corporation John Van Voorhis, Fraunhofer Center for Experimental Software Engineering – Maryland Mike Stark, GSFC John Kelly, JPL

Project Information Contents 1 Description of Initiative 2 Description of Lessons Learned Study 3 Results: Characterization 4 Results: Lessons Learned 5 Implications & Next Steps • This initiative – a collaboration between JPL, GSFC, CSC, and FC-MD – funded by the Office of Safety and Mission Assurance (OSMA) Software Assurance Research Program • Major objectives include: – A lessons learned report about current state-of-thepractice in inspections at NASA – Running of experiments to investigate tailoring and training of new inspection techniques – Running and support of pilot studies to provide long-term support for piloting new inspection techniques

Lessons Learned: Process Contents 1 Description of Initiative 2 Description of Lessons Learned Study 3 Results: Characterization 4 Results: Lessons Learned 5 Implications & Next Steps “Inspection” defined broadly: Any technical examination process during which a product is examined for defect detection by more than just the author. • Analyzed existing process recommendations. • Contacted key personnel at centers for potential participants, then participants directly. • Distributed characterization questionnaire. • Performed semi-directed interview - elicit processes, attitudes, experiences. • Analysis of qualitative data for lessons learned.

Lessons Learned: Subjects Contents 1 Description of Initiative 2 Description of Lessons Learned Study 3 Results: Characterization 4 Results: Lessons Learned 5 Implications & Next Steps • 17 subjects – 9 GSFC & Wallops; 4 JPL; 2 GRC; 1 JSC; 1 LRC – 7 years on average at current center – Majority participated in >10 inspections at that center – Projects: ® Flight SW; data capture; control centers; mission planning ® Team size mostly <10 people; 2/3 of projects last longer than 1 year. – ¾ of respondents currently doing inspections. • Not a comprehensive or random sample… • But experienced people and representative environments.

Characterization Results (1) Contents 1 Description of Initiative 2 Description of Lessons Learned Study 3 Results: Characterization 4 Results: Lessons Learned 5 Implications & Next Steps • What do people inspect? – Most often: requirements, high-level design, low-level design (14/17) – Also test plans (12/17) and code (13/17) ® Only 1 picked code inspections as most beneficial: took over management in mid-development. ® Other respondents who inspect code § develop many small projects with little mission criticality § don’t have chance to influence requirements. – Teams tend to start inspections in the first phase where: ® They have input; ® The document is sufficiently stable and complicated

Characterization of Development Errors

Characterization Results (2) Contents 1 Description of Initiative 2 Description of Lessons Learned Study 3 Results: Characterization 4 Results: Lessons Learned 5 Implications & Next Steps • Why do people inspect? – “Changing/missing/misinterpreted requirements”: on average most important, most often mentioned as source of errors. ® “Unstable requirements” also a major development problem. – “Coding errors” were mentioned by everyone but consistently ranked least important. – Most important thing to look for in any phase is consistency with the previous documents. • How do people inspect? – Almost all knew of or trained on a process. (JPL, SSDM, RA) – 9 used a process. – Very few use checklists. – Only 8 ever collected metrics. ® No correlation with size/mission criticality. ® Those who still do use relatively simple metrics. ® Simple tools for those who do: websites, Excel, Access…

Characterization Results (3) Contents 1 Description of Initiative 2 Description of Lessons Learned Study 3 Results: Characterization 4 Results: Lessons Learned 5 Implications & Next Steps • Do people want to inspect? – Not at first. – Many were convinced by being involved in effective inspections. • No failing projects with inspections • Some used inspections to repair projects • Several subjects felt that all successful development required inspections.

Lessons Learned Results (1) Contents 1 Description of Initiative 2 Description of Lessons Learned Study 3 Results: Characterization 4 Results: Lessons Learned 5 Implications & Next Steps • Communication Benefit - Most important – Team cohesion: Understanding who to go to with a question. – Communication to the customer ® Metrics ® “White box reviews” / adaptability – Getting the right technical info to the right people. – Can survive without inspections if communication happens anyway – Effective practice: combine inspections with other meetings • Training Benefit – Many projects required significant time for learning, had staffing issues – Team building: Getting on board – Cross-training, novice training • Defect Reduction Benefit – It happens - mainly by more communication and better trained staff.

Lessons Learned Results (2) Contents 1 Description of Initiative 2 Description of Lessons Learned Study 3 Results: Characterization 4 Results: Lessons Learned 5 Implications & Next Steps • Using Perspectives during Review – Perspective: Point-of-view and existing expertise against which a reviewer inspects a software product. – [Mars Climate Orbiter, Mishap Investigation Board Phase I Report: had inspections but critical staff were missing] – Everybody who had a choice looked to optimize the set of perspectives. ® Even teams with low formality otherwise. – Some would reschedule if a critical perspective was missing – Typically, no checklists but different perspectives implicitly assumed to look for different things. – When checklists are used: ® “Don’t use them as a replacement for thinking. ”

Lessons Learned Results (3) Contents 1 Description of Initiative 2 Description of Lessons Learned Study 3 Results: Characterization 4 Results: Lessons Learned 5 Implications & Next Steps • Staffing / Mgmt. – Rare that initial plans include process/metrics support. ® “Inadequate staffing” second-largest development problem. ® Understaffed projects, even if they want inspection help, have more pressing concerns. – Functional support lacking, especially for metrics collection. – Losing follow-up: surest way to demotivate people. ® Chief benefit of a more formal process is action item tracking with suitable rigor. – Support is necessary… – … and worth paying for.

Implications and Next Steps Contents 1 Description of Initiative 2 Description of Lessons Learned Study 3 Results: Characterization 4 Results: Lessons Learned 5 Implications & Next Steps • Candidates for further study: – Perspective-Based Reading ® Focus on consistency in requirements/design; individual review ® Communication/Training: Novice and cross-training; explicit responsibilities and expertise ® Perspectives: Make important perspectives explicit; go beyond checklists – JPL Formal Inspections ® Positive feedback; often formed the basis for processes ® Communication: Formal assignment of roles ® Staffing/Mgmt: Very strong on follow-up tracking – Daimler. Chrysler: inspection “sampling” techniques ® Staffing/Mgmt: Seems well-suited to software acquisition and projects on tight schedule

Request for Participation Contents 1 Description of Initiative 2 Description of Lessons Learned Study 3 Results: Characterization 4 Results: Lessons Learned 5 Implications & Next Steps • We are looking for: – Civil servants to directly participate – Civil servants who are overseeing contracts to allow/encourage contractors to participate • Controlled experiments: Participants spend 1 day for: – Receiving training in state-of-the-art techniques that can be taken away and used on real projects. – Receiving feedback on types of defects detected and effectiveness of the training, and some comparison to their usual approach • Pilot studies: Participants work with us on actual projects and: – Receive training in state-of-the-art techniques tailored for their environment & project – Receive extended support for inspections including ® data collection ® consultation ® analysis & feedback

Contact Info Contents 1 Description of Initiative 2 Description of Lessons Learned Study 3 Results: Characterization 4 Results: Lessons Learned 5 Implications & Next Steps Forrest Shull fshull@fc-md. umd. edu 301 -403 -8970