Making a Practical Impact in SW Verification Andreas

Making a Practical Impact in SW Verification Andreas Kuehlmann Software Integrity Group, Synopsys July 19, 2015

Introduction © 2015 Synopsys, Inc. 2

A Typical SW Project Complex Software New Features More Bugs Fix Bugs More Complex SW New Features More Bugs The Grand Challenge: • There seems to be only a way down • Most projects don’t know where they are on that path © 2015 Synopsys, Inc. 3 Fix Bugs

The Challenges and Opportunities Size of SW applications continues to grow Number of SW developers increases steadily § Apollo 11: § 145 k. LOC (lines of code) § Microsoft Windows: § 2. 3 MLOC (1992) § 40 MLOC (2009) § Automobile: § 50 k. LOC (1981) § 100 MLOC (2014) Software tools market grows SW compliance regulations start emerging Health. Care FSI § Security testing growing fastest § Dynamic ready for disruption © 2015 Synopsys, Inc. 4 Automotive Privacy

Six Non-technical Challenges in SW Testing © 2015 Synopsys, Inc. 5

Challenge 1: Developers don't think functionally, they think operationally • Can’t train them to become experts in temporal logic – This was tried in the HW world with little success © 2015 Synopsys, Inc. 6

Challenge 1 (cont. ): Developers don't think functionally, they think operationally • Can’t expect them to understand global loop invariants and be able to maintain them • Can’t explain bugs in such terms • Developers understand gdb – Explain bugs in gdb’s terms! © 2015 Synopsys, Inc. 7 http: //homepages. ius. edu/rwisman/C 455/html/notes/Ch apter 2/Loops/lpinv. htm

Challenge 2: Developers (like all humans) have a very limited memory and context switch is expensive! In 1885, Herman Ebbinghaus did an experiment where subjects memorized a list of meaningless three letter words and tracked how quickly his subjects forgot the words. Source: Gerald Weinberg: Quality Software Management: Systems Thinking © 2015 Synopsys, Inc. 8

Challenge 2 (cont): Developers (like all humans) have a very limited memory and context switch is expensive! Applied Software Measurement, Capers Jones, 1996 • Test the code while it is fresh in your mind – TDD – Real-time code analysis –… © 2015 Synopsys, Inc. 9

Challenge 3: Don’t get in the developer’s way! • Developers have a low tolerance for false bug alarms – 1 st false bug report “Well, it didn’t get this right” – 2 nd false bug report “Annoying” – 3 rd false bug report “This tool is useless” – 4 th false bug report “Boss, why are you wasting my time, get rid of that tool!” © 2015 Synopsys, Inc. 10

Challenge 4: Developers have big egos! “My code is right! – I don’t need any tools!” Yet “What idiot broke the build again!” © 2015 Synopsys, Inc. 11

Challenge 5: Not all developers are A or B grade • Need to enable the tail end and pull them to the left – “Guard-rails” for developers © 2015 Synopsys, Inc. 12

Challenge 6: Management has often little insight into the quality of the SW code developed by their teams Competing factors for project delivery of a SW product: 1. Number and complexity of features 2. 3. 4. Cost to develop them Time needed for developing them Quality and security of the delivered product 1, 2, and 3 have clearly measurable metrics W/o metrics, 4 is the weak player and often sacrificed © 2015 Synopsys, Inc. 13

From my Personal Point of View…. If you want to make a broader practical impact… • Don’t start from a solution and then chase a problem (“the worm and the elephant”) – This leading to success is the exception – Instead, pick an urgent and broad problem – Even at the expense of “academic beauty” • Try to get ahead of the train – Instead of following which I have seen too often in the SW verification research community – Security, Io. T, Self-driving cars, …. • Stay connected with reality – There are millions of SW developers out there in the “stone age” – There a huge amount of low-hanging fruits • There is a lot of luck involved – use your instinct! – Don’t be afraid for change! © 2015 Synopsys, Inc. 14