See More Learn More Tell More year 2

See More, Learn More, Tell More (year 2: 2004) Tim Menzies West Virginia University tim@menzies. us Galaxy Global: Robert (Mike) Chapman Justin Di Stefano Research Heaven, West Virginia

Problem Research Heaven, West Virginia • We have many do-ings – But what are we learn-ing? • What general lessons about software quality assurance can we offer NASA? • Problem of external validity – It worked “there” but will it work “here”? • Mountains of data – Seek “the pearls in the dust” 2

Hypothesis 1. NASA SARP sees enough projects to let us make externally valid conclusions • 2. Sometimes! Those conclusions are useful • 3. Research Heaven, West Virginia YES! Those conclusions can be found via data mining methods • • • Well… Standard data miners can get too clever Often, simpler methods suffice 3

Approach • while not (( end of time OR end of money )) – – • • • Research Heaven, West Virginia chase data sets extract cost-benefit patterns from data check the stability of those patterns report stable conclusions Process metrics: cost estimation data from JPL Product metrics: NASA metric’s data program What else: go to SAS, beg for data Cost method 1 method 2 method 3 Faults found 4

Importance/ Benefits Research Heaven, West Virginia Specifically#1: • better cost estimation • Generally: –NASA does a lot of software –What guidance should we offer developers? –How good is that guidance CLCS • Has that guidance been certified? • Do we know how general are those guidelines? So many tools… Stop telling Me that “it” worked, once Specifically#2: • cost- effective defect detectors • based on static code measures Tell me how often it will work 5

Relevance To NASA Research Heaven, West Virginia • Data comes from NASA – Process metrics – Product metrics • Conclusions apply to NASA projects 6

Accomplishments • • Research Heaven, West Virginia Can demonstrate conclusions stable across multiple NASA projects Process conclusions – Accurate cost-estimation: after only 12 projects – ? ? better than prior state-of-the-art • PRED(30) = 69% – For 69% of projects, – actual*0. 7 <= estimate <= actual*1. 30 • Product conclusions – Old news: static code measures can generate detect defectors – New news: prior pessimism concerning such detectors unfounded • some measures and some generators yield same conclusions across multiple NASA projects – Implications: can now tailor SQA methods • Mix and match SQA methods • Desired goals within available money Cost method 1 method 2 static code defect detectors Faults found 7

Next Steps Research Heaven, West Virginia • Data mining needs data – Got data? • Then meet your new best friend • Current plans – Data reduction experiments • Can we collect fewer process metrics and still do cost estimation? • Can we learn good defect detectors sooner? – Model-based metrics: Got data? • Can we learn defect detectors from reports of MATLAB inspections? • Can we learn how to speed up model checking? – Better static code detectors • Would “disjunctive learning” help us? 8