Ivan Pribela Zoran Budimac Gordana Raki USING SOFTWARE

Ivan Pribela, Zoran Budimac, Gordana Rakić USING SOFTWARE METRICS IN EDUCATIONAL ENVIRONMENT

Content • • Motivation Assessment process A case study Conclusion

Motivation • Automated assessment systems are very helpful – Reduce load for instructors – Quicker feedback for students • But have limited scope – Black-box (input-output) tests, unit tests – Lately code style checks • Software metrics are not used – Utilized only by plagiarism detection – Not to help students improve their skills

Motivation (cont. ) • Software metrics can tell a lot about a program – Have highly diverse methodologies and objectives – Difficult to define quality for various programs in uniform way – What which metric means for particular assignment? – What metrics matter the most? • We explore the usage of software metrics in automated assessment – To cover testing of aspects that demand instructor attention – Algorithm complexity – Number and size of programming units and functions

Assessment process Student solution Test report

Student solution MODULE Triplets 1; . . . VAR x, y, z: INTEGER; BEGIN FOR x : = 1 TO Gr DO FOR y : = 1 TO Gr DO FOR z : = 1 TO Gr DO IF x*x + y*y = z*z Write. Ln; Write. String('x = Write. Card(x, 2); Write. String(', y Write. Card(y, 2); Write. String(', z Write. Card(z, 2) END END END Triplets 1. THEN '); = ');

Parsing the solution • e. CST generator (the SMIILE tool) Student solution Parse e. CST – Recognizes input language – Calls appropriate scanner & parser – Generates an e. CST representation • Enriched Concrete Syntax Tree – Modified Concrete Syntax Tree – Enriched with marker nodes • Unit: class, module… • Loop Statement: for, while, repeat… • Branch Statement: if, case, switch… – Independent of programming language – Basis for calculation of software metrics

Measuring e. CST • Metrics calculator (the SMIILE tool) Student solution Parse Metric values in xml Measure e. CST – Software metrics tool in the development – Analyzes the e. CST representation – Calculates software metrics – Exports results in XML file • Final result is one XML file with values of all calculated metrics • Current prototype of SMIILE tool supports several software metrics – Lines of Code – Cyclomatic Complexity – …

Metric values in XML Student solution Parse Metric values in xml Measure e. CST <? xml version="1. 0"? > <source. Element end. Line="24" first. Line="1" name="" type=""> <metrics loc="24" cc="4"/> <source. Element end. Line="24" first. Line="1" name="Triplets 1" type="CONCRETE_UNIT_DECL"> <metrics loc="24" cc="4"/> <source. Element end. Line="23" first. Line="9" name="FOR" type="LOOP_STATEMENT"> <metrics loc="15" cc="4"/> <source. Element end. Line="22" first. Line="10" name="FOR" type="LOOP_STATEMENT"> <metrics loc="13" cc="3"/> <source. Element end. Line="21" first. Line="11" name="FOR" type="LOOP_STATEMENT"> <metrics loc="11" cc="2"/>. . . </source. Element>

Transform XML to properties • XSL Transformator Student solution Parse e. CST – Uses XSLT stylesheet • Input XML file Metric values Measure – Values of calculated software metrics in xml Transform Matric values as properties • Output XML file – Can be manipulated easily inside the testing system

Metric values as properties Student solution Parse Metric values e. CST Measure in xml Transform Matric values as properties <? xml version="1. 0" encoding="UTF-8"? > <metrics> <Triplets 1> <loc>24</loc> <cc>4</cc> </Triplets 1> </metrics>

Testing the values • Testing system (Testovid) Student solution Parse – Implemented as a framework for running domain specific testers e. CST • Domain specific testers – Written as Apache Ant scripts Metric values • Using software metrics Measure in xml Transform Matric values Script with metric control values as properties – Script runs SMIILE tool – Transforms XML file with metrics values – Loads calculated values – Freely uses them for grading, intelligent advice generation… • Final report Test report – Contains advices – Success/failure information – Presented to the student

Test report Course: Assignment: Student: Time: Student solution Parse Metric values e. CST Measure Matric values Script with metric control values Compilation 100% (from 2 points) All is well, no errors. Correctness 33% (from 6 points) Not all triplets were found, check loop boundaries. in xml Transform Data structures and algorithms Assignment 1 - Pythagorean triplets John Doe 05. 09. 2012. 11: 15: 00 as properties Optimality 50% (from 2 points) Try using Euclid's formula. -+-+-+-+- Test Total: 5 out of 10 points. Test report

A case study • Data structures and algorithms course • Testing efficiency of a Modula 2 program • Using cyclomatic complexity metric – detect loop and branch statements • Created a domain specific tester for Testovid – differentiate between typical student solutions – award points accordingly • The assignment – Write a program which prints Pythagorean triplets, positive integer numbers x, y and z for which x 2+y 2=z 2

Solution 1: Naive solution MODULE Triplets 1; . . . VAR x, y, z: INTEGER; zreal: REAL; BEGIN FOR x : = 1 TO Gr DO FOR y : = 1 TO Gr DO zreal : = REAL(Sqrt(LONGREAL(x*x + y*y))); z : = TRUNC(zreal); IF zreal = FLOAT(z) THEN Write. Ln; Write. String('x = '); Write. Card(x, 2); Write. String(', y = '); Write. Card(y, 2); Write. String(', z = '); Write. Card(z, 2) END END Triplets 1. • Cyclomatic complexity 3 • Efficiency Average • Points 50%

Solution 2: Brute force solution MODULE Triplets 2; . . . VAR x, y, z: INTEGER; BEGIN FOR x : = 1 TO Gr DO FOR y : = 1 TO Gr DO FOR z : = 1 TO Gr DO IF x*x + y*y = z*z Write. Ln; Write. String('x = Write. String(', y Write. String(', z END END END Triplets 2. • Cyclomatic complexity 4 • Efficiency THEN '); Write. Card(x, 2); = '); Write. Card(y, 2); = '); Write. Card(z, 2) Bad • Points 0%

Solution 3: Using Euclid’s formula MODULE Triplets; . . . VAR x, y, z, m, n: CARDINAL; BEGIN FOR m : = 1 TO Gr DO FOR n : = 1 TO m-1 DO x : = m*m - n*n; y : = 2*m*n; z : = m*m + n*n; Write. Ln; Write. String('x = '); Write. Card(x, 2); Write. String(', y = '); Write. Card(y, 2); Write. String(', z = '); Write. Card(z, 2) END END Triplets. • Cyclomatic complexity 2 • Efficiency Good • Points 100%

Non solution MODULE Triplets 4; . . . VAR x, y, z, m, n, w, i, temp : CARDINAL; BEGIN w : = 1; n : = 0; FOR i : = 1 TO Gr DO m : = n + w; x : = m*m - n*n; y : = 2*m*n; z : = m*m + n*n; Write. Ln; Write. String('x = '); Write. Card(x, 2); Write. String(', y = '); Write. Card(y, 2); Write. String(', z = '); Write. Card(z, 2); temp : = w; w : = 3*w + 4*n; n : = 2*temp + 3*n END Triplets 4. • Cyclomatic complexity 1 • Efficiency Excellent • Points 100% • Lose points for correctness

Summary • • The greater the cyclomatic complexity the worst the solution efficiency Awarded points should be reverse proportional to the cyclomatic complexity – Inefficient solutions – no points – Average solutions – half the maximum points – Efficient solutions – all the points • Instructor should use knowledge and experience to – Choose metrics – Define minimum and maximum metric values – Define awarded points for those cases • Testing system can automatically – Classify student solutions – Grade them accordingly Solution Efficiency Metric Points Triplets 1 Average 3 50% Triplets 2 Bad 4 0% Triplets 3 Good 2 100% Triplets 4 Excellent 1 !!! 100%

Conclusion • Utilized software metrics in the assessment process • Increased the scope of aspects that can be covered by automatic tests • Platform and programming language independent • Support a wide range of metrics • Left great flexibility in selecting interesting metrics • Can provide hints and advices to students • Added intelligent assistance • Improved student learning experience

Thank you for your attention Questions?