Applications of Metamorphic Testing Chris Murphy University of

Applications of Metamorphic Testing Chris Murphy University of Pennsylvania cdmurphy@cis. upenn. edu November 17, 2011

About Me Lecturer, University of Pennsylvania Ph. D-Computer Science, Columbia Univ, 2010 Advisor: Prof. Gail Kaiser Research: software testing, CS education Seven years experience in software development industry

Problem: When testing a piece of software, how can we know that we’ve created enough test cases?

Problem: When testing a piece of software, how can we create more test cases? Solution: We use properties of the software to create new test cases from existing ones (particularly those that have not failed). Result: This approach, known as metamorphic testing, is more effective at testing certain types of software than other approaches.

Today's Talk What is metamorphic testing? How is metamorphic testing used to find bugs in software? How can metamorphic testing be applied to applications that do not have test oracles? What are the open research questions related to metamorphic testing?

UPenn CIS 573 Software Engineering Graduate-level software engineering course Just over 100 students Focuses on software maintenance issues: – – – Testing Formal verification Debugging Fault Localization Refactoring

It's your first day of work as a software engineer at Blooble. Soft. Your boss gives you 6, 000 lines of code and a specification and says “find the bugs”. Where would you start?

Specification Code Test Case Generation Strategy Test Cases

You've created 4, 837 test cases. All of them pass. How do you know when you're done creating test cases?

Testing Requirements Measurable Adequacy Criteria Test Cases Desired Adequacy Level Coverage Level Acceptable?

Your test cases are achieving 100% coverage. None of them have found any bugs. Are those test cases useful?

Maybe those test cases can be used to create new test cases. The more test cases, the better. Right? This is the idea behind “metamorphic testing”. [Chen et al. , HKUST TR CS-98 -01, 1998]

A simple example (really, really)

Let's say you're testing a cosine function. (I know, I know. . . ) You have a test case {45º, 0. 7071}, i. e. cos(45º) = 0. 7071 How could we use this test case to create new test cases?

We know that the cosine function exhibits certain properties. That is, if we make certain changes to the input, we can predict the effect on the output. These are referred to as “metamorphic properties”.

What are the metamorphic properties of the cosine function? cos(x + 360º) = cos(x) That is, if we add 360 to the input, the output should not change. cos(x - 360º) = cos(x) cos(x + 180º) = -1 * cos(x)

Given our original test case {45°, 0. 7071}, we can create three follow-on test cases. Property: cos(x + 360º) = cos(x) Input: 45º + 360º = 405º Output: cos(45º) = 0. 7071 Property: cos(x - 360º) = cos(x) Input: 45º - 360º = -315º Output: cos(45º) = 0. 7071 Property: cos(-1 * x) = -1 * cos(x) Input: -1 * 45º = -45º Output: -1 * cos(45º) = -0. 7071

Initial test case {x, f(x)} f x f(x) t g t(x) f f(t(x)) = g(f(x)) Follow-on test case {t(x), f(t(x))} A metamorphic property of a function f is a pair of functions (t, g) such that f(t(x)) = g(f(x)) for all inputs x

but wait, isn’t that the same as…

Program invariants: -1 ≤ cos(x) ≤ 1 Describe legal ranges/values of a function, but not how it should react when the input is changed. Algebraic properties: cos²(x) = 1 – sin²(x) Describe the relationships between multiple functions, but not a single function.

simple categories of properties

Initial test case Permute Add Multiply Include Exclude a b c d e f sum s c e b a f d sum s a+2 b+2 c+2 d+2 e+2 f+2 sum s+12 2 a 2 b 2 c 2 d 2 e 2 f sum a b c d e f 2 s sum g sum s+g s-f

Initial test case #1 a b c d e f sum s Initial test case #2 g h i j k l sum t Compose a b c d e f sum g h i j k l s+t sum 2 s+2 t Combination of properties 2 h 2 d 2 a 2 k 2 e 2 g 2 i 2 c 2 l 2 f 2 b 2 j

Common Metamorphic Properties • Additive: Increase (or decrease) numerical values by a constant • Multiplicative: Multiply numerical values by a constant • Permutative: Randomly permute the order of elements in a set • Invertive: Create the “opposite” of a set • Inclusive: Add a new element to a set • Exclusive: Remove an element from a set • Compositional: Compose a set [Murphy et al. , SEKE’ 08]

Other Types of Properties • Noise-based: include input values that will not affect the output • Semantically Equivalent: create inputs that are have the same “meaning” as the original • Heuristic: create inputs that are “close” to the original • Statistical: create inputs that exhibit the same statistical properties

one more example

Consider a function that takes a set of Points (x-y coordinates) and calculates the total distance from the first to the last, via the rest. What are that function’s metamorphic properties?

Okay, I think I get it. But does it really work? !? !

In order to find bugs… 1. The original test case must pass, even though there is a bug. 2. The follow-on test case must fail. But how can this be? !? !

/* Return the smallest value in the array */ int find. Min(int A[]) { int min = A[0]; for (int i = 1; i < A. length-1; i++) { if (A[i] < min) min = A[i]; } return min; }

/* Return the smallest value in the array */ int find. Min(int A[]) { int min = A[0]; for (int i = 1; i < A. length-1; i++) { if (A[i] < min) min = A[i]; } return min; } Test case { {2, 1, 4, 3}, 1} 100% statement coverage! 100% branch coverage! PASS!

/* Return the smallest value in the array */ int find. Min(int A[]) { int min = A[0]; for (int i = 1; i < A. length-1; i++) { if (A[i] < min) min = A[i]; } return min; } Test case { {2, 1, 4, 3}, 1} Metamorphic property: If we permute the input, the output remains the same. Follow-on test case: { {4, 2, 3, 1} FAIL!

metamorphic testing in the real world

Bioinformatics [Chen et al. , BMC Bioinf. , 2009] Machine Learning [Xie et al. , JSS, 2011] Network Simulation [Chen et al. , FTDS, 2009] Computer Graphics [Guderlei et al. , QSIC, 2007]

what types of applications is metamorphic testing good for?

Applications that deal primarily with numerical input and numerical output. Applications that use graph-based algorithms. Compilers. [Zhou et al. , ISFST’ 04] Applications that do not have test oracles.

Program Test Input Actual Output Specification Expected Output Oracle

what if there is no oracle?

Machine Learning

Discrete Event Simulation

f x f(x) t t(x) g f f(t(x)) Actual =? g(f(x)) Expected If f(t(x)) = g(f(x)) that does not mean that the output is correct. But if f(t(x)) != g(f(x)) then one (or both) must be incorrect.

example: Rapid. Miner

Rapid. Miner is a suite of machine learning algorithms implemented in Java. In its Naïve. Bayes implementation, a confidence level c is reported whenever it classifies an example e using a model M created from a training data set T. That is: c = Classify(M(T), e)

We expect that if we modify T to include an extra instance of e, then the confidence level should double, since we are twice as certain about the classification. That is: Classify(M(T+e), e) = 2 * Classify(M(T), e) Our testing detected violations of this property, thus revealing a bug. [Murphy et al. , ICST’ 09]

empirical study

Goal: Show that metamorphic testing is more effective than other techniques at finding bugs in applications without test oracles. Approach: Use mutation analysis to insert faults into the applications, and see how many are detected using various techniques.

Application domains investigated: 1. Machine Learning (C 4. 5, Marti. Rank, Support Vector Machines, PAYL) 2. Discrete Event Simulation (JSim) 3. Information Retrieval (Lucene) 4. Optimization (gaffitter)

Techniques investigated: 1. Metamorphic Testing 2. Runtime Assertion Checking 3. Partial Oracle

Experimental Results [Murphy et al. , ISSTA’ 09]

can we do better?

That experiment used applicationlevel metamorphic properties. What if we test at the function level, too? And continuously conduct those tests while the software is running? This is known as Metamorphic Runtime Checking. [Murphy et al. , TR CUCS-042 -09, 2009]

Experimental Results

research directions

When I run my test, I see that the metamorphic property is violated. Does that mean there's a bug? Well, not necessarily. . How can we know whether the metamorphic properties are sound?

I've used the guidelines to identify as many metamorphic properties as I could. Does that mean that's all of them? Well, not necessarily. . How can we know whether the set of metamorphic properties is complete?

I've used the guidelines to identify as many metamorphic properties as I could. Does that mean that's all of them? Well, not necessarily. . Could we detect (likely) metamorphic properties automatically?

I have a function for which I expect that, if I double the input, the output should be doubled. Could I verify that property without actually executing the code? Well, probably. . Can metamorphic properties be verified statically?

summary

Metamorphic testing is a method of creating new test cases from existing ones. It depends heavily on the software’s metamorphic properties, which are often numerical. Metamorphic testing is particularly effective at finding bugs in applications that do not have test oracles.

thanks!

Applications of Metamorphic Testing Chris Murphy University of Pennsylvania cdmurphy@cis. upenn. edu