Semantic Mutation Testing John A Clark Haitao Dan

  • Slides: 28
Download presentation
Semantic Mutation Testing John A. Clark, Haitao Dan, and Robert M. Hierons Andreas Voraberger

Semantic Mutation Testing John A. Clark, Haitao Dan, and Robert M. Hierons Andreas Voraberger 13. 01. 2014 Andreas Voraberger AK Softwaretechnologie 1 WS 2013/14 1

Content • Introduction • Traditional Mutation Testing • • The idea Mutation count Possibilities

Content • Introduction • Traditional Mutation Testing • • The idea Mutation count Possibilities Advantages / Disadvantages • Semantic Mutation Testing • • • The idea Implementing semantic mutation Advantages Semantic Error Model Common misunderstandings Refinement Translation Example (Cruise-control system) Conclusion Andreas Voraberger AK Softwaretechnologie 1 WS 2013/14 2

Introduction • Testing is important but expensive • 50% of development budget • Test

Introduction • Testing is important but expensive • 50% of development budget • Test automation • More efficient/effective • Mutation Testing • Distinguishing between descriptions (N) • Semantic Mutation Testing • Distinguishing between semantics (L) Andreas Voraberger AK Softwaretechnologie 1 WS 2013/14 3

Introduction • Code defined by the pair: (N, L) • N: the description (source

Introduction • Code defined by the pair: (N, L) • N: the description (source code) • L: the semantics of the language Andreas Voraberger AK Softwaretechnologie 1 WS 2013/14 4

Traditional Mutation Testing Andreas Voraberger AK Softwaretechnologie 1 WS 2013/14 5

Traditional Mutation Testing Andreas Voraberger AK Softwaretechnologie 1 WS 2013/14 5

The idea • Mutants are classes of faults • Produce mutants (N’) • Making

The idea • Mutants are classes of faults • Produce mutants (N’) • Making changes to the program • Apply mutation operators • {+, -} • {<=, <} • {delete parts} • (N, L) (N‘, L) • Test Cases should distinguish N/N‘ (mutant killed) Andreas Voraberger AK Softwaretechnologie 1 WS 2013/14 6

Mutation count • Single mutation operator creates large number of mutations • Even in

Mutation count • Single mutation operator creates large number of mutations • Even in small programs • Only use “first-order mutants“ • Produced by application of one operator • Hypothesis: all “first-order mutants“ kill most “higher-order mutants“ • Equivalent mutant: not kill able Andreas Voraberger AK Softwaretechnologie 1 WS 2013/14 7

Possibilities 1. Judge test data against the mutants created 2. Produce test data to

Possibilities 1. Judge test data against the mutants created 2. Produce test data to kill all mutants • A test set that is good in distinguishing N from N’ is good at finding faults Andreas Voraberger AK Softwaretechnologie 1 WS 2013/14 8

Advantages • Target particular classes of faults • Optimal for small slips or typos

Advantages • Target particular classes of faults • Optimal for small slips or typos • Simulate other test criteria • Mutants lead to errors provide coverage Andreas Voraberger AK Softwaretechnologie 1 WS 2013/14 9

Disadvantages • High number of mutants, even with • Only “first-order mutants“ • Small

Disadvantages • High number of mutants, even with • Only “first-order mutants“ • Small programs • Solution: “selective mutation“ • Equivalent mutants • Leads to manual effort • Increases costs • Doesn’t aim on semantic mistakes Andreas Voraberger AK Softwaretechnologie 1 WS 2013/14 10

Semantic Mutation Testing (SMT) Overview Andreas Voraberger AK Softwaretechnologie 1 WS 2013/14 11

Semantic Mutation Testing (SMT) Overview Andreas Voraberger AK Softwaretechnologie 1 WS 2013/14 11

The idea • Produce mutants (L’) • (N, L) (N, L‘) • Discover misunderstandings

The idea • Produce mutants (L’) • (N, L) (N, L‘) • Discover misunderstandings related to semantic mistakes • Requires description language with a semantics that allows manipulation Andreas Voraberger AK Softwaretechnologie 1 WS 2013/14 12

Implementing semantic mutation • Parameterisable system • Mutate through changing the parameters • Express

Implementing semantic mutation • Parameterisable system • Mutate through changing the parameters • Express semantics that can be manipulated • A set of rewrite rules • Simulate a mutation of the semantics • Changes to the syntax throughout the description Andreas Voraberger AK Softwaretechnologie 1 WS 2013/14 13

Advantages • Lead to far fewer mutants • Fewer equivalent mutants • Tests translation,

Advantages • Lead to far fewer mutants • Fewer equivalent mutants • Tests translation, refinement, migration, porting code • Eliminates misunderstandings Andreas Voraberger AK Softwaretechnologie 1 WS 2013/14 14

Semantic Mutation Testing (SMT) Scenarios Andreas Voraberger AK Softwaretechnologie 1 WS 2013/14 15

Semantic Mutation Testing (SMT) Scenarios Andreas Voraberger AK Softwaretechnologie 1 WS 2013/14 15

Semantic Error Model • Change of description in development process • Abstract Concrete 1.

Semantic Error Model • Change of description in development process • Abstract Concrete 1. 2. 3. 4. 5. Requirements Specification Design Code Machine code • Misunderstandings in every transformation Andreas Voraberger AK Softwaretechnologie 1 WS 2013/14 16

Semantic Error Model Andreas Voraberger AK Softwaretechnologie 1 WS 2013/14 17

Semantic Error Model Andreas Voraberger AK Softwaretechnologie 1 WS 2013/14 17

Semantic Error Model • Refinement • Transformation to different abstraction level • Translation •

Semantic Error Model • Refinement • Transformation to different abstraction level • Translation • Transformation on the same abstraction level Basis for misunderstandings Solution: SMT Andreas Voraberger AK Softwaretechnologie 1 WS 2013/14 18

Common misunderstandings • Find set of semantic misunderstandings • For different languages • Based

Common misunderstandings • Find set of semantic misunderstandings • For different languages • Based on studies or experience • Implement tests, finding them • Set of operators should reflect the environment • Different mistakes depending on programmer skills Andreas Voraberger AK Softwaretechnologie 1 WS 2013/14 19

Refinement • Semantic mistakes between abstraction levels • Problems on copying syntax • Maybe

Refinement • Semantic mistakes between abstraction levels • Problems on copying syntax • Maybe different semantics • Examples • • • Truncation rules Precedence rules Datatypes Floating point precision Binary representation Andreas Voraberger AK Softwaretechnologie 1 WS 2013/14 20

Migration / Porting (Translation) • Migration / Porting to a different language • Maybe

Migration / Porting (Translation) • Migration / Porting to a different language • Maybe different semantics • Use a semantic test case generation tool • Suits of semantic mutation operators for common combinations • Point to common failures • Example: C to Z • Division of negative numbers • Branching structures • floating-point comparison Andreas Voraberger AK Softwaretechnologie 1 WS 2013/14 21

Semantic Mutation Testing (SMT) Example Andreas Voraberger AK Softwaretechnologie 1 WS 2013/14 22

Semantic Mutation Testing (SMT) Example Andreas Voraberger AK Softwaretechnologie 1 WS 2013/14 22

Example • Shows • cruise-control system • Not only slips have to be tested

Example • Shows • cruise-control system • Not only slips have to be tested • Developers can make semantic mistakes Andreas Voraberger AK Softwaretechnologie 1 WS 2013/14 23

Statechart for a cruise-control system Andreas Voraberger AK Softwaretechnologie 1 WS 2013/14 24

Statechart for a cruise-control system Andreas Voraberger AK Softwaretechnologie 1 WS 2013/14 24

Cruise-control system (Case 1) • State: no_vehicle_in_front • Events: • brake • level=increase •

Cruise-control system (Case 1) • State: no_vehicle_in_front • Events: • brake • level=increase • What happens? Andreas Voraberger AK Softwaretechnologie 1 WS 2013/14 25

Cruise-control system (Case 2) • State: no_vehicle_in_front • Events: • Vehicle detected • level=increase

Cruise-control system (Case 2) • State: no_vehicle_in_front • Events: • Vehicle detected • level=increase • What happens? Andreas Voraberger AK Softwaretechnologie 1 WS 2013/14 26

Conclusion • Semantic Testing is necessary • SMT is a working practice • Captures

Conclusion • Semantic Testing is necessary • SMT is a working practice • Captures a different failure type than MT • Focuses on misunderstandings • Normal MT still necessary Andreas Voraberger AK Softwaretechnologie 1 WS 2013/14 27

Quelle • John A. Clark, Haitao Dan, and Robert M. Hierons. Semantic mutation testing.

Quelle • John A. Clark, Haitao Dan, and Robert M. Hierons. Semantic mutation testing. In Third International Conference on Software Testing, Verification, and Validation Workshops, pages 100 -109. IEEE, 2010 Andreas Voraberger AK Softwaretechnologie 1 WS 2013/14 28

Semantic Mutation Testing John A Clark Haitao DanSemantic Mutation Testing John A Clark Haitao Dan
Mutation Testing Mutation Testing n n Mutation testingMutation Testing Mutation Testing n n Mutation testing
MUTATION TESTING MUTATION TESTING Mutation testing is aMUTATION TESTING MUTATION TESTING Mutation testing is a
Mutation What is a Mutation A mutation isMutation What is a Mutation A mutation is
MUTATION CONTINUATION B Chromosomal mutation Aberration Mutation thatMUTATION CONTINUATION B Chromosomal mutation Aberration Mutation that
Semantic Contentbased Access To Hypervideo Databases Haitao JiangSemantic Contentbased Access To Hypervideo Databases Haitao Jiang
What is Mutation Testing The premise in mutationWhat is Mutation Testing The premise in mutation
Data Mutation Software Testing AData Method Mutation forData Mutation Software Testing AData Method Mutation for
Semantic Web Examples Semantic Web Examples Semantic WebSemantic Web Examples Semantic Web Examples Semantic Web
Mutations Mutation damage to genetic material A mutationMutations Mutation damage to genetic material A mutation
Mutation Transposition and Repair A Gene Mutation 1Mutation Transposition and Repair A Gene Mutation 1
Mutation A mutation is a change in theMutation A mutation is a change in the
Mutations Spiderman bite Mutation A mutation is aMutations Spiderman bite Mutation A mutation is a
What is a Mutation Mutation change in aWhat is a Mutation Mutation change in a
What is mutation A mutation is a changeWhat is mutation A mutation is a change
RATES OF POINT MUTATION The rate of mutationRATES OF POINT MUTATION The rate of mutation
Chapter 7 a DNA mutation and repair MutationChapter 7 a DNA mutation and repair Mutation
a a mutation forward mutation a a reversea a mutation forward mutation a a reverse
Mutations What is a mutation A mutation isMutations What is a mutation A mutation is
Mutation and Mutagenesis Mutation is a sudden heritableMutation and Mutagenesis Mutation is a sudden heritable
Mutations What is a Mutation A mutation isMutations What is a Mutation A mutation is
Mutation Mutation Means change Definition An event thatMutation Mutation Means change Definition An event that
DNA Mutation Mutation is the process by whichDNA Mutation Mutation is the process by which
Mutation A mutation is a change in theMutation A mutation is a change in the