Designing and Experimenting Conformance and Compatibility Test Suites

Designing and Experimenting Conformance and Compatibility Test Suites on Android-Based Systems Student: Shu-Wei Chou Advisor: Dr. Ying-Dar Lin Degree Program of ECE and CS Colleges, National Chiao Tung University

Outline n Android software architecture ¨ ¨ n Motivation ¨ n n n Conformance and compatibility problem of platforms Survey ¨ ¨ n Dalvik Virtual Machine Android SDK Java(TM) Compatibility Kit (JCK) TAHI IPv 6 Conformance Test Suite Problem statement Design of CTS Experiment Contributions Schedule References High Speed Network Laboratory @ NCTU 2

Android software architecture Application Home Contacts Phone … Browser Application Framework Activity Manager Package Manager Window Manager Telephony Manager Content Providers Resource Manager View System Location Manager Libraries Notification Manager Android Runtime Surface Manager Media Framework SQLite Core Libraries Open. GL | ES Free. Type Web. Kit Dalvik Virtual Machine SGL SSL libc Linux Kernel Display Driver Camera Driver Flash Memory Driver Binder (IPC) Driver Keypad Driver Wi. Fi Driver Audio Driver Power Management Source : http: //developer. android. com/guide/basics/what-is-android. html High Speed Network Laboratory @ NCTU 3

Dalvik Virtual Machine *. java *. class javac JVM Normal Java program *. class *. dex dx DVM Android program High Speed Network Laboratory @ NCTU 4

DVM vs JVM DVM JVM Format . dex . class Architecture Register-based Stack-based JIT compiler No Yes App-to-VM One-to-one Many-to-one High Speed Network Laboratory @ NCTU 5

Android SDK version : 1. 5_r 3 Packages Classes Methods 128 ≒ 2480 ≒ 49600 Packages android. appwidget android. content. pm android. content. res android. database. sqlite android. graphics ． ． ． High Speed Network Laboratory @ NCTU Classes Activity. Group Activity. Manager Alarm. Manager Alert. Dialog Alias. Activity Application Date. Picker. Dialog ． ． ． Methods close. Context. Menu() create. Pending. Result (int request. Code, Intent data, int flags) dismiss. Dialog(int id) dispatch. Key. Event (Key. Event event) dispatch. Touch. Event (Motion. Event ev) find. View. By. Id(int id) ． ． ． 6

Motivation n Conformance problem of platforms ¨ System vendors implement different platforms with different packages, classes or methods ¨ Compatibility problem of applications High Speed Network Laboratory @ NCTU 7

Vendor A 1 2 4 6 A App of vendor A 3 A 8 1 4 3 2 5 2 4 1 2 6 Android SDK Vendor B 2 1 B App of vendor B 4 A package B 3 6 B package 2 3 4 C package 1 2 5 Vendor package [1. . 9] Vendor C 1 2 3 4 Classes C App of vendor C 4 C 5 High Speed Network Laboratory @ NCTU 3 4 2 3 4 5 8

Compatibility table between vendor and application Vendor Application A B C A Y N N B N Y N C N N Y High Speed Network Laboratory @ NCTU 9

Survey n Java(TM) Compatibility Kit (JCK) Test Suites Number of Test Cases Summary JCK-compiler-6 a 19119 Check the condition that. java file is compiled to. class file. JCK-runtime-6 a 58903 Check the condition that. class file is executed. JCK-devtools-6 a 18692 Focus on java and xml schema. n n ANVL(Automated Network Validation Library) TAHI IPv 6 Conformance Test Suite High Speed Network Laboratory @ NCTU 10

Windows CE Test Kit (1/3) CETK Active. Sync Device Test suites High Speed Network Laboratory @ NCTU 11

Windows CE Test Kit (2/3) High Speed Network Laboratory @ NCTU 12

Windows CE Test Kit (3/3) High Speed Network Laboratory @ NCTU 13

JCK-runtime-6 a (1/4) api - Java API test suite lang - The Java Language Specification Tests vm - The Java Virtual Machine Specification Tests xml_schema - W 3 C XML Schema Test Suite • java_applet - API tests for the java. applet package • java_awt - AWT tests • java_beans - API tests for the java. beans package • java_io - API tests for the java. io package • java_lang - API Tests for java. lang • java_math - API tests for the java. math package • java_net - API tests for the java. net package • java_nio - API Tests for java. nio • java_rmi - API Tests for Remote Method Invocation (RMI) • java_security - index. html • java_sql - API tests for the java. sql package • java_text - API Tests for java. text … High Speed Network Laboratory @ NCTU 14

JCK-runtime-6 a (2/4) API Tests for the java. lang Package • Abstract. Method. Error - Tests for Abstract. Method. Error • Abstract. Method. Error serial - Abstract. Method. Error Serialized Compatibility Tests • Arithmetic. Exception - Tests for Arithmetic. Exception • Arithmetic. Exception serial - Arithmetic. Exception Serialized Compatibility Tests • Array. Index. Out. Of. Bounds. Exception - Tests for Array. Index. Out. Of. Bounds. Exception • Array. Index. Out. Of. Bounds. Exception serial – Array. Index. Out. Of. Bounds. Exception Serialized Compatibility Tests • Array. Store. Exception - Tests for Array. Store. Exception • Array. Store. Exception serial - Array. Store. Exception Serialized Compatibility Tests • Assertion. Error - Tests for Assertion. Error • Assertion. Error serial - Test Specifications and Descriptions for Assertion. Error. serial • Boolean - Test Specifications and Descriptions for Boolean • Boolean serial - Boolean Serialized Compatibility Tests • Byte - Test Specifications and Descriptions for Byte … High Speed Network Laboratory @ NCTU 15

JCK-runtime-6 a (3/4) Test Specifications and Descriptions for Boolean n n n public static final Boolean FALSE public static final Boolean TRUE public static final Class TYPE public Boolean(String s) public Boolean(boolean value) public boolean. Value() public int compare. To(Boolean b) public boolean equals(Object obj) public static boolean get. Boolean(String name) public int hash. Code() public static boolean parse. Boolean(String s) public String to. String() public static String to. String(boolean b) public static Boolean value. Of(String s) public static Boolean value. Of(boolean b) High Speed Network Laboratory @ NCTU 16

JCK-runtime-6 a (4/4) n public Boolean(String s) n Description Domain testing of input and output conditions, and external pre-conditions for class Boolean, constructor public Boolean(String s). n n Assertion testing Assertion Allocates a Boolean object representing the value true if the string argument is not null and is equal, ignoring case, to the string "true". Test Case ID Boolean 2005 n Test Descriptions n n n See: Test descriptions for Ctor. Tests. java (Boolean 2005) High Speed Network Laboratory @ NCTU 17

High Speed Network Laboratory @ NCTU 18

High Speed Network Laboratory @ NCTU 19

Classification (java. lang) n n Constructor Serialization Constant Parameters ¨ ¨ ¨ String, int, Object, boolean, char, double, float, long, byte Class[], char[], byte[], long[], Thread[], Accessible. Object[], Stack. Trace. Element[] Class<T>, List<String> java. lang. Class, java. lang. String Class. Loader, Annotation, URL, Throwable, Print. Stream, Print. Writer, Thread. Group, Input. Stream, Inet. Address, Permission, File. Descriptor, String. Buffer, String. Builder, Char. Sequence, Properties, Security. Manager, Runable, Reference. Queue, Invocation. Handler High Speed Network Laboratory @ NCTU 20

public Arithmetic. Exception() public Array. Index. Out. Of. Bounds. Exception() High Speed Network Laboratory @ NCTU 21

public Arithmetic. Exception() - Test public Status Ctor 001() { Arithmetic. Exception e = new Arithmetic. Exception(); if( e. get. Message() == null) { return Status. passed("OKAY"); } else { return Status. failed("FAILED"); } } public Status Ctor 003() { int zero = 0; try { int x = 5 / zero; return Status. failed("FAILED"); } catch(Arithmetic. Exception e) { return Status. passed("OKAY"); } } High Speed Network Laboratory @ NCTU 22

public Array. Index. Out. Of. Bounds. Exception() - Test public Status Ctor 001() { Array. Index. Out. Of. Bounds. Exception e = new Array. Index. Out. Of. Bounds. Exception(); if( e. get. Message() == null) { return Status. passed("OKAY"); } else { return Status. failed("FAILED"); } } public Status Ctor 004() { int[] array = {0, 1, 2, 3, 4}; int[] tested. Values = {-5, 5}; boolean is. Passed = true; for(int i = 0; i < tested. Values. length; i++) { try { int x = array[tested. Values[i]]; ref. println("Error for: " + tested. Values[i]); is. Passed = false; } catch(Array. Index. Out. Of. Bounds. Exception e) { } } if(is. Passed) { return Status. passed("OKAY"); } else { return Status. failed("FAILED"); } } High Speed Network Laboratory @ NCTU 23

public Abstract. Method. Error(String message) public Arithmetic. Exception(String message) High Speed Network Laboratory @ NCTU 24

public Abstract. Method. Error(String message) - Test public Arithmetic. Exception(String message) - Test High Speed Network Laboratory @ NCTU 25

public static Class for. Name(String name, boolean initialize, Class. Loader loader) throws Class. Not. Found. Exception High Speed Network Laboratory @ NCTU 26

protected Package define. Package(String name, String spec. Title, String spec. Version, String spec. Vendor, String impl. Title, String impl. Version, String impl. Vendor, URL seal. Base) throws Illegal. Argument. Exception protected final Class define. Class(String name, byte[] b, int off, int len, Protection. Domain protection. Domain) throws Class. Format. Error High Speed Network Laboratory @ NCTU 27

ANVL – Test Suite Development RFC TAL Analyze Negative Tests Test Method High Speed Network Laboratory @ NCTU C language Test Suite Code 28

ANVL – Example RFC 1723, section 3, page 3, Protocol Extensions. RFC 1058, section 3. 1, pages 19– 20, Message formats. RFC The portion of the datagram from the address family identifier through the metric may appear up to 25 times. TAL 1. ANVL sends a datagram with 26 tuples to DUT. 2. DUT should not accept the packet. 3. ANVL sends a RIP request to the DUT. 4. DUT should return an unchanged routing table. C Code High Speed Network Laboratory @ NCTU Test Method Test Suite Code 29

TAHI IPv 6 Conformance Test Suite n Conformance Test Specification ¨ for host n n n n n IPv 6 Specification ICMPv 6 for IPv 6 Specification Neighbor Discovery IPv 6 Stateless Address Autoconfiguration IPv 6 Path MTU Discovery Default Router Preferences, More-Specific Routes, and Load Sharing IPSec AH and ESP for IPv 6 IPSec ESP for IPv 6(UDP) IPSec AH and ESP for IPv 4(UDP) Mobility Support in IPv 6 (Link to v 6 PC MIPv 6 page) Default Address Selection for Internet Protocol version 6 (IPv 6) DNS discovery IPv 6 over IPv 4 Tunnel Robustness Address Architecture Multicast Listener Discovery for IPv 6 (v 1) ----------------------DHCP for IPv 6 (-28)(contributed by Linux Technology Center, IBM) High Speed Network Laboratory @ NCTU 30

IPv 6 Specification (for host) No. Title 1 Initialization 2 Source and Destination Address (ping) 3 Unrecognized Next Header in IPv 6 Header 4 Payload Length is Odd value 5 Traffic Class Non-Zero 6 Flow Label Non-Zero 7 Hop Limit 0 8 Unrecognized Next Header in Extension Header 9 Hop-by-Hop Options Header Position 10 Destination Options Header Pad 1 Option 11 Destination Options Header Pad. N Option 12 Destination Options Header Unrecognized Option (00) 13 Destination Options Header Unrecognized Option (01) 14 Destination Options Header Unrecognized Option (10) … High Speed Network Laboratory @ NCTU … 31

Problem statement n To verify the conformance of platforms with android SDK and reduce the compatibility issue of applications on android-based systems n Sub problems ¨ Providing test cases to verify the conformance of platforms ¨ Designing automatic framework test tool High Speed Network Laboratory @ NCTU 32

Conformance Test Suite (CTS) n To provide one method ¨ To verify if device conforms android SDK ¨ To avoid incompatible application Automatic framework n Selective modules n Test suites with methodology n High Speed Network Laboratory @ NCTU 33

Design of CTS Based on Android SDK n Process n ¨ Loading to device under test ¨ Run CTS ¨ Display test report n Design test cases ¨ Packages, Classes and methods High Speed Network Laboratory @ NCTU 34

CTS architecture User Interface Test Cases Android SDK High Speed Network Laboratory @ NCTU 35

JCK vs JUnit vs Android JCK JUnit Android JCK Test Framework JUnit Test Framework Android Test Framework Java packages My Java Program My Android Program High Speed Network Laboratory @ NCTU 36

JCK example package javasoft. sqe. tests. api. java. lang. Abstract. Method. Error; import java. io. Print. Writer; import javasoft. sqe. javatest. Status; import javasoft. sqe. javatest. lib. Multi. Test; public class Ctor. Tests extends Multi. Test { /* standalone interface */ public static void main(String argv[]) { Ctor. Tests test = new Ctor. Tests(); test. run(argv, System. err, System. out). exit(); } High Speed Network Laboratory @ NCTU 37

JUnit example import junit. framework. Test; import junit. framework. Test. Suite; public class Test. All { public static Test suite() { Test. Suite suite = new Test. Suite(”Test. Suite Test”); suite. add. Test. Suite(Test. Sample. class); return suite; } } High Speed Network Laboratory @ NCTU 38

Android example (1/3) //Math. Test. java import android. test. Android. Test. Case; import android. util. Log; public class Math. Test extends Android. Test. Case { protected double f. Value 1; protected double f. Value 2; protected double f. Re; static final String LOG_TAG = “Math. Test”; protected void set. Up() { f. Value 1= 2. 0; f. Value 2= 3. 0; f. Re = 5. 0; } public void test. Add() { Log. d( LOG_TAG, “test. Add” ); assert. True( LOG_TAG+” 1〃, ( (f. Value 1 + f. Value 2 ) == f. Re ) ); } } High Speed Network Laboratory @ NCTU 39

Android example (2/3) //Example. Suite. java import junit. framework. Test. Suite; public class Example. Suite extends Test. Suite { public Example. Suite() { add. Test. Suite( Math. Test. class ); } } High Speed Network Laboratory @ NCTU 40

Android example (3/3) High Speed Network Laboratory @ NCTU 41

Android package architecture Package Interfaces Public Methods Classes Protected Methods High Speed Network Laboratory @ NCTU Enums Exceptions Errors Inherited Methods 42

Android. app Package Number Parameter Count Number Class 37 0 157 Method 493 1 196 2 82 3 36 4 11 5 7 6 2 7 1 10 1 Parameter combination High Speed Network Laboratory @ NCTU 123 43

Conformance Test Suite public class ACTS extends Activity implements Test. Listener Android. Test. Runner test. Runner = new Android. Test. Runner() test. Runner. set. Test( new acts. Test. Suite() ) public void add. Error public void add. Failure public void end. Test public void start. Test High Speed Network Laboratory @ NCTU public class acts. Test. Suite extends Test. Suite public class Test. Case 1 extends Android. Test. Case public class Test. Case 2 extends Android. Test. Case 44

Test Case Design public class Activity public final void dismiss. Dialog (int id) {int: -2147483647~ 2147483648} public void set. Up() public void test. Method. Name() public void test. Android. Test. Case Setup. Properly() High Speed Network Laboratory @ NCTU is. Pass = true; Activity a = new Activity(); try { a. dismiss. Dialog(Test. Value); is. Pass = false; }catch (Illegal. Argument. Exception e){ } 45

Coverage verification Debug. start. Method. Tracing("test"); Debug. stop. Method. Tracing(); Generate HTML report Tcl script Execute Applicatoin Parsing class and method Generate test. trace file dmtracedump High Speed Network Laboratory @ NCTU 46

Compatibility test study 1. Modify Android source code 2. Build new image 3. Load to android handheld Design 3 conditions 1. Method disappear 2. Method’s parameter inconsistency 3. Version issue Load App Observe Result High Speed Network Laboratory @ NCTU 47

Experiment n n Market place Android handheld High Speed Network Laboratory @ NCTU 48

Contributions n Using CTS tool we can ¨ Check if device conforms android SDK ¨ Avoid incompatible condition in applications High Speed Network Laboratory @ NCTU 49

Schedule n Study android SDK - before 9/30 Design CTS tool framework - before 10/31 Test cases design - before 11/30 Analysis the result of test - before 12/31 n Writing and revising paper - after 99/1/1 n n n High Speed Network Laboratory @ NCTU 50

References 1. 2. 3. 4. The Java™ Compatibility Kit (JCK) read-only source bundles (https: //jck. dev. java. net/) JT Harness Project (https: //jtharness. dev. java. net/) JCK 6 b(http: //www. openjdk. org/groups/conformance/docs/JCK 6 b. Users Guide/html/p 10. html) Android SDK(http: //developer. android. com/guide/index. html) High Speed Network Laboratory @ NCTU 51