29 Clang Tutorial CS 453 Automated Software Testing

/ 29 Clang Tutorial CS 453 Automated Software Testing

Clang Tutorial, CS 453 Automated Software Testing 1 / 29 Overview • Clang is a library to convert a C program into an abstract syntax tree (AST) and manipulate the AST • Ex) finding branches, renaming variables, pointer alias analysis, etc • Example C code • 2 functions are declared: my. Print and main • main function calls my. Print and returns 0 • my. Print function calls printf • my. Print contains if and for statements • 1 global variable is declared: global //Example. c #include <stdio. h> int global; void my. Print(int param) { if (param == 1) printf("param is 1"); for (int i = 0 ; i < 10 ; i++ ) { global += i; } } int main(int argc, char *argv[]) { int param = 1; my. Print(param); return 0; }

Clang Tutorial, CS 453 Automated Software Testing 2 / 29 Example AST • Clang generates 3 ASTs for my. Print(), main(), and global • A function declaration has a function body and parameters ASTs for my. Print() AST for global ASTs for main()

Clang Tutorial, CS 453 Automated Software Testing 3 / 29 Structure of AST • Each node in AST is an instance of either Decl or Stmt class • Decl represents declarations and there are sub-classes of Decl for different declaration types • Ex) Function. Decl class for function declaration and Parm. Var. Decl class for function parameter declaration • Stmt represents statements and there are sub-classes of Stmt for different statement types • Ex) If. Stmt for if and Return. Stmt class for function return

Clang Tutorial, CS 453 Automated Software Testing 4 / 29 Decl (1/4) Function declaration • A root of the function AST is a Decl node • A root of function AST is an instance of Function. Decl which is a sub-class of Decl 14 int main(int argc, 15 char *argv[]) { 16 int param = 1; 17 my. Print(param); 18 return 0; } Legend

Clang Tutorial, CS 453 Automated Software Testing 5 / 29 Decl (2/4) • Function. Decl can have an instance of Parm. Var. Decl for a function parameter and a function body • Parm. Var. Decl is a child class of Decl • Function body is an instance of Stmt • In the example, the function body is an instance of Compound. Stmt which is a sub-class of Stmt Function parameter declarations Function body 14 int main(int argc, char *argv[]) { 15 int param = 1; 16 my. Print(param); 17 return 0; 18 } Legend

Clang Tutorial, CS 453 Automated Software Testing 6 / 29 Decl (3/4) • Var. Decl is for a local and global variable declaration • Var. Decl has a child if a variable has a initial value • In the example, Var. Decl has Integer. Literal Global variable declaration Initial value Local variable declaration 14 int main(int argc, char *argv[]) { 15 int param = 1; 16 my. Print(param); 17 return 0; 18 } Legend

Clang Tutorial, CS 453 Automated Software Testing 7 / 29 Decl (4/4) • Function. Decl, Parm. Var. Decl and Var. Decl have a name and a type of declaration • Ex) Function. Decl has a name ‘main’ and a type ‘void (int, char**)’ Types 14 int main(int argc, char *argv[]) { 15 int param = 1; 16 my. Print(param); 17 return 0; 18 } Names Types Legend

Clang Tutorial, CS 453 Automated Software Testing 8 / 29 Stmt (1/9) • Stmt represents a statements • Subclasses of Stmt • Compound. Stmt class for code block • Decl. Stmt class for local variable declaration • Return. Stmt class for function return Statements 14 int main(int argc, char *argv[]) { 15 int param = 1; 16 my. Print(param); 17 return 0; 18 } Legend

Clang Tutorial, CS 453 Automated Software Testing 9 / 29 Stmt (2/9) • Expr represents an expression (a subclass of Stmt) • Subclasses of Expr • Call. Expr for function call • Implicit. Cast. Expr for implicit type casts • Decl. Ref. Expr for referencing declared variables and functions • Integer. Literal for integer literals Expressions (also statements) 14 int main(int argc, char *argv[]) { 15 int param = 1; 16 my. Print(param); 17 return 0; 18 } Legend

Clang Tutorial, CS 453 Automated Software Testing 10 / 29 Stmt (3/9) • Stmt may have a child containing additional information • Compound. Stmt has statements in a code block of braces (“{}”) int param = 1; my. Print(param); 14 int main(int argc, char *argv[]) { 15 int param = 1; 16 my. Print(param); 17 return 0; 18 } Legend return 0;

Clang Tutorial, CS 453 Automated Software Testing 11 / 29 Stmt (4/9) • Stmt may have a child containing additional information (cont’) • The first child of Call. Expr is for a function pointer and the others are for function parameters Declarations for Decl. Stmt Legend Function pointer for Call. Expr Function parameter for Call. Expr Return value for Return. Stmt

Clang Tutorial, CS 453 Automated Software Testing 12 / 29 Stmt (5/9) • Expr has a type of an expression • Ex) a node of Call. Expr has a type ‘void’ • Some sub-classes of Expr can have a value • Ex) a node of Integer. Literal has a value ‘ 1’ Legend Values Types Value Types

Clang Tutorial, CS 453 Automated Software Testing 13 / 29 Stmt (6/9) • my. Print function contains If. Stmt and For. Stmt in its function body 6 void my. Print(int param) { if (param == 1) 7 printf("param is 1"); 8 for (int i=0; i<10; i++) { 9 global += i; 10 } 11 12 }

Clang Tutorial, CS 453 Automated Software Testing 14 / 29 Stmt (7/9) Condition variable Condition • If. Stmt has 4 children • A condition variable in Var. Decl • In C++, you can declare a variable in condition (not in C) • A condition in Expr • Then block in Stmt • Else block in Stmt 6 void my. Print(int param) { if (param == 1) 7 printf("param is 1"); 8 for (int i = 0 ; i < 10 ; i++ ) { 9 global += i; 10 } 11 12 } Then block Else block

Clang Tutorial, CS 453 Automated Software Testing 15 / 29 Stmt (8/9) Initialization Condition variable Condition • For. Stmt has 5 children • Initialization in Stmt • A condition variable in Var. Decl • A condition in Expr • Increment in Expr • A loop block in Stmt Increment 6 void my. Print(int param) { if (param == 1) 7 printf("param is 1"); 8 for (int i = 0 ; i < 10 ; i++ ) { 9 global += i; 10 } 11 12 } Loop block

Clang Tutorial, CS 453 Automated Software Testing 16 / 29 Stmt (9/9) • Binary. Operator has 2 children for operands • Unary. Operator has a child for operand Two operands for Binary. Operator A operand for Unary. Operator 6 void my. Print(int param) { if (param == 1) 7 printf("param is 1"); 8 for (int i = 0 ; i < 10 ; i++ ) { 9 global += i; 10 } 11 12 }

Clang Tutorial, CS 453 Automated Software Testing 17 / 29 Traversing Clang AST (1/3) • Parse. AST() starts building and traversal of an AST • The callback function Handle. Top. Level. Decl() in ASTConsumer is called for each top-level declaration • Handle. Top. Level. Decl() receives a list of function and global variable declarations as a parameter void clang: : Parse. AST (Preprocessor &pp, ASTConsumer *C, ASTContext &Ctx, …) • A user has to customize ASTConsumer 1 class My. ASTConsumer : public ASTConsumer 2 { public: 3 My. ASTConsumer(Rewriter &R) {} 4 5 virtual bool Handle. Top. Level. Decl(Decl. Group. Ref DR) { 6 for(Decl. Group. Ref: : iterator b=DR. begin(), e=DR. end(); b!=e; ++b){ 7 … // variable b has each decleration in DR 8 } 9 return true; 10 } 11 12 };

Clang Tutorial, CS 453 Automated Software Testing 18 / 29 Traversing Clang AST (2/3) • Handle. Top. Level. Decl() calls Traverse. Decl() which recursively travel a target AST from the top-level declaration by calling Visit. Stmt (), Visit. Function. Decl(), etc. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 class My. ASTVisitor : public Recursive. ASTVisitor<My. ASTVisitor> { bool Visit. Stmt(Stmt *s) { Visit. Stmt is called when Stmt is encountered printf("t%s n", s->get. Stmt. Class. Name() ); return true; } bool Visit. Function. Decl(Function. Decl *f) { Visit. Function. Decl is called when if (f->has. Body()) { Function. Decl is encountered Stmt *Func. Body = f->get. Body(); printf("%sn", f->get. Name()); } return true; } }; class My. ASTConsumer : public ASTConsumer { virtual bool Handle. Top. Level. Decl(Decl. Group. Ref DR) { for (Decl. Group. Ref: : iterator b = DR. begin(), e = DR. end(); b != e; ++b) { My. ASTVisitor; Visitor. Traverse. Decl(*b); } return true; } … };

Clang Tutorial, CS 453 Automated Software Testing 19 / 29 Traversing Clang AST (3/3) • Visit. Stmt() in Recursive. ASTVisitor is called for every Stmt object in the AST • Recursive. ASTVisitor visits each Stmt in a depth-first search order • If the return value of Visit. Stmt is false, recursive traversal halts • Example: main function of the previous example Recursive. ASTVisitor will visit all nodes in this box (the numbers are the order of traversal) 1 2 5 3 4 6 7 8 9 10 11

Clang Tutorial, CS 453 Automated Software Testing Guideline for HW #2 • Initialization of Clang • Line number information of Stmt • Useful Functions 20 / 29

Clang Tutorial, CS 453 Automated Software Testing 21 / 29 Initialization of Clang • Initialization of Clang is complicated • To use Clang, many classes should be created and many functions should be called to initialize Clang environment • Ex) Complier. Instance, Target. Options, File. Manager, etc. • It is recommended to use the initialization part of the sample source code from the course homepage as is, and implement your own ASTConsumer and Recursive. ASTVisitor classes

Clang Tutorial, CS 453 Automated Software Testing 22 / 29 Line number information of Stmt • A Source. Location object from get. Loc. Start() of Stmt has a line information • Source. Manager is used to get line and column information from Source. Location • In the initialization step, Source. Manager object is created • get. Expansion. Line. Number() and get. Expansion. Column. Number() in Source. Manager give line and column information, respectively bool Visit. Stmt(Stmt *s) { Source. Location start. Location = s->get. Loc. Start(); Source. Manager &srcmgr=m_srcmgr; //you can get Source. Manager from the initialization part unsigned int line. Num = srcmgr. get. Expansion. Line. Number(start. Location); unsigned int col. Num = srcmgr. get. Expansion. Column. Number(start. Location); … }

Clang Tutorial, CS 453 Automated Software Testing 23 / 29 Useful Functions • dump() and dump. Color() in Stmt and Function. Decl to print AST • dump() shows AST rooted at Stmt or Function. Decl object • dump. Color() is similar to dump() but shows AST with syntax highlight • Example: dump. Color() of my. Print Function. Decl 0 x 368 a 1 e 0 <line: 6: 1> my. Print 'void (int)' |-Parm. Var. Decl 0 x 368 a 120 <line: 3: 14, col: 18> param 'int' `-Compound. Stmt 0 x 36 a 1828 <col: 25, line: 6: 1> `-If. Stmt 0 x 36 a 17 f 8 <line: 4: 3, line: 5: 24> |-<<<NULL>>> |-Binary. Operator 0 x 368 a 2 e 8 <line: 4: 7, col: 16> 'int' '==' | |-Implicit. Cast. Expr 0 x 368 a 2 d 0 <col: 7> 'int' <LValue. To. RValue> | | `-Decl. Ref. Expr 0 x 368 a 288 <col: 7> 'int' lvalue Parm. Var 0 x 368 a 120 'param' 'int' | `-Integer. Literal 0 x 368 a 2 b 0 <col: 16> 'int' 1 |-Call. Expr 0 x 368 a 4 e 0 <line: 5: 5, col: 24> 'int' | |-Implicit. Cast. Expr 0 x 368 a 4 c 8 <col: 5> 'int (*)()' <Function. To. Pointer. Decay> | | `-Decl. Ref. Expr 0 x 368 a 400 <col: 5> 'int ()' Function 0 x 368 a 360 'printf' 'int ()' | `-Implicit. Cast. Expr 0 x 36 a 17 e 0 <col: 12> 'char *' <Array. To. Pointer. Decay> | `-String. Literal 0 x 368 a 468 <col: 12> 'char [11]' lvalue "param is 1" `-<<<NULL>>>

Clang Tutorial, CS 453 Automated Software Testing 24 / 29 References • Clang, http: //clang. llvm. org/ • Clang API Documentation, http: //clang. llvm. org/doxygen/ • How to parse C programs with clang: A tutorial in 9 parts, http: //amnoid. de/tmp/clangtut/tut. html

Clang Tutorial, CS 453 Automated Software Testing Appendix: Example Source Code (1/4) • This program prints the name of declared functions and the class name of each Stmt in function bodies 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 Print. Functions. c #include <cstdio> #include <string> #include <iostream> #include <sstream> #include <map> #include <utility> #include #include #include #include "clang/ASTConsumer. h" "clang/AST/Recursive. ASTVisitor. h" "clang/Basic/Diagnostic. h" "clang/Basic/File. Manager. h" "clang/Basic/Source. Manager. h" "clang/Basic/Target. Options. h" "clang/Basic/Target. Info. h" "clang/Frontend/Compiler. Instance. h" "clang/Lex/Preprocessor. h" "clang/Parse. AST. h" "clang/Rewrite/Core/Rewriter. h" "clang/Rewrite/Frontend/Rewriters. h" "llvm/Support/Host. h" "llvm/Support/raw_ostream. h" using namespace clang; using namespace std; class My. ASTVisitor : public Recursive. ASTVisitor<My. ASTVisitor> { public: 25 / 29

Clang Tutorial, CS 453 Automated Software Testing Appendix: Example Source Code (2/4) 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 bool Visit. Stmt(Stmt *s) { // Print name of sub-class of s printf("t%s n", s->get. Stmt. Class. Name() ); return true; } bool Visit. Function. Decl(Function. Decl *f) { // Print function name printf("%sn", f->get. Name()); return true; } }; class My. ASTConsumer : public ASTConsumer { public: My. ASTConsumer() : Visitor() //initialize My. ASTVisitor {} virtual bool Handle. Top. Level. Decl(Decl. Group. Ref DR) { for (Decl. Group. Ref: : iterator b = DR. begin(), e = DR. end(); b != e; ++b) { // Travel each function declaration using My. ASTVisitor. Traverse. Decl(*b); } return true; } private: My. ASTVisitor; }; int main(int argc, char *argv[]) { 26 / 29

Clang Tutorial, CS 453 Automated Software Testing Appendix: Example Source Code (3/4) 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 if (argc != 2) { llvm: : errs() << "Usage: Print. Functions <filename>n"; return 1; } // Compiler. Instance will hold the instance of the Clang compiler for us, // managing the various objects needed to run the compiler. Compiler. Instance The. Comp. Inst; // Diagnostics manage problems and issues in compile The. Comp. Inst. create. Diagnostics(NULL, false); // Set target platform options // Initialize target info with the default triple for our platform. Target. Options *TO = new Target. Options(); TO->Triple = llvm: : sys: : get. Default. Target. Triple(); Target. Info *TI = Target. Info: : Create. Target. Info(The. Comp. Inst. get. Diagnostics(), TO); The. Comp. Inst. set. Target(TI); // File. Manager supports for file system lookup, file system caching, and directory search management. The. Comp. Inst. create. File. Manager(); File. Manager &File. Mgr = The. Comp. Inst. get. File. Manager(); // Source. Manager handles loading and caching of source files into memory. The. Comp. Inst. create. Source. Manager(File. Mgr); Source. Manager &Source. Mgr = The. Comp. Inst. get. Source. Manager(); // Prreprocessor runs within a single source file The. Comp. Inst. create. Preprocessor(); // ASTContext holds long-lived AST nodes (such as types and decls). The. Comp. Inst. create. ASTContext(); // A Rewriter helps us manage the code rewriting task. Rewriter The. Rewriter; 27 / 29

Clang Tutorial, CS 453 Automated Software Testing 28 / 29 Appendix: Example Source Code (4/4) 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 } The. Rewriter. set. Source. Mgr(Source. Mgr, The. Comp. Inst. get. Lang. Opts()); // Set the main file handled by the source manager to the input file. const File. Entry *File. In = File. Mgr. get. File(argv[1]); Source. Mgr. create. Main. File. ID(File. In); // Inform Diagnostics that processing of a source file is beginning. The. Comp. Inst. get. Diagnostic. Client(). Begin. Source. File(The. Comp. Inst. get. Lang. Opts(), &The. Comp. Inst. get. Preprocessor()); // Create an AST consumer instance which is going to get called by Parse. AST. My. ASTConsumer The. Consumer; // Parse the file to AST, registering our consumer as the AST consumer. Parse. AST(The. Comp. Inst. get. Preprocessor(), &The. Consumer, The. Comp. Inst. get. ASTContext()); return 0;