A Framework for Writing Trigger Action Todo Comments

A Framework for Writing Trigger -Action Todo Comments in Executable Format Pengyu Nie, Rishabh Rai, Junyi Jessy Li, Sarfraz Khurshid, Raymond J. Mooney, and Milos Gligoric August 29, 2019 ESEC/FSE 2019 Tallinn, Estonia supported by

Motivation (1) From Google Guava protected Abstract. Streaming. Hasher(int chunk. Size, int buffer. Size) { // TODO(kevinb): check more preconditions // (buffer. Size >= chunk. Size) if this is ever public check. Argument(buffer. Size % chunk. Size == 0); . . . } Trigger The method is public Action check. Argument(buffer. Size >= chunk. Size); The comment is written in natural language, thus a developer needs to … • locate the comment • check if the trigger evaluates to true • execute the action Tedious and error-prone 2

Motivation (2) From Apache Struts 2012/11: The comment was added 2015/5: The trigger was satisfied 2015/10: The action was executed 2017/10: The comment was removed void test. Dynamic. Attributes. Support() throws Exception {. . . // TODO : remove expected. JDK 15 and if() after switching to Java 1. 6 String expected. JDK 15 = "<input type="text". . . "; String expected. JDK 16 = "<input type="text". . . "; if (result. contains("foo="bar". . . ")) assert. Equals(expected. JDK 15, result); else assert. Equals(expected. JDK 16, result); . . . } Trigger switch to Java 1. 6 Action remove a variable and if statement 3

Motivation (3) • Trigger-action comments are frequently used • We have found hundreds of trigger-action comments in 10 open-source projects • Trigger-action comments are 10+% of developer comments • Like any other comments written in natural language, they frequently become outdated or forgotten • We contacted 10 developers from large software companies and asked if they write trigger-action comments; 7 (out of 8 replies) answered yes • They confirmed they have a lot of trigger-action comments, and lack of automation is a big issue 4

Trig. It • A framework for writing trigger-action comments as specifications in executable format • Developers write triggers and actions in the Trig. It DSL • Embedded in the host language (Java) • Utilizes native syntax (if-statements) with different semantics • Trig. It checks the triggers after compilation and before execution • If a trigger satisfies, Trig. It executes the associated actions • Trig. It removes all its specifications before execution 5

Example Trig. It Specification (1) From Google Guava protected Abstract. Streaming. Hasher(int chunk. Size, int buffer. Size) { // TODO(kevinb): check more preconditions // (buffer. Size >= chunk. Size) if this is ever public if (trig. It. Is. Public()) check. Argument(buffer. Size >= chunk. Size); check. Argument(buffer. Size % chunk. Size == 0); . . . } @Trigt. It. Method boolean trig. It. Is. Public() { return Trig. It. get. Method("<init>", int. class). is. Public(); } 6

Example Trig. It Specification (2) From Apache Struts void test. Dynamic. Attributes. Support() throws Exception {. . . // TODO : remove expected. JDK 15 and if() after switching to Java 1. 6 if (Trig. It. get. Java. Version(). ge(Trig. It. JAVA 6)) { String expected. JDK 16 = "<input type="text". . . "; assert. Equals(expected. JDK 16, result); } else { String expected. JDK 15 = "<input type="text". . . "; String expected. JDK 16 = "<input type="text". . . "; if (result. contains("foo="bar". . . ")) assert. Equals(expected. JDK 15, result); else assert. Equals(expected. JDK 16, result); }. . . } 7

Design Choices • Preserves semantics of comments from the execution point • Each Trig. It specification is local to its context similar to inline comments • Trig. It specifications are removed from the compiled code (after Trig. It's processing), thus they do not cause side-effects during execution • Java compiler checks partial correctness of Trig. It specifications • Integrated as a bot that sends a notification to developers or a code review with changes when a trigger evaluates to true 8

Trig. It DSL • Query Expression / Trigger • Action Statement / Global Action • Local Action • Trig. It Method 9

Trig. It DSL (1/4) • Query Expression / Trigger if ( <query expression> ) { // action } • Examples • • AST: Trig. It. get. Classes(). find. Any(“C”). is. Present() Build: Trig. It. get. Java. Version(). ge(Trig. It. JAVA 8) Issue: Trig. It. is. Closed(“github…issues/1897”) Time: Tirg. It. after(2019, 4, 2) 10

Trig. It DSL (2/4) • Action Statement / Global Action @Trig. It. Method void global. Action() { if (/* trigger */) { <action statement> } } • Examples • Trig. It. get. Method(m 1()). remove() • Trig. It. get. Field(f). set. Private() 11

Trig. It DSL (3/4) • Local Action if (/* trigger */) { /* code executed if trigger evaluates to true */ } else { /* code executed if trigger evaluates to false */ } 12

Trig. It DSL (4/4) • Trig. It Method • @Trig. It. Method annotation • void return type: specifies trigger + global action(s) @Trig. It. Method void trig. It. Deprecate. Make. Checked() { if (Trig. It. after(2018, 4, 30)) Trig. It. get. Method. By. Name("immediate. Checked. Future"). remove(); } • boolean return type: specifies a trigger that can be reused @Trigt. It. Method boolean trig. It. Is. Public() { return Trig. It. get. Field(f). is. Public(); } if (trig. It. Is. Public()) { /* action 1 */ } if (trig. It. Is. Public()) { /* action 2 */ } 13

Trig. It Workflow . java Files Compiler . class Files Test/ Execution 14

Trig. It Workflow Trig. It . java Files Compiler . class Files Test/ Execution 15

Trig. It Workflow Trig. It . java Files Compiler . class Files rewrite Trig. It specifications Test/ Execution • Rewrite • removes all other code and only keeps Trig. It specifications 16

Trig. It Workflow Trig. It . java Files • Rewrite Compiler . class Files rewrite Trig. It specifications evaluate Test/ Execution AST Build Configurations Issue Tracker System Time • removes all other code and only keeps Trig. It specifications • Evaluate • checks all triggers 17

Trig. It Workflow Trig. It . java Files • Rewrite Compiler . class Files rewrite AST Build Configurations Issue Tracker System Time • removes all other code and only keeps Trig. It specifications • Evaluate Trig. It specifications evaluate Test/ Execution apply Warning Messages Patch to. java Files • checks all triggers • Apply • executes actions by changing the bytecode, sends warning messages, or changing source code • removes all Trig. It specifications 18

Evaluation • RQ 1: Can developers quickly implement correct triggers and actions using Trig. It? • RQ 2: What’s the build overhead of Trig. It? • RQ 3: What’s the overhead for writing Trig. It specifications over trigger-action comments, measured by number of tokens? 19

User Study Design • Can developers quickly implement correct triggers and actions using Trig. It? • Tasks • Read a brief tutorial of Trig. It syntax (10 min) • Migrate 3 trigger-action comments to Trig. It specifications • The first one is warm-up and not counted in results (without telling the participants) • 2 groups of tasks: • Warm-up, Task A from Google Guava, Task B from Apache Struts • Warm-up, Task C from Apache Ignite, Task D from Jenkinsci Jenkins • Details available on our webpage: cozy. ece. utexas. edu/trigit 20

User Study Participants • 2 groups of tasks: • Warm-up, Task A from Google Guava, Task B from Apache Struts • Warm-up, Task C from Apache Ignite, Task D from Jenkinsci Jenkins • 20 participants • 8 professionals (6 developers in large software companies, 2 researchers) • 12 students (4 undergrads, 8 Ph. D students) Task A+B Task C+D Professionals 6 2 Students 6 6 21

User Study Results Task Time (min) Confidence Correctness Grouped By Avg. Median Trigger Action Syntax A 4. 7 4. 5 4. 2 4. 5 12/12 9/12 B 4. 3 5. 0 4. 4 4. 5 12/12 C 7. 6 7. 5 3. 9 4. 0 8/8 7/8 8/8 D 7. 2 6. 5 4. 2 4. 0 7/8 8/8 22

User Study Results Confidence Correctness Avg. Median Trigger Action Syntax A 4. 7 4. 5 4. 2 4. 5 12/12 9/12 B 4. 3 5. 0 4. 4 4. 5 12/12 C 7. 6 7. 5 3. 9 4. 0 8/8 7/8 8/8 D 7. 2 6. 5 4. 2 4. 0 7/8 8/8 Professionals 5. 0 4. 5 5. 0 16/16 15/16 Students 6. 1 5. 5 4. 0 23/24 24/24 22/24 5. 7 5. 0 4. 2 4. 0 39/40 37/40 Task Role Time (min) Grouped By All Participants obtained correct solutions with very little training 23

Build Overhead • RQ 2: What’s the build overhead of Trig. It? • Subjects: We migrated 44 trigger-action comments from 10 projects to Trig. It specifications. • Environment: Intel i 7 -6700 U@3. 40 GHz, 16 GB RAM, Ubuntu 18. 04 LTS • Results: average 8. 85%, median 6. 63%, min 1. 91%, max 26. 75% 24

Complexity of the Trig. It DSL • RQ 3: What’s the overhead for writing Trig. It specifications over trigger-action comments, measured by number of tokens? • Results: 58. 74% • • • Trigger-action comment: average 12. 2, median 12. 0, min 6, max 26 Trig. It specifications: average 18. 0, median 16. 5, min 11, max 36 Trigger: average 7. 3, median 6. 0, min 4, max 19 Action: average 7. 4, median 5. 0, min 2, max 27 Boilerplate code: average 3. 3, median 3. 0, min 3, max 7 Boilerplate code Trigger Action 25

Anecdotal Experience • During our inspection of trigger-action comments in open source projects, we observed some comments are forgotten, i. e. , the trigger is satisfied but the action is not executed • We reported 6 cases and the developers responded quickly • Some triggers and actions in comments are not specified enough: “Unfortunately Java 8 still causes some issues with some Google internal infrastructure. Java 8 is allowed in tests but not in the main part of the code (yet). ” • Knowing the details of the projects, e. g. , internal testing reports, will help writing correct Trig. It specifications 26

Conclusion • A framework for writing trigger-action comments in executable format • Introduces DSL for triggers and actions • Preserves semantics and developers’ workflow • Trig. It is easy to learn and introduces small overhead • Trig. It can run as a bot that sends a notification to developers or a code review with changes when a trigger evaluates to true • Webpage: cozy. ece. utexas. edu/trigit Pengyu Nie <pynie@utexas. edu> 27

Backup slides after this point 28

User Study Participants’ Prior Experience • Programming experience: average 8. 4 years • Java skills (on a scale of 1 -5): 3. 7 • None of the participants has contributed to the open-source projects where we selected the tasks from 29

User Study Error Analyze • Trigger error: 1 & action error: 1 • Misunderstanding the trigger-action comments • Syntax errors: 3 • Putting the local action in a wrong place • We added additional checks in Trig. It to prevent this kind of errors protected Abstract. Streaming. Hasher(int chunk. Size, int buffer. Size) { // TODO(kevinb): check more preconditions // (buffer. Size >= chunk. Size) if this is ever public check. Argument(buffer. Size % chunk. Size == 0); . . . } @Trigt. It. Method boolean trig. It. Is. Public() { if (Trig. It. get. Method("<init>", int. class). is. Public()) check. Argument(buffer. Size >= chunk. Size); } 30

Example Trig. It Specifications (3) From Elasticsearch public void test. Shadow. Replica. Natural. Relocation() throws Exception { … // TODO: uncomment the test below when https: //github. com/elasticsearch/issues/17695 // is resolved. if (Trig. It. is. Closed("https: //github. com/elasticsearch/issues/17695")) assert. Indices. Dirs. Deleted(nodes); } 31

Example Trig. It Specifications (4) From Google Guava // TODO(b/72241893): Remove by 2018 -04 @Deprecated public static <V, X extends Exception> Checked. Future<V, X> immediate. Checked. Future( @Nullable. Decl V value) { return new Immediate. Successful. Checked. Future<>(value); } @Trig. It. Method void trig. It. Deprecate. Make. Checked() { if (Trig. It. after(2018, 4, 30)) Trig. It. get. Current. Class(). get. Method. By. Name("immediate. Checked. Future"). remove(); } 32

Study of Trigger-Action Comments • We inspected 471 trigger-action comments and classify the trigger/action’s specificity • high: migration feasible • medium: migration could potentially be done • low: hard to understand, migration is not feasible • 28% have both specific trigger and specific action • 70% have either specific trigger or specific action 33

Trig. It Implementation • Implemented as a standalone Java library, can be easily integrated into existing build process • Command line interface • Maven plugin • Implemented with Java bytecode manipulation, using ASM • Extra checks for the correctness of triggers and actions • Trig. It. get. Class("C"). get. Field("f"). is. Private() Trig. It will check if class C and field f exist • Debug options: --assume-true, --no-action 34

Using Bytecode Analysis & Manipulation • The bytecode format was more stable in recent years than the Java language syntax; this may lower the cost of maintenance. • We wanted to avoid dependencies on IDEs and enable integration of Trig. It with any build process, even if the build is running on a continuous integration service. Future versions of Trig. It may have a tighter connection with IDEs based on the adoption and feedback that we received from the community. 35

Example Patch (Before Trigger Satisfies) From Google Guava protected Abstract. Streaming. Hasher(int chunk. Size, int buffer. Size) { if (trig. It. Is. Public()) check. Argument(buffer. Size >= chunk. Size); check. Argument(buffer. Size % chunk. Size == 0); . . . } @Trigt. It. Method boolean trig. It. Is. Public() { return Trig. It. get. Method("<init>", int. class). is. Public(); } 36

Example Patch (The Patch) From Google Guava protected Abstract. Streaming. Hasher(int chunk. Size, int buffer. Size) { - if (trig. It. Is. Public()) - check. Argument(buffer. Size >= chunk. Size); + check. Argument(buffer. Size >= chunk. Size); check. Argument(buffer. Size % chunk. Size == 0); . . . } -@Trigt. It. Method -boolean trig. It. Is. Public() { - return Trig. It. get. Method("<init>", int. class). is. Public(); -} 37

Related Work • Pengyu Nie, Junyi Jessy Li, Sarfraz Khurshid, Raymond Mooney, and Milos Gligoric. 2018. Natural language processing and program analysis for supporting todo comments as software evolves. In Workshops of the AAAI Conference on Artificial Intelligence. 775– 778. • Margaret-Anne Storey, Jody Ryall, R. Ian Bull, Del Myers, and Janice Singer. 2008. TODO or to bug. In International Conference on Software Engineering. 251– 260. • Giriprasad Sridhara. 2016. Automatically detecting the up-to-date status of To. Do comments in Java programs. In India Software Engineering Conference. 16– 25. • Dorsaf Haouari, Houari Sahraoui, and Philippe Langlais. 2011. How good is your comment? A study of comments in Java programs. In International Symposium on Empirical Software Engineering and Measurement. 137– 146. • Lin Tan, Ding Yuan, Gopal Krishna, and Yuanyuan Zhou. 2007. /*i. Comment: bugs or bad comments? */. In Symposium on Operating Systems Principles. 145– 158. • Shin Hwei Tan, Darko Marinov, Lin Tan, and Gary T. Leavens. 2012. @t. Comment: Testing Javadoc comments to detect comment-code inconsistencies. In International Conference on Software Testing, Verification, and Validation. 260– 269. 38