Deep Reinforcement Learning for Syntactic Error Repair in

Deep Reinforcement Learning for Syntactic Error Repair in Student Programs Rahul Gupta, Aditya Kanade, Shirish Shevade Computer Science & Automation Indian Institute of Science Bangalore, India AAAI 2019

Problem • Novice students often struggle with the formal syntax of programming languages • Providing real time personalized feedback is infeasible for instructors in massive open online courses • Most existing automated feedback generation techniques focus on semantic errors • Compiler error messages do not always localize the errors accurately and are often difficult to understand

Example

Objective • Develop a syntactic error repair technique that can assist novice programmers by automatically correcting common syntactic errors in programs Solution strategy • Error repair as a game • initial state – incorrect program • actions – navigation and editing • goal state – the corrected version

Our contributions • Programming language (PL) correction framework amenable to RL • Training through self-exploration, i. e. , without any supervision • Outperforms fully supervised, SOTA technique, Deep. Fix, on a publicly available dataset of thousands of C programs with typographic errors • 5 X accelerated training using expert demonstrations for only a small fraction of training dataset The code is available at https: //bitbucket. org/iiscseal/rlassist

PL correction framework State = <string, cursor> Init State

PL correction framework Init State = <string, cursor> Apply Navigation actions Action Edit actions Compiler Update Actions & Transitions 1. Reduces combinatorial state space by rejecting spurious edits 2. Also prevents arbitrary changes

PL correction framework Init State = <string, cursor> Action Apply Navigation actions Edit actions Compiler New state Update Reward Episode Termination 1. Step penalty 2. Error resolution reward 3. Complete fix reward

PL correction framework Init State = <string, cursor> Action Apply Navigation actions Edit actions Compiler New state Update Reward Episode Termination 1. Reaching goal state 2. Reaching end of program 3. Reaching max episode length

State as a sequence of tokens PL correction framework State Encoder Init State = <string, cursor> Action Apply Navigation actions Edit actions Compiler New state Update Reward State encoding Termination Sequence of tokens: • Lexemes • line breaks • cursor

Encoded state x 1 x 2 xn LSTM y 2 y 1 Model: A 3 C with LSTM encoder yn Mean Pooling Layer State embedding Linear Layer 1 Linear Layer 2 Softmax V(s) π (a|s) Loss Reward

RLAssist: video demonstration

Dataset • Publicly available dataset of C programs written by students for an introductory programming class • 93 diverse programming problems • Programs of length up to 450 tokens • Divided in 5 folds for cross validation. Each fold has non-overlapping sets of problems for training and testing

Training performance RLAssist+Demo RLAssist Plus 1. Expert demonstrations for 10% training data 2. Small edit penalty Episodes in millions 5 million episodes, 3 weeks 1. 6 million episodes, 4 days!

Test performance & comparison with Deep. Fix Dataset statistics Erroneous programs 6975 Error Msgs. 16766 Technique Avg. Tokens 203 Results Completely fixed programs Partially fixed programs Error messages resolved Deep. Fix 1625 (23. 3%) 1129 (16. 2%) 5156 (30. 8%) RLAssist 1699 (24. 4%) [ + 4. 6% ] 1310 (18. 8%) [ + 15. 2% ] 5884 (35. 1%) [ + 13. 8% ] RLAssist+Demo 1854 (26. 6%) [ + 14. 1% ] 1426 (20. 4%) [ + 26. 3% ] 6652 (39. 7%) [ + 29% ]

Program embedding visualization cursor is set to the first token of the line preceding the erroneous line cursor is set to the first token of the erroneous line cursor is set to the error location after the program has been fixed PCA projection of embeddings of 350 programs in three different states

Limitations & Future work • RLAssist is slower to train but can be trained much faster with expert demonstrations • RLAssist is programming language agnostic but has only been evaluated on C programs. In future, we will experiment with other programming languages as well • We plan to extend RLAssist to target more classes of errors, and develop learning algorithms that can learn and exploit deeper syntactic and semantic properties of programs

Conclusion • We address the problem of syntactic error repair in student programs and present RLAssist, a deep reinforcement learning based end-to-end solution • We empirically show that RLAssist outperforms a fully supervised state-ofthe-art syntactic error repair technique, Deep. Fix, on a publicly available dataset of thousands of C programs without using any labelled data for training • Moreover, we show that RLAssist trains much faster and converges to a better policy when expert demonstrations are available for as little as one tenth of its training data

Acknowledgments • We thank Sonata Software for partially funding this work • We also thank Google for a student travel grant to support travel to present this work