Language Extensions for MATLAB Amina Aslam Toheed Aslam

Language Extensions for MATLAB Amina Aslam Toheed Aslam Andrew Casey Maxime Chevalier- Boisvert Jesse Doherty Anton Dubrau Rahul Garg Maja Frydrychowicz Nurudeen Lameed Jun Li Soroush Radpour Olivier Savary Laurie Hendren Mc. Gill University Leverhulme Visiting Professor Department of Computer Science University of Oxford 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 Intro - 1

Overview • How does one make language extensions for MATLAB using Mc. Lab? • Meta. Lexer • Aspects for MATLAB • ["Types" for MATLAB] 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 Intro - 2

Mc. Lab Extensible Front-end. m. m source MATLABto-Natlab Scanner (Meta. Lexer) Parser (Beaver) AST attributes, rewrites (Jast. Add) XML 11/10/2020 Attributed AST Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 Aspect. Matlab Other 3

Meta. Lexer • • 11/10/2020 Modular Lexer Generator M. Sc. thesis, Andrew Casey AOSD 2011 www. sable. mcgill. ca/metalexer Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 Intro - 4

Given a front-end specification for a language (i. e. MATLAB), current method to implement a front-end for an extension of that language (i. e. Aspect. Matlab)? Lexical specification for original language Grammar and actions for original language 11/10/2020 Modified lexical specification for extended language Grammar and actions for original language Grammar rules for extension Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 5

Desired Modular Meta. Lexer Approach Lexical rules for extension Lexical specification for original language Grammar and actions for original language 11/10/2020 Lexical specification for original language Grammar and actions for original language Grammar rules for extension Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 6

We also want to be able to combine lexical specifications for diverse languages. • Java + HTML • Java + Aspects (Aspect. J) • Java + SQL • MATLAB + Aspects (Aspect. Matlab) 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 7

Scanning Aspect. J 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 8

Would like to be able to reuse and extend lexical specification modules • Nested C-style comments • Javadoc comments • Floating-point constants • URL • regular expressions • … 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 9

First, let’s understand the traditional lexer tools (lex, flex, jflex). • programmer specifies regular expressions + actions • tools generate a finite automatonbased implementation • states are used to handle different language contexts 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 10

JFlex Lexing Structure Specification in one file. � Lexing rules associated with a state. � Changing states associated with action code. 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 11

Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 12

Current (ugly) method for extending jflex specifications - copy&modify Copy jflex specification. � Insert new scanner rules into copy. �Order of rules matters! � Introduce new states and action logic for converting between states. � Principled way of weaving new rules into existing rules. � Modular and abstract notion of state and changing between states. � 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 13

Meta. Lexer Structure Each component specified in its own file. Layout specified in its own file. Components define lexing rules associated with a state, rules produce meta-tokens. � Layout defines transitions between components, state changes by metalexer (regular expressions + matching pairs of start/end symbols). � 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 14

Example Structure of a Meta. Lexer Specification for MATLAB Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 15

Extending a Meta. Lexer Specification for Matlab Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 16

Sharing component specifications with Meta. Lexer Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 17

Scanning a properties file Properties Key Value Util_Patterns 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 18

util_properties. mlc helper component 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 19

key. mlc component 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 20

value. mlc component 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 21

properties. mll layout 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 22

Meta. Lexer is implemented and available: www. sable. mcgill. ca/metalexer properties. mll util_patterns. mlc Meta. Lexer properties. jflex key. mlc value. mlc 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 23

Key problems to solve: • How to implement the meta-token lexer? • How to allow for insertion of new components, replacing of components, adding new embeddings (metalexer transitions). • How to insert new patterns into components at specific points. 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 24

Implementing the meta-token lexer Recognize a meta -pattern, i. e. when to go to a new component and when to return. Recognize the matching suffix. 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 25

Implementing inheritance (structured weaving). Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 26

Implementing Meta. Lexer layout inheritance • Layouts can inherit other layouts • %inherit directive put at the location at which the inherited transition rules (embeddings) should be placed. • each %inherit directive can be followed by: • %unoption • %replace • %unembed • new embeddings 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 27

Implementing Meta. Lexer component inheritance 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 28

Weaving in an inherited component Woven output New Component adds some rules and inherits original component. Original Component O 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 29

Results: Applied to three projects with complex scanners: • Aspect. J (abc and extensions) • Matlab (Annotations and Aspect. Matlab extensions) • Meta. Lexer 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 30

Aspect. J and Extensions 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 31

Meta. Lexer scanner implemented in Meta. Lexer • 1 st version of Meta. Lexer written in JFlex, one for components and one for layouts. • 2 nd version implemented in Meta. Lexer, many shared components between the component lexer and the layout lexer. 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 32

Related Work for Meta. Lexer • Ad-hoc systems with separate scanner/ LALR parser – Polyglot – Jast. Add – abc • Recursive-descent scanner/parser – ANTLR and systems using ANTLR • Scannerless systems – Rats! (PEGs) • Integrated systems – Copper (modified LALR parser which communicates with DFAbased scanner) 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 33

Metalexer Conclusions • Meta. Lexer allows one to specify modular and extensible scanners suitable for any system that works with JFlex. • Two main ideas: meta-lexing and component/layout inheritance. • Used in large projects such as abc, Mc. Lab and Meta. Lexer itself. • Available at: www. sable. mcgill. ca/metalexer 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 34

Aspect. Matlab • Simple Aspect-Oriented extension to MATLAB • M. Sc. thesis, Toheed Aslam • Analysis by Jesse Doherty, applications by Anton Dubrau, extensions by Olivier Savary-Belanger • AOSD 2010 • www. sable. mcgill. ca/mclab 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 Intro - 35

Why Aspect. Matlab? • • Aspect. J Aspect. C++ Aspect. Cobol … 11/10/2020 • Test the Mc. Lab framework for extensibility • Bring a simple and relevant version of AOP to scientists. • simple language constructs • focus on arrays and loops Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 36

What is an Aspect? Event Observer • Pattern specifying events to match. • Action to do before, after or around the matched events. • Action can use context information from the matched event. 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 37

Example: Profiling Array Sparsity 00090000005000 0100000300 0000400000 0070000000 11/10/2020 • Capture the sparsity and size at each operation on the whole array. • Capture the number of indexed references to each array. • Print out a summary for each array, allowing the programmer to identify good candidates to implement as sparse arrays. Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 38

Background - MATLAB Class classdef my. Class properties data … count = 0; end methods … end 11/10/2020 helper functions function x=get. Count(this) x = this. count; end Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 39

Aspect Definition aspect my. Aspect properties … end data count = 0; methods … end patterns … end actions … end 11/10/2020 helper functions function x=get. Count(this) x = this. count; end pointcuts foocalls : call(foo); advice foocounter : before foocalls this. count = this. count + 1; end Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 40

Function and Operator Patterns p. Call. Foo : call(foo); p. Exec. Bar : execution(bar); p. Call. Foo 2 args : call(foo(*, *)); p. Execution. Main : mainexecution(); end patterns plus. Op : op(+); times. Op : op(. *) || op(*); matrix. Ops: op(matrix); all. But. Minus: op(all) & ~op(-); end 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 41

Array Patterns a(i) = b(j, k) also, new value Context Info name indices object (value) line number location file name patterns p. Set. X : set(a); p. Get. X : get(b); array. Set : set(*); array. Whole. Get : get(*()); array. Indexed. Get : get(*(. . )); end 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 42

Loop Patterns t 1 = [1, 3, 5, 7, 9, …, n]; for it 2 = 1: 2: n = 1: numel(t 1) i = t 1(t 2); … … … end patterns p. Loop. I : loop(i); p. Loop. Head. I : loophead(i); p. Loop. Body. I : loopbody(i); end 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 43

Scope Patterns p. Within. Foo : within(function, foo); p. Within. Bar : within(script, bar); p. Within. My. Class : within(class, my. Class); p. Within. Loops : within(loops, *); p. Within. All. Abc : within(*, abc); end 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 44

Compound Patterns • Logical combinations of primitive patterns p. Call. Foo : call(foo) & within(loops, *); p. Get. Or. Set : (get(*) | set(*)) & within(function, bar); end 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 45

Before & After Actions a. Count. Call : before p. Call this. count = this. count + 1; disp(‘calling a function’); end a. Execution : after execution. Main total = this. get. Count(); disp([‘total calls: ’, num 2 str(total)]); end 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 46

Context Exposure actions a. Count. Call : before p. Call : (name, args) this. count = this. count + 1; disp([‘calling ’, name, ‘ with args(’, args, ‘)’]); end a. Execution : after execution. Main : (file) total = this. get. Count(); disp([‘total calls in ’, file, ‘: ’, num 2 str(total)]); end 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 47

Around Actions actcall : around p. Call. Foo : (args) disp([‘before foo call with args(’, args , ‘)’]); proceed(); disp([‘after foo call with args(’, args , ‘)’]); end actions actcall : around p. Call. Foo : (args) % proceed not called, so varargout is set varargout{1} = bar(args{1}, args{2}); end 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 48

Actions Weaving Order foo(); 11/10/2020 actions before 1 : before p. Call. Foo before 1(); … around 1 : around p. Call. Foo around 1(); … after 1 : after p. Call. Foo after 1(); … before 2 : before p. Call. Foo before 2(); … around 2 : around p. Call. Foo around 2(); … after 2 : after p. Call. Foo after 2(); … end Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 49

Compiler Structure Base. m Matlab Woven Base Matlab . m Aspects Front-end Matlab Impl. of Aspects Post-processing AST (Matlab+Aspects) Woven AST Separator Matcher & Weaver Aspect. Matlab Compiler Matlab AST Transformations Aspect. Info Name Resolution Analysis Resolved Name Set Simplified AST 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 50

Name Resolution Analysis patterns p. Call. Foo : call(foo); p. Get. Foo : get(foo); end actions before 1 : before p. Call. Foo … before 2 : before p. Get. Foo … end before 1(); before 2(); if is. Fun(foo) is. Var(foo) foo(); function 11/10/2020 foo(); variable Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 foo(); unresolved 52

Scientific Use Cases • Domain-Specific Profiling of Programs – Tracking array sparsity – Tracking array size-growing operations – Counting floating-point operations • Extending Functionality – Interpreting loop iteration space – Adding units to computations 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 53

Related Work for Aspect. Matlab • Aspect. J (Kiczales et al. , ECOOP '01) – abc (The de Moor and Hendren gang, AOSD '05) – Array pointcuts (Chen et al. , JSES '07) – Loop pointcuts (Harbulot et al. , AOSD '06) • Aspect. Cobol (Lammel et al. , AOSD '05) • Domain-Specific Aspects in Matlab (Cardoso et al. , DSAL workshop held at AOSD '10) 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 54

Conclusions • Mc. Lab supports extensions to MATLAB • We developed Meta. Lexer to support modular and extensible lexers, and then used it in Mc. Lab. • We designed and implemented Aspect. Matlab as an exercise in using Mc. Lab for extensions, and also to provide simple and relevant AOP for scientists. 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 55

Typing Aspects • Types for MATLAB, somewhat in the spirit of aspects. • Designed by what programmers might want to say. • Checked at run-time, but some static analysis could be done. 11/10/2020 Mc. Lab, Laurie Hendren, Leverhulme Lecture #2 Intro - 56

Simple Example MATLAB function 57

58

59

MATLAB prorammers often expect certain types 60

61

High-level types in MATLAB 62

63

Simple Example 64

65

Capturing reflective information • a. type • a. value • a. dims • a. basetype 66

Capturing dimensions and basetype • <n> can be used as a dimension spec • value of n is instantiated from the runtime dimension • repeated use in same atype statement implies equality 67