Template Haskell 14 years on Simon Peyton Jones

Template Haskell 14 years on Simon Peyton Jones Tim Sheard Microsoft Resaerch 2016

What is Template Haskell? An extremely successful meta-programming extension for Haskell Introduced in 2002, joint work with Tim Sheard (of Meta. ML fame). Widely used: over 900 (out of 8, 200) packages on Hackage

The basic idea cube : : Int -> Int cube = $(mk. Pow 3) You can write code (mk. Pow) that generates source Haskell …that gets spliced in where you invoke it $(mk. Pow 3) x -> x*x*x …and then is typechecked and compiled just as if you’d written that code in the first place

The basic idea mk. Pow’ : : Int -> Exp. Q mk. Pow’ 1 e = e mk. Pow’ n e = [| $e * $(mk. Pow’ (n-1) e) |] mk. Pow : : Int -> Exp. Q mk. Pow n = [| x -> $(mk. Pow’ n [| x |]) |] [| … |] is a quotation It generates a syntax tree of type Exp. Q $(…) is a splice It takes a syntax tree of type Exp. Q, and inserts it at that point

The basic idea printf : : String -> Exp. Q …$(printf “Name: %s; age: %d”) “Fred” 45… Splices are run at compile time only �No runtime code generation (c. f. Meta. ML) A bit like cpp, but more civilised; e. g. no syntax errors in expanded code What about �No “x is not in scope” errors? �No “Cannot match Int with Bool” errors?

Hygiene

Hygiene const : : Exp. Q -> Exp. Q const e = [| x -> $e |] -- In a library f a b = $(const [| a+b |]) -- ---> f a b = x -> a+b g x b = $(const [| x+b |]) -- ---> g x b = x -> x+b -- Eeek!

Hygiene const : : Exp. Q -> Exp. Q const e = [| x -> $e |] -- In a library f a b = $(const [| a+b |]) -- ---> f a b = x -> a+b g x b = $(const [| x+b |]) -- ---> g x b = x 0 -> x+b -- OK! Intuitively: the source code should determine the binding structure Anything else is fundamentally non-modular; may fail randomly

Hygiene Not enough just to draw quoted binders from a different alphabet const : : Exp. Q -> Exp. Q const e = [| x -> $e |] -- In a library f = [| x -> $(const [| x+1 |]) |] -- $(f) ---> x 0 -> x 1 -> x 0 + 1 Quoted binders must be “fresh” somehow

Staging

Staging constraints n. List : : Int -> Exp. Q n. List 0 e = [| [] |] n. List n e = [| $e : $(n. List (n-1) e) |] f x = $(n. List 3 [| True |]) ---> f x = [True, True] g x = $(n. List 3 [| x |]) ---> f x = [x, x, x]

Staging constraints f x = $(n. List 3 [| True |]) ---> f x = [True, True] g x = $(n. List 3 [| x |]) ---> f x = [x, x, x] h x = $(n. List x [| True |]) ---> ? ? ? The splice in ‘h’ runs at compile time; but x is not available until run time. Staging error! Easy to fix: type environment carries stage index

Cross stage persistence f 1 : : Int -> Exp. Q f 1 x = [| y -> x + y |] -- OK f 2 : : (Int, Int) -> Exp. Q f 2 x = [| y -> fst x + y |] -- OK f 3 : : [Int] -> Exp. Q f 3 xs = [| g -> map g xs |] -- OK f 4 : : (Int->Int) -> Exp. Q f 4 g = [| ys -> map g ys |] -- Not OK

Cross stage persistence f 1 : : Int -> Exp. Q f 1 x = [| y -> x + y |] -- OK means (driven by the cross-stage info) f 1 x = [| y -> $(lift x) + y |] class Liftable a where lift : : a -> Exp. Q instance Liftable Int where lift n =. . . wait and see. . . instance (Liftable a, Liftable b) => Liftable (a, b) where lift (x, y) = [| ( $(lift x), $(lift y) ) |]

A stage restriction module M where n. List : : Int -> Exp. Q n. List 0 e = [| [] |] n. List n e = [| $e : $(n. List (n-1) e) |] foo = $(n. List 3 True) We’d need to compile ‘n. List’ (to bytecode or machine code) before even typechecking ‘foo’. That’s awkward. Restriction: in a top-level splice you can only call imported top-level functions.

The fork in the road

The fork in the road 1. We want to generate only well-typed terms 2. We want � to generate patterns, types, declarations; � compile time reflection; � analyse code, not just generate code I don’t know how to achieve both at once So TH (now) does them separately

1. Typed Template Haskell The Walled Garden

Generating well-typed terms Horrible possibility: suppose not $(f True) generates an ill-typed expression, or one that does not have type Bool. Then error messages speak of the expanded code. Wanted: static guarantee of well-typed-ness

Generating well-typed terms Well-established solution (Meta. ML) �[|| e ||] : : TExp. Q t �$$(e) : : t if if e : : t e : : TExp. Q t If v : : TExp. Q t, then v is the syntax tree representing a well-typed term of type t f : : TExp. Q Int -> TExp. Q Int f x = [|| $$x + 1 ||] h y = $$(f [|| y ||])

Generating well-typed terms Well-established solution (Meta. ML) �[|| e ||] : : TExp. Q t �$$(e) : : t if if e : : t e : : TExp. Q t f : : TExp. Q Int -> TExp. Q Int f x = [| $x + 1 |] h y = $(f [| y |]) Splices must run during typechecking, so that we can ensure that y : : TExp. Q Int All is good!

Error messages n. List : : Int -> n. List n e | n == 0 = | n > 0 = | otherwise = TExp. Q -> TExp. Q [|| [] |] [|| $$e : $$(n. List (n-1) e) ||] error “Negative arg to n. List” Want a civilised error message with line number etc, not a compiler crash!

The quotation monad n. List : : Int -> n. List n e | n == 0 = | n > 0 = | otherwise = TExp. Q -> TExp. Q [|| [] ||] [|| $$e : $$(n. List (n-1) e) ||] fail “Negative arg to n. List” fail : : String -> Q a type TExp. Q t = Q (TExp t) The quotation monad The syntax tree type Q can be an exception monad, supporting civilised failure

Error reporting is. Ext. Enabled exts. Enabled location report recover fail : : : Extension -> Q Bool Q [Extension] Q Loc Bool -> String -> Q Q a -> Q a String -> Q a

Useful side effects run. IO add. Dependent. File add. Top. Decls add. Mod. Finalizer get. Q : : Typeable put. Q : : Typeable : : IO a -> Q a : : File. Path -> Q () : : [Dec] -> Q () : : Q () -> Q () a => Q (Maybe a) a => a -> Q ()

But Typed TH just isn’t expressive enough Expressions we can generate but not type: $(f 2) --> x 1. x 2. True $(f 3) --> x 1. x 2. x 3. True So f : : Int -> TExp. Q ? ? ? $(sel 1 3) --> x. case x of (a, b, c) -> a $(printf “%d, %s”) --> (x: : Int) (y: : String). show x ++ “, ” ++ show y

But Typed TH just isn’t expressive enough Other splicable entities Patterns f $(sel. Pat 3 1) = x --> f (x, _, _) = x Types f : : $(sel. Type 3 1) --> f : : (a, b, c) -> a Declarations $(gen. Sel 3 1) --> sel_3_1 (a, b, c) = a $(gen. Enum 3) --> data T 3 = D 1 | D 2 | D 3

2. Untyped Template Haskell The Wild West “I'm just using a quotation as a convenient way to build a syntax tree. Please don't even try to typecheck it; just wait until it is finally spliced into a top-level splice”

The Exp type Exp. Q = Q Exp data Exp = | | | Var. E Name Con. E Name Lit. E Lit App. E Exp Lam. E [Pat] Exp …etc… The syntax tree type Exp, Lit, Pat, etc are all algebraic data types representing Haskell source syntax Defined in Language. Haskell. Syntax

Quotations are just syntactic sugar [| $e : $(n. List (n-1) e) |] The syntax tree type means do { h <- e ; t <- n. List (n-1) e ; return (Con. E ‘: h t) } And you are perfectly at liberty to write the do-notation form instead of the quasi-quote ‘: is the TH Name of the (: ) operator

Fresh names new. Name : : String -> Q Name [| f -> map f $e |] Generate fresh name The syntaxa tree type means do { fn <- new. Name “f” -- fn : : Name Use it in ; t <- e pattern ; return (Lam. E [Var. P fn] (App. E (Var. E ‘map) (Var. E fn)) t)) } The Name of ‘map’ And in a term

Mixing it up new. Name : : String -> Q Name do { fn <- new. Name “f” -- fn : : The Name syntax tree type ; t <- e ; return (Lam. E [Var. P xn] (App. E (Var. E ‘map) (Var. E fn)) t)) } can also be written like this do { fn <- new. Name “f” ; body <- [| map $(Var. E fn) $e |] ; return (Lam. E [Var. P xn] body) } Can mix convenient quasi-quote notation with do-notation

Patterns, declarations, types type type Exp. Q Pat. Q Dec. Q Typ. Q TExp. Q t = = = Q Q Q Exp Pat Dec Typ (TExp t) The syntax tree type Plus quotation notation for �decls [d| data T = T 1 | T 2 |] �patterns [p| (x, y) |] �types [t| Int -> Int |] No hope of typechecking quotes [d| f : : $(g True); f $(h) = $(j) |]

Splices run in the renamer No hope of typechecking quotes [d| f : : $(g True); f $(h) = $(j) |] Splicing must happen in the renamer, to bring variables into scope let x = True in $(blah) -> x Sadly: give up on guarantee that spliced-in terms or declarations are type-correct and fit their context. Instead, rely on typechecking after splicing

Hygiene Unlike Scheme, we do rename/typecheck before running the splice. $(mlet x 3 (x+1)) $(mlet [| x |] [| 3 |] [| x+1 |]) -- Rejected -- Ditto So hygiene is still easy to enforce

Staging top-level splices module M where f x =. . . no mention of h or g. . . $(blah) –- Defines function h -- blah cannot mention h or g g y =. . . mentions h. . . Can’t really type, and then run, ‘blah’ if it mentions ‘g’ (which mentions stuff bound by ‘blah’) So TH enforces a top-to-bottom discipline

Reification

Reification data T = T 1 Int T | T 2 $(make. Lens ‘T) Want make. Lens to �Get hold of the definition of T (reification) �Chew on it �Spit out some T-related boilerplate definitions make. Lens : : Name -> Q [Dec] make. Lens n = do { d <- reify n ; case d of … }

Reification lookup. Name : : Bool -> String -> Q (Maybe Name) reify : : Name -> Q Info reify. Fixity : : Name -> Q (Maybe Fixity) reify. Instances : : Name -> [Type] -> Q [Instance. Dec] reify. Roles : : Name -> Q [Role] reify. Annotations : : Data a => Ann. Lookup -> Q [a] reify. Module : : Module -> Q Module. Info reify. Con. Strictness : : Name -> Q [Decided. Strictness] data Info = | | | Var. I Name Type (Maybe Dec) Ty. Con. I Dec Class. I Dec [Instance. Dec] …etc…

Exp vs TExp t newtype TExp t = TExp { un. Type : : Exp } -- un. Type : : TExp t -> Exp TExp is an abstract type, with a phantom ‘t’ Can only be constructed using (typed) quotations [|| blah ||] Can only be decomposed by converting to Exp

Quasiquotes (Geoff Mainland, ICFP Freiburg)

Quasiquotes f x = [p| blah |] means (confusingly) f x = $(quote. Exp p “blah”) data Quasi. Quoter = quote. Exp : : quote. Pat : : quote. Type : : quote. Dec : : Quasi. Quoter String -> Q { Exp, Pat, Type, [Dec] } User-written parsers, for totally different langauges

Quasiquotes [pads| funny PADS stuff |] Works for declarations, patterns, types, as well as expressions (one parser for each) Depends crucially on generating untyped syntax trees A useful way of embedding completely different language in Haskell

Open questions

Open issues THSyn vs Hs. Syn; keeping them in sync Pain of quotation. Other language (Scheme, Scala…) do implicit quotation. f e instead of f [| e |] Reification of an Id whose type is not yet settled f x = $(do { xinfo <- reify ‘x; … }) but this is ok (because of staging) x = blah; $(do { reify ‘x; …})