Lua Socket behind the scenes Diego Nehab Short

Lua. Socket behind the scenes Diego Nehab

Short Bio • Graduated from PUC in CS & E, 1999; • Worked in Tecgraf 1995 -2002; • MSc in PL with Roberto, 2001; • 3 rd year Ph. D candidate at Princeton; • Computer Graphics.

Outline of talk • A few historical notes • Case study: SMTP support • Protocol abstraction • Message abstraction • Implementation highlights • Conclusions

Historical notes • 1. 0, 1999, 1. 5 k C, 200 man • 1. 1, 2000, 1. 5 k C, 1. 3 k Lua, 500 man • added protocol support for HTTP, SMTP, FTP • 1. 2, 2001, 2 k C, 1. 3 k Lua, 900 man • buffered input and non-blocking I/O • UDP support • object oriented syntax

Historical notes • 1. 3, 2001, 2. 3 k C, 1. 6 k Lua, 1. 2 k man • streaming with callbacks • added select function • 1. 4, 2001 -2, 2. 2 k C, 2. 2 k Lua, 1. 9 k man • LTN 7 • added URL module • named parameters

Current version • 2. 0, 2005, 4. 6 k C, 2. 5 k Lua, 4. 7 k man • • Extensible C architecture, split in modules LTN 12 (sources, sinks and filters) MIME support (partial but honest) David Burgess Multipart messages support LTN 13 (finalized exceptions) Package proposal Improved non-blocking code, robust to signals. . .

Outline of talk • A few historical notes • Case study: SMTP support • Protocol abstraction • Message abstraction • Implementation highlights • Conclusions

SMTP (RFC 2821) [lua: roberto] telnet mail. tecgraf. puc-rio. br 25 220 tecgraf. puc-rio. br ESMTP Sendmail 8. 9. 3/8. 9. 3 helo lua 250 tecgraf. puc-rio. br Hello lua, pleased to meet you mail from: <roberto@inf. puc-rio. br> 250 <roberto@inf. puc-rio. br>. . . Sender ok rcpt to: <diego@tecgraf. puc-rio. br> 250 <diego@tecgraf. puc-rio. br>. . . Recipient ok data 354 Enter mail, end with ". " on a line by itself Subject: World domination: instructions. Commence stage two. . 250 RAA 10452 Message accepted for delivery quit 221 tecgraf. puc-rio. br closing connection from rcpt body

Protocol abstraction status, error = smtp. send { from = "<roberto@inf. puc-rio. br>", rcpt = "<diego@tecgraf. puc-rio. br>", body = "Subject: World domination: instructions. rn". . "Comence stage two. " } • What if body is large?

LTN 12 sources • Use callback function that produces data; • Returns one chunk each time called; • Signals termination returning nil. function ltn 12. source. file(handle) return function() local chunk = handle: read(BLOCKSIZE) if not chunk then handle: close() end return chunk end

Using sources status, message = smtp. send { from = "<roberto@inf. puc-rio. br>", rcpt = "<diego@tecgraf. puc-rio. br>", body = ltn 12. source. file(io. open("/mail/body", "r")) } • What if body is complicated?

Message Format (RFC 2822) From: Roberto Ierusalimschy <roberto@inf. puc-rio. br> To: Diego Nehab <diego@tecgraf. puc-rio. br> Subject: World domination: roadmap. headers Content-Type: multipart/mixed; boundary=part This message contains attachments --part Content-Type: text/plain Please see attached roadmap. headers part 1 body --part Content-Type: text/html; name="roadmap. html". . . --part-- part 2

Message abstraction declaration = { headers = { subject = "World domination", from = "Roberto <roberto@inf. puc-rio. br>", to = "Diego <diego@tecgraf. puc-rio. br>" }, preamble = "This message contains attachments. ", [1] = { headers = {. . . }, body = "Please see attatched roadmap. " }, [2] = { headers = {. . . }, body = ltn 12. source. file(io. open("/plans/roadmap. html", "r")) } }

Our message API status, message = smtp. send { from = "<roberto@inf. puc-rio. br>", rcpt = "<diego@tecgraf. puc-rio. br>", body = smtp. message(declaration) } • Transform declaration into an LTN 12 source; • Pass source as body to sending function.

How hard is it? • • Message structure is recursive; Need to return chunks but mantain context; Nightmare to write in C!; Use coroutines; Write function recursively, naturally; Call yield with each chunk; Next call resumes wherever we left.

Zoom in on attachments [2] = { headers = { ["content-type"] = 'text/html; name="roadmap. html"', ["content-disposition"] = 'attachment; filename ="roadmap. html"' }, body = ltn 12. source. file(io. open("/plans/roadmap. html", "r")) } • Would like to send PDF; • Binary data has to be encoded (Base 64); • Want to encode on-the-fly.

LTN 12 filters and chains • Filters process data one chunk at a time; • MIME module provides common filters: • base 64, quoted-printable, stuffing, line-wrap. . . • Can chain two filters together: factory • Produce a filter with the composite effect • Can chain a filter with a source: factory • Produce a source that returns filtered data.

Zoom in on attachments [2] = { headers = { ["content-type"] = 'application/pdf; name="roadmap. pdf"', ["content-disposition"] = 'attachment; filename ="roadmap. pdf"', ["content-description"] = 'Detailed world domination plan', ["content-transfer-encoding"] = 'BASE 64' }, body = ltn 12. source. chain( ltn 12. source. file(io. open("/plans/roadmap. pdf", "r")), ltn 12. filter. chain( mime. encode("base 64"), mime. wrap("base 64") ) ) }

Creating filters: high-level • Chunks can be broken arbitrarily; • Filters have to keep context between calls; function ltn 12. filter. cycle(low, ctx, extra) return function(chunk) local ret, ctx = low(ctx, chunk, extra) return ret end function mime. normalize(marker) return ltn 12. filter. cycle(mime. eol, 0, marker) end

Creating filters: low-level int eol(lua_State *L) { int ctx = lua. L_checkint(L, 1); size_t isize = 0; const char *input = lua. L_optlstring(L, 2, NULL, &isize); const char *last = input + isize; const char *marker = lua. L_optstring(L, 3, CRLF); lua. L_Buffer buffer; lua. L_buffinit(L, &buffer); while (input < last) ctx = translate(*input++, ctx, marker, &buffer); lua. L_pushresult(&buffer); lua_pushnumber(L, ctx); return 2; }

Creating filters: low-level #define candidate(c) (c == CR || c == LF) int translate(int c, int last, const char *mark, lua. L_Buffer *buffer) { if (candidate(c)) { if (candidate(last)) { if (c == last) lua. L_addstring(buffer, mark); return 0; } else { lua. L_addstring(buffer, mark); return c; } } else { lua. L_putchar(buffer, c); return 0; } }

SMTP dependencies socket tp smtp mime ltn 12

Error checking • Function return convention • Return nil, followed by message on error; function metat. __index: greet(domain) local r, e = self. tp: check("2. . ") if not r then return nil, e end r, e = self. tp: command("HELO", domain) if not r then return nil, e end return self. tp: check("2. . ") end • Tedious, error prone, virotic, not finalized.

LTN 13 exceptions • try = newtry(finalizer): factory; • On success, try returns all arguments; • On failure, throws the second argument; • Calls finalizer before raising the exception. • foo = protect(bar): factory; • foo executes bar in a protected environment; • Returns nil followed by any thrown error.

No 'if' statements function metat. __index: greet(domain) self. try(self. tp: check("2. . ")) self. try(self. tp: command("HELO", domain)) return self. try(self. tp: check("2. . ")) end • Internal functions throw exceptions; • try calls tp. close() on error; • External functions can be protected.

Conclusions • Hope you like our API, we do; • It is easy to implement; • Function factories + clusures, coroutines • It is fast; • Time critical in C, management in Lua; • Questions?