Scripting Website Interaction with Nightmare JS Rob Ribeiro

Scripting Website Interaction with Nightmare. JS Rob Ribeiro @azurelogic http: //azurelogic. com

Overview • Why? • Phantom. JS • Related Libraries • Nightmare. JS • Getting Started • Usage • Demo • Tips

Why? • Web scraping and scripting – Techniques for extracting information or interacting with automated tools – Run tests, fill forms, extract data, build custom APIs • Why another tool for this? – Many scrapers don’t handle Java. Script • Angular, Ember, Knockout • Need something more powerful – Scrapers don’t render • Screenshots

Phantom. JS • http: //phantomjs. org/ • A headless Web. Kit scriptable “browser” with a Java. Script API that has support for various web standards • Web. Kit – Web browser layout engine – Powers Safari and formerly Chrome • Uses – Headless JS testing with Jasmine, QUnit, Mocha, etc. – Programmatic screenshots – WEBSITE AUTOMATION

Related Ghastly Libraries • Slimer. JS • Casper. JS • Spooky. JS

Slimer. JS • http: //slimerjs. org/ • Similar to Phantom • Uses Gecko (Firefox) instead of Web. Kit • Fully HTML 5/ES 6 -compatible • Still in early stages • Not totally headless yet

Casper. JS • http: //casperjs. org/ • Navigation scripting/testing • Sits on top or Phantom or Slimer • Provides a slightly better API • Standalone script runner, like Phantom and Slimer

Spooky. JS • https: //github. com/Waterfall. Engineering/Spooky. JS • Library to drive Casper from Node. js • Just exposes Casper’s API nicely inside Node apps

Nightmare. JS • http: //www. nightmarejs. org/ • https: //github. com/segmentio/nightmare • A high level wrapper for Phantom that runs on Node. js – Allows interop with other npm modules: nodemailer, express, etc. • Simple, understandable API – Much simpler than Phantom and Casper • Better interaction between Phantom and Node contexts • Supports custom plugins

Phantom vs Nightmare

Getting Started • Install Node: http: //nodejs. org/ • Phantom. JS: http: //phantomjs. org/download. html • Python 2. 7 • Node. JS: $ npm install --save nightmare • Windows: good luck! – There are issues compiling node-gyp on Windows. Requires Visual Studio. – It’s much easier to get VMWare Player + Ubuntu 14. 04 running!

Usage • var Nightmare = require(‘nightmare’); • new Nightmare( ) • Add script chain of goto, wait, type, click, etc. • . run( );

Demo

Tips • Set your viewport and useragent at the beginning of the script New Nightmare( ). viewport(1920, 1080). useragent(“Mozilla/5. 0 (Windows NT 6. 3; WOW 64) Apple. Web. Kit/537. 36 (KHTML, like Gecko) Chrome/38. 0. 2125. 111 Safari/537. 36"”) • Remember to. wait( ) after all. goto( ) calls to allow for page loading. . . goto(http: //google. com). wait( ). . . • Use screenshots and callbacks for debugging problems with your script

Thanks! @azurelogic http: //azurelogic. com