USING THE DOWNLOAD TOOL DALLAS ALTERYX USERS GROUP

USING THE DOWNLOAD TOOL DALLAS ALTERYX USERS GROUP BRIAN PURCELL MARCH 2019

DOWNLOAD TOOL • CONNECTORS PALLET • USE CASES • FTP/SFTP TRANSFERS • APPLICATION PROGRAMMING INTERFACES • WEB SCRAPING • PROGRAMMING LANGUAGES NOT REQUIRED • THEN LEVERAGE OTHER ALTERYX PROCESSES

ANATOMY OF A DOWNLOAD WORKFLOW • TEXT INPUT (OR FORMULA) –URL STRING • OTHER FIELDS COULD BE CONTAINED—DEPENDING ON USE CASE • DOWNLOAD TOOL • CHECK CONFIGURATION • PARSING / SHAPING TOOLS • JSON PARSE, REGEX, XML, TEXT TO COLUMNS, ETC.

FTP DOWNLOAD WORKFLOW • EXTERNAL VENDOR PROVIDES PROGRAM INFORMATION • THE FILE IS UPLOADED ON A NIGHTLY BASIS • FILENAME HAS AN APPENDED DATE • A DESTINATION PATH IS DEFINED AND CONFIGURED • THE JOB RUNS DAILY ON OUR COMPANY’S ALTERYX SERVER

FTP DOWNLOAD CONFIGURATION

SHOW CENSUS FTP ALTERYX JOB

PURPOSE – WEBSCRAPING / API • ENRICH EXISTING DATA • EASILY ACQUIRE EXTERNAL DATA • OBTAIN COMPETITIVE INFORMATION • AUTOMATE MUNDANE TASKS

DIFFERENCES API WEBSCRAPING • INTENDED FOR DATA EXTRACTION • INTENDED FOR BROWSER TRAFFIC • OFTEN REQUIRES AN APPLICATION, APPROVAL, AND TOKEN • MORE INFORMATION AVAILABLE THAN API’S • MAY REQUIRE A SUBSCRIPTION OR PER USE FEE • UNSTRUCTURED TEXT OUTPUT • DEFAULT OUTPUTS – JSON/XML • MOST SITES DON’T OFFER ANAPI • GRAY AREA – LEGALLY • CONSULT YOUR LEGAL RESOURCE

API—APPLICATION PROGRAMMING INTERFACE • WEBSITES OPTIMIZED FOR DATA TRANSFER • THE “UNDER THE HOOD” OFCONNECTIVITY • THINK OF THE APPS THAT USE GOOGLE MAPS Realtor. com Likely Uses Google’s API within their Mobile App

WEBSCRAPING Browser View Alteryx View / Page. Source Right Click in Browser, view Source

HARDEST PART OF WEBSCRAPING? ? ?

SHOW MARCH MADNESS ALTERYX JOB

API’S APPLICATION PROGRAMMING INTERFACE • COMMONLY USED FOR MOBILE APPLICATIONS • METHODS – GET, POST, PUT, DELETE • URL ENDPOINT • PARAMETERS • KEY / TOKEN

THE PROCESS • FIND THE REGISTRATION PORTAL • SUBMIT AN APPLICATION, STORE YOUR KEY • REVIEW API DOCUMENTATION • BUILD A BASIC QUERY • PARSE • PAGENATE (IF NEEDED)

NOAA • HTTPS: //WWW. NCDC. NOAA. GOV/CDOWEB/WEBSERVICES/V 2 • 1527 DIFFERENT DATA REPORT CATEGORIES • TEMPERATURE, WIND, PRECIP • SOIL TEMP, DEGREE DAYS, CLOUD COVERAGE • 9, 563 STATIONS • 6, 340 IN THE US • 361 IN TEXAS

GHCN - GLOBAL HISTORICAL CLIMATOLOGY NETWORK REPORT Example of Data However, not usable for data analysis– embedded in a pdf We prefer a structured data table. Which will allow us to enrich our data by appending weather conditions to internal data (examples) -- Weather driven industries – HVAC, lawn care, utilities -- Logistics – snow/heavy rain on shipping

Pick One

• “PARAMETER NAME”=“VALUE” • JOIN MULTIPLE PARAMETERS WITH “&” • SOME ARE REQUIRED, OTHERS ARE OPTIONAL • AUTHENTICATION COULD BE HANDLED THROUGH A PARAMETER

AUTHENTICATION • NEARLY ALL API’S REQUIRE AN ACCOUNT • COMMONLY SEEN METHODS • HEADER – DOCUMENTATION SHOULD IDENTIFY THE HEADER NAME • PARAMETER VALUE – INSERTED ALONG WITH OTHER PARAMETERS • BASIC – USE THE USER NAME &PASSWORD TAB ON DOWNLOAD CONFIGURATION • VERY HARD – TWO STEP, AUTHENTICATION API CALL, THEN 2 ND API CALL • TWITTER, TABLEAU

SHOW NOAA ALTERYX EXAMPLE

PAGENATION • API’S TYPICALLY LIMIT DATA RESULTS TO MAINTAIN BANDWIDTH • ALTER YOUR URL REQUEST • OFFSET, ROWCOUNT – COMMON PARAMETERS TO RETURN THE NEXT PAGE OF RESULTS • HOW TO HANDLE • MOST API’S WILLRETURN A MAX COUNT • RECONFIGURE YOUR API TO SEND AAPI REQUEST FOR EACH PAGE (USING OFFSET) • ADVANCED--CREATE AN ITERATIVE MACRO

SHOW TWITTER ALTERYX EXAMPLE

SOMETIMES YOU’LL GET LUCKY • API EXPLORER/DEBUGGER • AZURE / QUALTRICS • SIMULATES HTML REQUEST WITH ENDPOINTS, PARAMETERS, AND HEADERS • HTTPS: //DEV. APPLICATIONINSIGHTS. I O/APIEXPLORER/METRICS? APPID=D EMO_APP&API KEY=DEMO_KEY • PLUG IT INTO ALTERYX

TIPS • CHECK URLENCODE • REVIEW HTML RESPONSE CODES • LIMIT URL REQUESTS • CACHE & RUN • SAVE OUTPUT TO YXDB • USE THROTTLE TOOL • TEST YOUR REGEX FIRST, THEN SCAPE • OFTEN WILL INDICATE ERROR • TOKEN • EXCEEDED ALLOTTED RECORDS • IMPROPER QUERY

POSSIBLE USE CASES • COMPETITIVE ANALYSIS • LOCATION, PRODUCT CHANGES, PRICING • AUTOMATED FORM COLLECTION • CREATE A SURVEY, ALTERYX WORKFLOW TO TRANSFORM THE DATA • GOOGLE API’S • GEOCODING, STREET VIEW IMAGES, TURN BY TURN DIRECTIONS

RESOURCES • FTP KNOWLEDGE BASES ARTICLES • HTTPS: //COMMUNITY. ALTERYX. COM/T 5/ALTERYX-KNOWLEDGE-BASE/FILE-TRANSFER-PROTOCOL-FTP-DOWNLOAD/TA-P/10599 • HTTPS: //COMMUNITY. ALTERYX. COM/T 5/ALTERYX-KNOWLEDGE-BASE/AUTOMATING-FILE-TRANSFER-PROTOCOL-FTP-DOWNLOADS/TAP/19186 • HTTPS: //COMMUNITY. ALTERYX. COM/T 5/ALTERYX-KNOWLEDGE-BASE/UPLOAD-TO-SFTP-FTP/TA-P/20845 • HTTPS: //PAGES. ALTERYX. COM/JAVELIN-WEBINAR-SERIES-MASTER. HTML • LIST OF API’S • • (GREAT WEBSCRAPING &API EXAMPLES) HTTPS: //WWW. PROGRAMMABLEWEB. COM/ REGEX TESTERS • HTTPS: //REGEXR. COM/ • THE REGEX COACH – FREE PROGRAM FREE SOFTWARE • POSTMAN – TEST API CALLS • FIDDLER – • USEFUL FOR WEBSCRAPING • TRACK YOUR COMPUTER’S HTML REQUESTS, INCL. URL’S, HEADERS, HTML REQUEST TYPE FIND ME ON LINKEDIN • HTTPS: //WWW. LINKEDIN. COM/IN/BRIAN-PURCELL-2000/ (UPLOAD)