Arc GIS for Server Administrative Scripting and Automation

Arc. GIS for Server: Administrative Scripting and Automation Shreyas Shinde Ranjit Iyer Esri UC 2014 | Technical Workshop |

Agenda • Introduction to server administration • Command line tools • Arc. GIS Server Manager • Overview of the administrative API • Automation • Popular scripting workflows - Publishing of services - Setting up permissions - Extracting popular extents - Monitoring services and databases - Scaling and statistics Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Introduction to server administration Arc. GIS Server Installation Media Keep it running… Provision hardware Software Authorization Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation Select a deployment pattern Arc. GIS Server Setup

The administrator’s toolbox Arc. GIS Server Administrator Directory Arc. GIS Server Manager Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation Console tools

Command line tools • Located under /arcgis/server/tools • A collection of most popular tools • Categories - Administrative credentials - • For resetting administrative accounts - SSL/certificate management - Backup and restore - Managing cache Lock the tools down using OS file permissions! Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Demo Command line tools Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Arc. GIS Server Manager • Browser based interactive application • Located at: http: //server. domain. com: 6080/arcgis/manager • Simple, easy-to-use • Ability to manage your entire server site - • Even though you are accessing Manager on a specific server node Great for common workflows! Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Demo le Tour de Manager Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Arc. GIS Server Manager – under the hood Simple HTTP requests to the administrative API Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Overview of the administrative API • Entire server administration is an API - • If Arc. GIS Server Manager can use it, so can you! RESTful - GET and POST of JSON data • Well organized and well documented • Lots of samples • Scriptable • Designed for automation! Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Create and manage your services. Start or stop them as wanted The administrative API Add new nodes to grow capacity. Remove them to shrink. Organize machines into clusters. Services Manage system properties, web adaptors, server directories. Machines + Clusters System environment Statistics (new at 10. 3) Admin API Manage all your database connections through a single interface. Security Databases Logs Harvest logs for auditing, understanding usage patterns and troubleshooting. Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation Get fine grained statistics on your GIS services. Configure GIS tier or web tier authentication. Manage accounts, roles and permissions on services. Configure SSL.

Why automate? • Common repeated workflows - • • • Plot popular extents for the data Scheduled actions - Restarting services when data is updated - Build map caches during low demand by moving machines to caching cluster Reactive actions - Adding capacity (server nodes) to a cluster when demand increases - Send email notifications when errors are logged Works with all programming/scripting languages that support HTTP! Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Where should you begin? Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Demo Publish Services Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Motivation • Publishing many services • Moving services between Dev, Staging, Production environments - Backup/restore is an option but sometimes you just want to republish services • Can publish from Desktop and Manager • Tedious if you have lots of services • Automation is your friend! Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Task : Publish a list of services listed in a file • Help system has a script that uses Arc. Py • Arc. Py is an python library that complements the Admin REST API • Provides many convenience functions Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Caveats • Works great, just need a few lines of code • However, you need Arc. Py on the machine • Script publishes service definitions sequentially - For every SD - Upload - Publish • With many SDs to publish this can be a very slow process • Publishing ~100 services took hours Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

We can do better • Run the script from any machine - • • No Arc. Py dependency Exploit concurrency - Parallel uploads - Parallel publishing If your script is amenable to concurrency, exploit it by all means Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Publishing a service with an SD • Two step process - Upload the SD to the Server - - QUICK Invoke the publishing tool - INTENSIVE Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Script design Upload Publish SD SD Upload Upld. ID Publish Upload Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Comparison results Speed up of ~3 Arc. Py publishing script Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation DIY multi-threaded publishing script

Demo Starting a service Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Options • Manager • Admin • Script Start Geometry Service Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Anatomy of an Admin script • Main function - • Drives the logic Internal utility functions - Get Token - Make HTTP requests - Parse JSON responses Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Demo Apply Permissions Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Task : Apply permissions to services • Permissions are the way to secure your service • Do it in Manager • Tedious for lots of services. Must automate! Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Demo • Show them previously published services in Manager • Show them roles • Run the script • Discuss the script Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Demo Health check for services Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Using the report API, Get all the folders get all the service instances. Using Geo-services REST API to test availability. Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation Print report

Demo Health check for databases Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Use find. Items to search for all enterprise geodatabases Use validate. Data. Item to test database connectivity. Print output Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Demo Popular extents Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Demo Elastic scaling Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Task : Dynamically add machines to your site • Add machines to reduce latency and increase throughput • Dynamically add them when a latency threshold is breached • De-allocate machines when traffic is less Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Demo • Start with a single node site • Make service requests and watch latency (we’ll use JMeter) • Run a script that adds a machine to a cluster • Watch the latency drop Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Statistics in 10. 3 (Sneak peak) Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Statistics at 10. 3 • • Server is always collecting statistics - Avg response - Number of requests - Number of timeouts - . . . Report defines the resource, metric and interval - Previous day, last 7 days, last 30 days - Custom time window • Server chooses intelligent data point intervals in the reports • REST API to harvest all this information Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Python is great!! Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Short detour… • Most modern languages (like Python) support Threads • Some languages provide concurrency as a language feature • - Go - Erlang - Clojure - Scala Suited for concurrent programming Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Go • Invented at Google • About concurrency • Lightweight threads (goroutines) • Typed thread-safe communication and synchronization (channels) • Readable concurrent code Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Erlang • Invented at Ericsson • Functional language • Light weight ‘processes’ • Can spawn 100 s of thousands of them • Concurrency via message passing • Actors model Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Python is great!! Don’t get us in trouble, we don’t want to start a rebellion Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Thank you… • Please fill out the session survey: First Offering ID: 1146 Second Offering ID: 1344 Online – www. esri. com/ucsessionsurveys Paper – pick up and put in drop box Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Questions? Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation

Resources GIS Tutorial for Python Scripting Esri Press, 2014 Python Scripting for Arc. GIS Esri Press, 2013 Paperback and e-book Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation Just released! Offers several hands-on tutorial exercises. Good reference text

Esri UC 2014 | Technical Workshop | Arc. GIS for Server: Administrative Scripting and Automation