GIS Data Processing Arc Toolbox Geoprocessing and Model

GIS Data: Processing Arc. Toolbox, Geoprocessing and Model. Builder Spring 2008 1 GISC 6382 Applied GIS UT-Dallas Ron Briggs

Arc. Toolbox, Geoprocessing, Model. Builder : Overview Arc. Toolbox provides access to geoprocessing • Arc. Toolbox is a dockable window in Arc. Catalog or Arc. Map • Comprises a set of Tools (750+) • Tools support all data types (coverages, shapefiles, gdb, raster) • Multi- step workflow available via Modelbuilder and/or scripting • Command line interface also available for traditionalists and/or power users 2 GISC 6382 Applied GIS UT-Dallas Ron Briggs

What is geoprocessing? • the modification and analysis of spatial (geographic) data. CAD Projections GDB Conversion Database Spatial analysis Data management 3 GISC 6382 Applied GIS UT-Dallas Ron Briggs

What is Geoprocessing used for? – Data processing/ data automation: Creation, Preparation, Conversion, Management – Data analysis/spatial modeling: Suitability, Sensitivity, Capacity, Risk 4 GISC 6382 Applied GIS UT-Dallas Ron Briggs

Accessing geoprocessing tools • Geoprocessing is carried out using tools stored in Arc. Toolbox – A dockable window available in Arc. Map, Arc. Catalog, Arc. Globe, or Arc. Scene • Tools run processes Opens Arc. Toolbox 5 GISC 6382 Applied GIS UT-Dallas Ron Briggs

Inside of Arc. Toolbox • Toolbox: Container for toolsets and tools Toolbox – Note: cannot have a toolbox within a toolbox • Toolset: Logical container of tools and other toolsets (i. e. folder) • Tool: Single geoprocessing operation (includes dialogs, models, and scripts) Toolset Tools GISC 6382 Applied GIS UT-Dallas Ron Briggs 6

Supported data types • The tools work with many data types Coverage Geodatabase Tables Shapefile Layer file CAD Behavior Raster Geoprocessing 7 GISC 6382 Applied GIS UT-Dallas Ron Briggs

The tools work with many data types …Caution • Often there are different tools for different data types. Often there is one: – For shapefiles and geodatabse features classes – Another for coverages – Another for raster datasets • And they have the same name! – For example, there is a separate Clip tool for each of these input data types – Similarly, there is a separate Define tool for each of these input data types • Occasionally, there is a capability only available for coverages – e. g. conversion of SDTS (Spatial Data Transfer Format) or VPF (a vector format used by military) – Convert first to coverage, then convert coverage to shapefile 8 GISC 6382 Applied GIS UT-Dallas Ron Briggs

Tools with the same name • Some tools have the same name – Must distinguish between these tools in scripts – Be sure to get the correct tool for the data type you are processing Clips a coverage Clips a feature class Clips a raster 9 GISC 6382 Applied GIS UT-Dallas Ron Briggs

A special note on the Coverages Tools toolbox • Only available if you have Arc. Info Workstation installed • The toolset titles (Analysis, Conversion, Data Management) duplicate many of the toolboxes in Arc. Toolbox as a whole – You need to use these tools to process coverages – In some cases, there are capabilities here not available for other data types • e. g. conversion of SDTS (Spatial Data Transfer Format) or VPF (a vector format used by military) • Convert first to coverage, then convert coverage to shapefile 10 GISC 6382 Applied GIS UT-Dallas Ron Briggs

System (Esri provided) toolboxes • All tools are organized into toolboxes • System toolboxes are provided by ESRI. Number depends on – License level (Arc. VIEW, Arc. Editor, Arc. Info) – Extensions licensed • and you can create your own toolboxes (and tools) GISC 6382 Applied GIS UT-Dallas Ron Briggs 11

Tool Counts—nearly 750 now (9. 2) • Counts vary depending on license/extensions – Arc. View: 102 tools – Arc. Editor: 104 tools – Arc. Info: 251 tools – Spatial Analyst extension: 158 tools – 3 D Analyst extension: 45 tools – Geostatistical Analyst extension: 1 tool • A total of 455 tools came with 9. 0 – Feature class tools – 120 – Spatial Analyst tools – 185 – Geocoding tools – 5 – 3 D Analyst tools– 20 – Linear Referencing– 10 – Coverage tools – 57 Counts as of 9. 0. 9. 1 has an additional 171 tools 9. 2 has additional 121 tools GISC 6382 Applied GIS UT-Dallas Ron Briggs 12

The geoprocessing framework: 2 four ways to run tools Command line 1 3 Models Dialog 4 Scripts 4 GISC 6382 Applied GIS UT-Dallas Ron Briggs Arc. Objects 13

Selecting the way to run tools: Choosing between the four ways Selection depends on a user’s needs and abilities 1. As a dialog inside of any Arc. GIS application • • Simple, one at a time processing (GIS for dummies? ) Good for ad hoc processing 2. As a command from a command line • • Very efficient for command line afficianados Supports intelligent auto-completion of commands 3. As a process in a model (Modelbuilder) • • An “executable flow chart” for GIS processing Powerful multi-step processing, yet simple to create and use 4. As a function call in a script or program • • Similar to AML in Arc. Info 7, but avoids learning proprietary language provides decision making and looping to support batch processing In this course we will use #1 and #3 14 GISC 6382 Applied GIS UT-Dallas Ron Briggs

Parameters • all tools have one or more parameters – pieces of information that the tool needs in order to run • some parameters are required, – such as the name of the input data set • some parameters are optional • most parameters have a default set, which – you should always review, and – change if it’s inappropriate for your application 15 GISC 6382 Applied GIS UT-Dallas Ron Briggs

Environment settings • “Customizable defaults” • Common parameters that are applied to all tools within a geoprocessing session • Settings work (“persist”) in all environments (dialogs, command line, models, scripts, ) General settings Current workspace Output coordinate system Geodatabase settings Raster settings XY Domain Cell size M and Z Domains Mask Coverage environment Comparison between prj files Geodatabase raster Statistics Compression Precision for new coverages GISC 6382 Applied GIS UT-Dallas Ron Briggs 16

Four levels of environment settings • User Profile – The settings will persist throughout all applications loaded by the user • Application Level – The settings will apply to all tools executed from that application • For a specific model or script – Settings apply to all tools within this specific model or script. • For an individual tool – Settings apply to a specific tool in a model or script Lower order (e. g. individual tool) settings override higher order (e. g. user profile) settings GISC 6382 Applied GIS UT-Dallas Ron Briggs 17

Using Modelbuilder Arc. GIS 9. 2 has some major enhancements to Modelbuilder: Whats_New_in_Arctoolbox. pdf 18 GISC 6382 Applied GIS UT-Dallas Ron Briggs

• Why Use Modelbuilder? document the processing steps (workflow) that you carried out – – • • You will forget You may need to do it again re-run the analysis to experiment with different parameters to see how results are affected rerun the analysis if you find a mistake carry out repetitive, multi-step tasks provide other users (e. g. operations staff) with an exact workflow that they can replicate 19 GISC 6382 Applied GIS UT-Dallas Ron Briggs

The Elements of Model. Builder • Model diagram window – Input variable – Tool – Derived data variable • Toolbar – Add Data or Tools – Layout – Zoom and Pan – Add connection – Run 20 GISC 6382 Applied GIS UT-Dallas Ron Briggs

Building a model: constructing • Create a new model • Drag tool into model • Drag data into model or onto a tool • Link data and tool 21 GISC 6382 Applied GIS UT-Dallas Ron Briggs

Building a model: running and documenting • Save and rename the model • Run model • Document the model 22 GISC 6382 Applied GIS UT-Dallas Ron Briggs

Setting model parameters Or double click on the tool’s process box to open its parameter dialog 23 GISC 6382 Applied GIS UT-Dallas Ron Briggs

Element States • Not ready to run (parameters not set) • Ready to run • Has been run (note the grey shadow) 24 GISC 6382 Applied GIS UT-Dallas Ron Briggs

An example, which we will now build 25 GISC 6382 Applied GIS UT-Dallas Ron Briggs

Alternatives to Modelbuilder: Options for Developing Additional Capabilities • Modelbuilder – Multi-step, sequential processing of Arc. Tools, using a visual development environment which comes as part of Arc. GIS • Python, Jscript and VBScript – standardized and relatively simple scripting languages for repetitive processing, including loops and decision trees, using Arc. Tools – Python scripts can be generated from Modelbuilder • Visual Basic for Applications – Permits writing of VB macros for sophisticated customization and development within standard Arc. GIS (Arc. Map/Arc. Catalog) interface (and thus requires license for these) – may incorporate Arc. Objects, the COM compliant software objects out of which Arc. GIS is constructed. • Arc. GIS Engine – Set of embeddable GIS components (Arc. Objects software objects) for use in building custom applications, independent of Arc. GIS interface – Runs under Windows, Unix and Linux, with support for Java, C++, 26 COM and. NET GISC 6382 Applied GIS UT-Dallas Ron Briggs

Appendix ESRI GIS Software 27 GISC 6382 Applied GIS UT-Dallas Ron Briggs

Environmental Systems Research Institute, Inc. (ESRI) Redlands, CA • • • Privately held by Jack & Laura Dangermond One of the 50 largest software companies in the world Leader in GIS with at least 1/3 rd of market 1 million users (2002) and 2, 600 employees Originator of commercial GIS in 1981 with release of Arc. Info Released their first GUI (graphics user interface) product, Arc. View, in 1991 using proprietary Avenue programming language (for MS Windows, UNIX, Macintosh) • These two product lines (Arc. View 3 and Arc. Info 7) combined together in Arc. GIS v. 8 released in 2000 – complete rewrite based on Microsoft COM/Active X software objects • Only run on MS Windows – completely different interface from earlier Arc. View and Arc. Info – old, Avenue based, Arc. View still available as Arc. View 3. 3, • But Old Avenue scripts will not run in Arc. GIS 8 – old command line Arc. Info 7 still available as Arc. GIS Workstation • Old AML scripts will run in Arc. GIS 9 • Arc. GIS 9 released in 2004 • Current 2008 release is Arc. GIS 9. 2 GISC 6382 Applied GIS UT-Dallas Ron Briggs 28

ESRI Product Line-up: Arc. GIS client products Arc. Reader (“adobe acrobat” for maps) & Arc. Explorer (spatial data viewer) – Free viewers for geographic data. Arc. GIS 9. x Desktop: two primary modules (MS NT/2000/XP only) 1. Arc. Map: for data display, map production, spatial analysis, data editing 2. Arc. Catalog: for data management and preview Arc. Toolbox, for specialized data conversions and analyses, available as a window in both Available capabilities within these modules are “tiered” • Arc. View: viewing, map production, spatial analysis, basic editing • Arc. Editor: Arc. View, plus specialized editing • Arc. Info: Arc. View & Arc. Editor plus special analyses and conversions Extensions: for special apps. : Spatial Analyst, 3 D Analyst, Geostatistics, Business Analyst, Network Analyst. Arc. Objects: build specialized capabilities within Arc. Map or Arc. Catalog using VB for Applications Arc. GIS Workstation (for UNIX and MS NT/2000/XP) – the old command line Arc. Info 7. 1 Arc. GIS Engine (MS NT/2000/XP) – – Set of embeddable GIS components (Arc. Objects software objects) for use in building custom applications Runs under Windows, Unix and Linux, with support for Java, C++, COM and. NET – Replaces Map. Objects which were based upon a previous generation of GIS objects Notes: Arc. GIS 8 released 2000 to integrate two previous standalone products: Arc. View and Arc. Info Arc. GIS 9 released 2004 providing the full capability that should have been in Arc. GIS 8!!! --full support for all data types (coverages, shapefiles, geodatabases) --full support for all previous geoprocessing analyses --Modelbuilder for scripting and repetitive processing 29 --Arc. Engine for building custom applications GISC 6382 Applied GIS UT-Dallas Ron Briggs Arc. View 3. 3 (the predecessor to Arc. GIS 8. x) the only GUI option for UNIX.

ESRI Product Line-up: Arc. GIS server products (Spring 2007) Arc. GIS Server – Permits the creation of server-based specialized GIS applications – Provides full range of GIS capabilities to a user without a desktop GIS Arc. IMS (Internet Map Server) – Software to develop Internet server-based mapping and basic analysis – Provides maps and simple query to the user without a desktop GIS SDE (Spatial Database Engine) – middleware to support spatial data storage in standard DBMS – Supports all major industry databases: • Oracle, SQL-Server, IBM DB 2, Ingres Note: SDE and Arc. IMS now distributed as part of Arc. GIS Server package as of 9. 2 Arc. GIS Services – Server based applications built and operated by ESRI or its partners – – and made available on the Internet for subscription Normally charged on a “per transaction” basis, but can be flat fee presumably built using Arc. GIS Server

Other ESRI Products: • Arc. PAD – Mapping on PDAs (“handhelds”) with Windows CE operating system • Arc. Logistics Route – Specialized business application for delivery routing • Arc. FM – water and telecom: industry specific facilities management • Arc. GIS Extensions – Spatial Analyst: raster data analysis – 3 D Analyst: 3 -dimensional data display – Geostatistics: surface analysis – Business Analyst: marketing and site selection – Survey Analyst: update of Arc. Info COGO (coordinate geometry) module – Network Analyst: network routing algorithims; shortest path, etc. – Maplex: automated, high quality labeling for maps (now part of Arc. GIS 9) – Publisher: creates. MXP maps for reading with Arc. Reader Extensions work irrespective of Arc. View/Arc. Editor/Arc. Info tier • Business. Map: – $99 standalone business mapping (originally Richardson-based Map. Lynx) 31 GISC 6382 Applied GIS UT-Dallas Ron Briggs

Arc. GIS System c: Arc. GIS Workstation Clients Arc. Info Arc. Editor Arc. View Arc. Map Arc. Catalog Arc. Toolbox Arc. Engine/ Arc. Objects Application Development & Customization $ Arc. IMS Services Consistent interface Increasing capability Arc. Explorer Arc. Server Services Arc. SDE Services Internet Arc. Pad Files Handheld/Wireless Browser (Personal Geodatabase, Shapefiles, Coverages, GISC 6382 Grids, tins, etc) Databases Multi-user Geodatabases (in Oracle, Server, Applied GIS UT-Dallas Ron SQL Briggs IBM DBII, etc) 32 Source: ESRI with mods.

Some additional detail… 33 GISC 6382 Applied GIS UT-Dallas Ron Briggs

Discontinued Products • Arc. CAD – CAD product from ESRI • PC Arc. Info – – 1 st effort at PC based GIS DOS based, command-line driven Data not compatible with Arc. Info UNIX Replaced by Arc. Info 8 and Arc. View 3. 2 • DAK (Data Automation Kit) – Subset of PC Arc. Info for data preparation for Arc. View 3. 2 • Atlas/GIS – once a leader in PC-based mapping – Bought by ESRI in 1996 & discontinued in 2001 GISC 6382 Applied GIS UT-Dallas Ron Briggs 34

Arc. GIS Version 8/9 With Version 8 & 9, now have two flavors: Desktop: – Largest Microsoft COM/Active. X application to date – Full GUI interface – Customization via Visual Basic for Applications – New data base concepts: Geodatabase – Runs on XP/2000/NT only – no UNIX version available Workstation: – classic, command-line Arc. Info with AMLs (Arc Macro Language) for customization – same as version 7 and earlier, with minor enhancements – the only option for UNIX, but also available on MS XP – With release of Arc. GIS 9, little reason to use unless under UNIX • Now, all capabilities have been moved to Arc. GIS 35 GISC 6382 Applied GIS UT-Dallas Ron Briggs

Arc. GIS Desktop Primary Characteristics • GUI-based tools – Arc. Catalog, Arc. Map, Arc. Toolbox • Geodatabase spatial data model supported in standard database environments – MS Access (. msb) for personal applications – File-based geodatabase with 9. 2 – Any industry db via SDE for multi-user applications • Modeling of real world as intelligent objects – Houses, poles, not points, lines, polygons • COM/Active. X components (Arc. Engine) for embedding geography in other applications Arc. Info 7: simple data complex applications Arc. Info 8/9: intelligent data simpler applications 36 GISC 6382 Applied GIS UT-Dallas Ron Briggs

Arc. GIS 9 Desktop Modules Arc. Catalog (schema editor, with VISIO generation) • The base application for Arc. Info Desktop • Windows Explorer-like interface • for organizing access to data and metadata • For launching other Desktop apps: MAP and TOOLBOX Arc. Map (object editor) • GUI for map creation and spatial data editing • Arc. Plot/Arc. Edit (from Arc. Info v. 7) & Arc. View 3. 2 View/Layout combined • Map projections on the fly (not via conversion as in AV) Arc. Toolbox (geoprocessor) • An interface to geoprocessing tools • In Arc. GIS 8 it was a separate module – In Arc. GIS Release 9 it’s an integrated window in Arc. Catalog or Arc. Map 37 GISC 6382 Applied GIS UT-Dallas Ron Briggs

Arc. GIS Desktop Capability Tiers: Each tier has the same interfaces (Arc. Catalog, Arc. Map, and Arc. Toolbox), but an increasing set of capabilities are available within them (and $ price rises accordingly!) Arc. View: – viewing, map production, spatial analysis, basic editing Arc. Editor: – Arc. View, plus topologic editing of geodatabases and editing of coverages Arc. Info: – Arc. View and Arc. Editor, plus more geoprocessing analysis, conversions, and full support for coverages. – Old, command line Arc. Info including AML support 38 GISC 6382 Applied GIS UT-Dallas Ron Briggs

Arc. GIS 9. 0 versus Arc. GIS 8. 3 • Main differences from 8. x are in Arc. Toolbox – Arc. Toolbox built into Arc. Catalog and Arc. Map rather than a separate module – All Arc. Toolbox tools support all data types (geodatabase, shapefiles, coverages) • 8. 3 primarily support coverages – Model. Builder diagrammatic modeling tool • Invaluable for tracking and replicating geoprocessing steps – New scripting capability for repetitive actions • Python, JScript and VBScript--simpler to use than VB for Applications, the only alternative in 8. x • Old aml (arc macro language from Arc. Info 7) also supported What Arc. GIS 8 should have been when it was first released! Incorporates just about everything from Arc. Info 7/Arc. View 3. 39 GISC 6382 Applied GIS UT-Dallas Ron Briggs

Arc. GIS 9. 1 & 9. 2 versus Arc. GIS 9. 0 For Spring 2007 we will use Arc. GIS 9. 2 Ormsby text contains copy of 9. 0 Gorr and Kurland contains copy of 9. 1 Main differences in 9. 1 from 9. 0 • General performance improvement and user enhancements (e. g. pause drawing) • Additional geoprocessing tools (63 in Arc. View, 86 in Arc. Editor, 22 in Arc. Info) • Improved labeling of features on maps (Big change here was in 9. 0) • Support for linear referencing • Network Analyst extension for transportation networks (shortest path, etc. . ) Main differences in 9. 2 from 9. 1 • Geodatabase and map documents (. mxd files) incompatible! • File based geodatabase – Faster, larger capacity (x 1, 000) than MS Access-based personal geodatabase – Higher precision of coordinate storage • • Identity/info. and measurement tools rewritten and greatly enhanced Graphing re-written – Supports brushing (linking) between graph & map (as in Anselin’s Geo. DA) software) • Supports cartographic representation 40 – Can re-position features on map without changing coordinate values in geodatabase GISC 6382 Applied GIS UT-Dallas Ron Briggs

Computing Evolution And it will all keep changing! Pervasive Computing Internet Desktop Workstation Mini Source: ESRI, Inc. Mainframe • Small Hardware (Nano) • Wireless Internet • Interoperable • Embedded GISC 6382 Applied GIS UT-Dallas Ron Briggs 41