Open Topography A gateway to high resolution topography

Open. Topography: A gateway to high resolution topography data and services Choonhan Youn, Viswanath Nandigam, Minh Phan San Diego Supercomputing Center (SDSC), UCSD

Outline • Open. Topography • Application Software - Tau. DEM (Terrain Analysis Using Digital Elevation Model) • Opal Toolkit • Implementation in detail • Test example – Tau. DEM Analysis using Lidar data

The Objective • Develop and deploy enhancements into the NSF funded Open. Topography (OT) data facility to manage and process high resolution topography data (including lidar based point cloud data) on XSEDE HPC resources. – Integrate OT system with the new Comet XSEDE HPC resource at SDSC – Develop new job manager for the open source Opal toolkit to ensure compatibility with job scheduler on Comet. 3

Open. Topography • Provides community access to high-resolution Earth science-oriented, topography data, and related tools and resources. • several spatial data access functions and geoprocessing applications packaged as Web services across multiple backend servers. • Aims to increase the amount of high resolution topography data (including lidar point clouds) available online. • Enhanced Web-based processing capabilities • With a focus on computationally intensive tasks • Community support • Software tools, tutorials, short courses, and workshops

Open. Topography: Impact • Top resource for lidar data/ Top 10 for free GIS data • Over ~75, 000 unique users ran in excess of 282, 929 custom processing jobs via the Open. Topography portal • Additional 372, 981 jobs run via APIs • Science Enabler • Total Peer-Reviewed Publications: 255 • Total Other Publications: 47

Open. Topography Architecture Application Tier Service Tier Infrastructure Tier

Tau. DEM(Terrain Analysis Using Digital Elevation Model) Suite • Channel Network and Watershed Delineation Software • Developed as C++ command line executable functions • MPICH 2 used for parallelization (single program multiple data) • Terrain analysis software for hydrologic analysis of DEMs • Adapted as science driver and community service to connect OT project • OT provides capability to derive DEM in Geo. TIFF format from high-resolution Lidar Data that can serve as input to Hydrologic Analysis using Tau. DEM. • An XSEDE Extended Collaborative Support Service (ECSS) support for the optimization • Tau. DEM functions are wrapped as Opal Web services on a gateway node. • Integration of key Tau. DEM functions into the standard OT point cloud processing workflow

Deriving hydrologically useful information from Digital Elevation Models Raw DEM Flow Field David Tarboton from Utah State University contributed slide Watersheds are the most basic hydrologic landscape elements Pit Removal (Filling) Channels, Watersheds, Flow Related Terrain Information

Opal toolkit - I • An open-source toolkit developed at SDSC. • Allows users to wrap scientific applications running on cluster and Grid resources as web services, without any modification to the scientific codes. • Allows both web forms-based access and programmatic access of all our applications. • Currently supports SOAP-based Web service access to a number of popular applications (NBCR, Open. Topography). • Allows science applications to be accessed through many workflow tools • Provides interoperation with other applications with Web Service interfaces. • Allows application developers to focus on the scientific tool and workflow development.

Opal Toolkit - II • Standard WSDL and SOAP API for management of scientific applications • Operations to launch jobs, query status, retrieve outputs • Ability to transfer input files using Base 64 encoded binary format, MIME attachments, and 3 rd party transfers • A “Dashboard” for accessing applications and usage statistics • Automatic interface generation using the XML specification for command-line arguments • Ability to connect to multiple resource providers and schedulers at the back-end • Fork, Condor, PBS

System Architecture Apache Tomcat Opal Web services toolkit SOAP/HTTP Postgres DB Job database node (opentopo. sdsc. edu) Service Consumers, Science Gateways ( Web Apps, Model Builders, etc. ) Gateway node (ot-beta. sdsc. edu) Data repository SSH connection Comet Computing Cluster

Configure Comet system • Install Tau. DEM suite • • • intel/2016. 3. 210 mvapich 2_ib/2. 1 gnu/4. 9. 2 gnutools/2. 69 cmake/3. 9. 1 gdal/2. 2. 0 • Application repository • /home/opentopo/ot-taudem/scripts • Temporary job repository • /oasis/scratch/comet/opentopo/temp_project

Opal Toolkit - III • Main implementation steps • Implement Opal job manager for managing the slurm scheduler. • Create Opal configuration file(s) for a scientific application. • Deploy the application as a Web service using Opal’s simple deployment mechanism (via Apache Ant). • Example: ant deploy -Dservice. Name=Tau. DEMPit. Remove. Service Dapp. Config=$PWD/configs/taudem. Pitremove_config. xml • Users can now access this application as a Web service via a unique URL, which is used only for programmatic access of the application. • Example: http: //ot-beta. sdsc. edu: 7070/opal 2/services/Tau. DEMPit. Remove. Service

Opal Application Config XML Example <app. Config xmlns="http: //nbcr. sdsc. edu/opal/types" xmlns: xsd="http: //www. w 3. org/2001/XMLSche ma"> <!-- metadata about the application being wrapped --> <metadata app. Name="Pit. Remove"> <usage> application description </usage> <info><![CDATA Usage: %. / taudem. Pitremove. sh [-i <data_file_name> | -l <data_file_url>] -o <outputfilename> 1. -i <data_file_name> | -l <data_file_url>: - must be zipped file (*. tar. gz) 2. -o <outputfilename>: specify the output filename ]]> </info> <!-- information about command-line arguments --> <types> <!- list of tagged parameters --> <tagged. Params> <param> <id>demfile</id> <tag>-i</tag> <param. Type>FILE</param. Type> <io. Type>INPUT</io. Type> <text. Desc>Input elevation grid file (*. tar. gz)</text. Desc> </param> <!-- grouping of parameters --> <groups><group><name>files</name> <elements>demfile dem. URL</elements> <required>true</required> <exclusive>true</exclusive> <text. Desc>File Inputs (choose one of the following)</text. Desc></group> <group><name>params</name> <elements>outputfilename</elements> <required>true</required> <text. Desc>Other parameters</text. Desc> </group> </types></metadata> <param<id>dem. URL</id> <tag> -l</tag> <param. Type>STRING</param. Type> <text. Desc>Input elevation grid file (*. tar. gz)</text. Desc></param> <param><id>outputfilename</id> <tag>-o</tag> <param. Type>STRING</param. Type> <text. Desc>Name of the output elevation grid with pits filled</text. Desc> </param> </tagged. Params> <!-- location of execution on the server --> <binary. Location>. /scripts/taudem. Pitremove. sh</bin ary. Location> <! -- do not validate arguments for now --> <validate. Args>false</validate. Args> <!-- overrule default job manager, if need be --> <job. Manager. FQCN> edu. sdsc. nbcr. opal. manager. Slurm. Job. Manager</job Manager. FQCN> <!-- whether this application is serial or parallel --> <parallel>true</parallel> </app. Config>

#!/bin/bash nflag= iflag= lflag= oflag= NUM_OF_PROCESSORS=1 INPUT_DEMFILE= INPUT_FILENAME= OUTPUT_FILENAME="output" MY_FILE= while getopts 'n: i: l: o: ' OPTION; do case $OPTION in n) nflag=1 NUM_OF_PROCESSORS="$OPTARG" ; ; i) iflag=1 INPUT_DEMFILE="$OPTARG" ; ; Application script file - I l) o) lflag=1 INPUT_DEMFILE="$OPTARG" wget --no-check-certificate $INPUT_DEMFILE exit. Val=$? if [ $exit. Val != 0 ]; then exit $exit. Val fi INPUT_DEMFILE=${INPUT_DEMFILE##*/} INPUT_FILENAME=${INPUT_DEMFILE%%. *} mkdir $INPUT_FILENAME tar zxf $INPUT_DEMFILE -C $INPUT_FILENAME rm -rf $INPUT_DEMFILE MY_FILE=`ls $INPUT_FILENAME` ; ; oflag=1 OUTPUT_FILENAME="$OPTARG" ; ; ? ) printf "Usage: %s: [-n value] [-i value] n" $(basename $0) >&2 exit 2 ; ; esac done shift $(($OPTIND - 1))

if [ "$nflag" ]; then printf 'Option -n "%s" specifiedn' "$NUM_OF_PROCESSORS" fi if [ "$iflag" ]; then printf 'Option -i "%s" specifiedn' "$INPUT_DEMFILE" fi if [ "$lflag" ]; then printf 'Option -l "%s" specifiedn' "$INPUT_DEMFILE" fi if [ "$oflag" ]; then printf 'Option -o "%s" specifiedn' "$OUTPUT_FILENAME" fi if [[ "$nflag" ]]; then if [[ "$iflag" || "$lflag" ]]; then ibrun -np $NUM_OF_PROCESSORS. /scripts/bin/pitremove -z $INPUT_FILENAME/$MY_FILE -fel ${OUTPUT_FILENAME}fel. tif 2>&1 | tee pitremove. txt exit. Val=$? echo "Exit value: $exitval” # only do this is job execution is successful [ -f "${OUTPUT_FILENAME}fel. tif" ] && if [ $exit. Val == 0 ]; then # make a tarball from the outputs - ignore the stderr from ls echo "******************” echo "Creating tarball from job outputs" tar czvf ${OUTPUT_FILENAME}. tar. gz ${OUTPUT_FILENAME}fel. tif else exit $exit. Val fi else echo "Please specify -i, -l options to run the simulation" fi else echo "Please specify -n, -i, -l options to run the simulation" Fi Application script file - II

#!/bin/bash #SBATCH -A sds 167 #SBATCH -J. /scripts/taudem. Pitremove. sh #SBATCH -p shared #SBATCH -t 00: 30: 00 #SBATCH --nodes=1 #SBATCH --ntasks-per-node=2 #SBATCH -e /oasis/scratch/comet/opentopo/temp_project/app. Tau. DEMPit. Remove. Service 15345460365651 613494886/stderr. txt #SBATCH -o /oasis/scratch/comet/opentopo/temp_project/app. Tau. DEMPit. Remove. Service 15345460365651 613494886/stdout. txt #SBATCH --export=ALL cd /oasis/scratch/comet/opentopo/temp_project/app. Tau. DEMPit. Remove. Service 15345460365651 613494886. /scripts/taudem. Pitremove. sh -l http: //otprocess 1. sdsc. edu/app. Las 2 Dem. Service 1534545705448986378578/output. tin. tif. tar. gz -o pit. Remove -n 2 Slurm script file

Opal Service List

Opal Job submission form

Opal Dashboard

Teton Conservation District, Wyoming LIDAR Example

Job Monitoring Example

Data Products Example

Summary • We were able to deploy the Opal toolkit on the gateway node, and Tau. DEM suite on Comet cluster that was allocated to this project. • We have integrated key Tau. DEM functions into the standard OT point cloud processing workflow to generate hydrologic data products. • Enhances the value and use of these data in hydrologic research by providing access to these computational products without users having to learn Tau. DEM or run HPC.