Terra Stream From Elevation Data to Watershed Hierarchies

Terra. Stream: From Elevation Data to Watershed Hierarchies Andrew Danner (Swarthmore), T. Moelhave (Aarhus), K. Yi (HKUST), P. K. Agarwal (Duke), L. Arge (Aarhus), H. Mitasova (NCSU) Thursday, 08 November 2007

Current Problem: Large Point Data Sets • LIDAR – NC Coastline: 200 million points – over 7 GB – Neuse River basin (NC): 500 million points – over 17 GB • Grid elevation models – Neuse River basin: * 20 ft – 2. 5 GB * 10 ft – 10 GB * 5 ft – 40 GB • Data too big for RAM – Must reside on disk – Disk is slow Andrew Danner 2

I/O-efficient Algorithms [AV 88] • Traditional algorithms optimize CPU computation Disk – Not aware of performance penalty of disk access B – Virtual memory, swap space can’t predict disk access RAM • I/O model – Memory is finite – Data is transferred in blocks CPU – Complexity measured in disk blocks transferred – M – Andrew Danner 3

Terra. Stream: Terrain processing pipeline Andrew Danner 4

Terra. Stream Goals • Scalable – All stages must work for 100+ million points/cells • General – Stages should work with either TIN or grid data • Automated – No need for manual intervention/preprocessing • Modular – Users only need to run the stages they want • Adaptable – Allow each stage to support multiple models Andrew Danner 5

Terra. Stream: Terrain processing pipeline Andrew Danner 6

Points to DEM • Grid DEM – Interpolation – Use quad tree to automatically tile terrain – Use quad tree neighbors for smooth boundary transitions • TIN – I/O efficient Delaunay triangulation – Constrained Delaunay also possible if constraints (breaklines) fit in memory • Height graph − View both grids and TINs as a height graph. − Nodes, neighbors, and edges between neighboring nodes − Definition of node, neighbor different in TIN/Grid − Design algorithms to work on height graphs Andrew Danner 7

Flow Modeling Andrew Danner 8

Flow Modeling • Identifying minima due to noise • Removing noise from terrains • Modeling flow directions, extracting river networks Andrew Danner 9

Coping with Noisy Data • Identifying minima likely due to noise – Topological persistence – Computed in Sort(N) I/Os – Assign a significance score to each minima (low score likely noise) – Provide mechanism for removing low scoring sinks – User can select score threshold Andrew Danner 10

Ph. D Defense Noise Removal Noisy terrain After noise removal Flooding in Sort(N) I/Os Other Mechanisms? Carving? Andrew Danner 11

110 From Elevation to River Networks 90 85 95 100 80 • • Where does water go? From higher elevation to lower elevation Single flow directions form a tree Support for multiple flow directions Sea Andrew Danner 12

Drainage Area 1 • How much area is upstream of each node? • Each node has initial drainage area (1) • Drainage area of internal nodes depends on drainage area of children 1 1 2 3 3 7 9 1 2 11 1 14 2 1 25 5 3 1 Andrew Danner 17 1 1 7 13

Computing Flow Directions/Drainage • Terraflow – Sort(N) I/Os on grids • Modified to work on height graphs – Same I/O bound – Now works on TINs • New implementation – More robust, portable – Incorporate new sink removal – Better handling of flat areas… Andrew Danner 14

Flow Modeling Improvements • Detection of flat areas – Improved method on grids if O(1) rows fit in memory • Routing on flat areas – Soille extension of Garbrecht & Martz • Flat areas usually result of hydrological conditioning with flooding Andrew Danner 15

Hierarchical Watershed Decomposition Andrew Danner 16

Watershed Hierarchies • • • Decompose a river network into a hierarchy of hydrological units All water in HU flows to a common outlet Hierarchy provides tunable level of detail Method used: Pfafstetter Want a solution scalable to large modern hi-res terrains Andrew Danner 17

1 83 9 2 Pfafstetter 2 1 7 3 1 7 Find main river Find four largest tributaries Label basins/interbasins Recurse until single path 1 5 14 11 9 1 • • 2 6 1 Andrew Danner 1 3 1 1 2 5 4 2 3 17 7 1 25 1 18

Example Watershed Boundaries 96 9 97 98 999 998 99 997 996 94 6 994 95 993 8 4 992 93 92 2 91 7 5 995 Andrew Danner All levels computed in one run. User selects level of detail with map algebra 3 991 1 19

A Complete Pipeline Andrew Danner 20

Implementation • • • TPIE: C++ primitives for I/O-efficient algorithms Standalone command line apps with GDAL GRASS: Open Source GIS Plugins Arc. GIS Plugins (soon) Test Data: – North Carolina LIDAR * Neuse river basin: 400 million points (NC Floodmaps) * Outer banks coastal data : 128 million points (NOAA CSC) – USGS 30 m NED Andrew Danner 21

Our Results • Experimental Results – Scales to over 400 million points – Other software tools crash at 25 million points – Keeps memory usage low using I/O efficient methods Format HG vertices (millions) Pipeline stage Dem Construction Building height graph Hydro. Conditioning Flow Modeling Routing Accumulation Watershed Delineation Total Andrew Danner 20 ft grid 10 ft grid 397 1590 19 h 56 m 27 h 12 m 0 h 07 m 0 h 30 m 1 h 17 m 7 h 25 m 1 h 26 m 6 h 34 m 1 h 40 m 7 h 35 m 2 h 28 m 14 h 39 m 25 h 54 m 63 h 34 m TIN 469 4 h 20 m 11 h 42 m 10 h 03 m 15 h 08 m 2 h 05 m 6 h 26 m 49 h 44 m 22

Future Directions – Grid Construction • • Interpolate leaves in parallel Test other interpolation methods Test with more data sources Finding the “ideal” resolution Andrew Danner 23

Future Directions – Noise Removal • • • Bridge detection/removal Hydrological conditioning with carving Scoring of sinks based on volume Other flow routing methods Further flat routing improvements Andrew Danner 24

Flow Routing and Bridges Use flooded terrain for connectivity but Use original terrain for routing Andrew Danner 25

Questions? Andrew Danner 26