Iowa Environmental Mesonet Using Open Source GIS Tools

Iowa Environmental Mesonet: Using Open Source GIS Tools and Web Services to Disseminate Environmental Data Daryl Herzmann Iowa State University 7 Nov 2007 Geospatial Interoperability Summit

Data, data, everywhere data 7 Nov 2007 Geospatial Interoperability Summit

7 Nov 2007 Geospatial Interoperability Summit

The Desperate Challenge The data comprising the IEM is made available to us in the following formats and network protocols. • Disparate Formats – – – – Grib METAR Text CSV Free form text ADAS III SHEF Excel (gasp!) 7 Nov 2007 • Disparate Network Protocols – – – HTTP FTP LDM IDD (tcp) TCP stream Modems 2007 Geospatial Interoperability Summit

Why an Open Source solution? 1. They are the best tools out there. UNIX mentality -> Do something small and do it well. 2. Transparency. We are a public institution doing public funded research. 3. Excellent Support • • 7 Nov 2007 Public bug trackers Community forums IRC Motivated developers 2007 Geospatial Interoperability Summit

Open Source Solution Post. GIS / Postgre. SQL Spatial Database 7 Nov 2007 Python Data processing and network services Map Server Mapping Engine + Scripting GDAL Swiss Army Knife of all things GIS Red Hat Linux Operating System PHP Dynamic Website 2007 Geospatial Interoperability Summit

Key Component: Post. GIS • Early IEM code (2001) was a series of flat files in Shapefile format and database of observations. This was a disaster. • Post. GIS provided the opportunity to manage data, metadata, and the GIS data in one location. – Network scalability – Transaction / stored procedure support – Speed / functionality with merge of GEOS library 7 Nov 2007 Geospatial Interoperability Summit

Look at all of my data! • 1 billion observations • 1 terabyte of data – Archives of NEXRAD, GOES Satellite, precipitation datasets, web cameras!, Weather Service products, ……. • Now what? Swim in all of my data? 7 Nov 2007 Geospatial Interoperability Summit

Here’s an idea, lets create a webpage where they can download the data! • Daryl: – Creates webpage • End user: – Finds my page – Fills out form – Saves data, attempts to figure out format – Attempts to find georeferencing for their GIS. – Hopefully loads data into app – Commits numerous errors along the way 7 Nov 2007 Geospatial Interoperability Summit

The Open Geospatial Consortium (OGC) Way 7 Nov 2007 Geospatial Interoperability Summit

7 Nov 2007 Geospatial Interoperability Summit

In 2002, we built OGC services, but they didn’t come…. • Why? – Where were the cool clients? – ESRI’s OGC support wasn’t there yet – It all felt slow as compared with local disk access – A bit ahead of its time 7 Nov 2007 Geospatial Interoperability Summit

In 2007, they have come! • Thanks to cool clients – Google Maps – Open. Layers – ESRI Interop Extension – Map Server and others • ~ 2 million hits per day (~20 “maps” per second) 7 Nov 2007 Geospatial Interoperability Summit

Supporting the Load • 12 systems comprise IEM web farm • Various levels of redundancy, but no spatial database redundancy yet • Roughly 30 GB in , 60 GB out each day • Linux IPVS provides redundant entry point and load balancing 7 Nov 2007 Geospatial Interoperability Summit

Available OGC Services 7 Nov 2007 Geospatial Interoperability Summit

http: //mesonet. agron. iastate. edu/ogc/ • CONUS GOES (Infrared, Visible) • CONUS NEXRAD Base Reflectivity, Storm Total Precip, 1 Hour Precip • NWS issued advisories, watches, and warnings • Iowa Rainfall Estimates • Iowa Road Conditions • IEM Data Services (temp, winds, etc) 7 Nov 2007 Geospatial Interoperability Summit

Raw NEXRAD Composites Have a large amount of noise due to ground clutter. A simple GDAL script comparing radar echos to its elevation quickly gets rid of most noise. 7 Nov 2007 Geospatial Interoperability Summit

Cleaned Composite 7 Nov 2007 Geospatial Interoperability Summit

Some IEM developed GIS webapps 7 Nov 2007 Geospatial Interoperability Summit

7 Nov 2007 Geospatial Interoperability Summit

7 Nov 2007 Geospatial Interoperability Summit

7 Nov 2007 Geospatial Interoperability Summit

7 Nov 2007 Geospatial Interoperability Summit

7 Nov 2007 Geospatial Interoperability Summit

7 Nov 2007 Geospatial Interoperability Summit

4 May 2003, heavy rains on freshly tilled soils 25 Sep 2005, heavy rains on mature crops 7 Nov 2007 Geospatial Interoperability Summit

GIS + webcams IEM developed software to have the cameras automatically track the sun and the moon. Can track NEXRAD identified storms as well! Done using FOSS GIS tools. 7 Nov 2007 Geospatial Interoperability Summit

What others have done. 7 Nov 2007 Geospatial Interoperability Summit

7 Nov 2007 Geospatial Interoperability Summit

Lessons Learned • Adopting OGC services can greatly reduce the complexity of serving out your data and using your own data! • Performance hasn’t been an issue, most of the overhead is with HTTP. • For Map Server, try to limit the number of mapfiles you maintain! 7 Nov 2007 Geospatial Interoperability Summit

Future Work • Increase Metadata! • Help support Net. CDF in GDAL (? ) • Work with end users on getting IEM data into their platform. 7 Nov 2007 Geospatial Interoperability Summit

Time for Questions? Daryl Herzmann 515. 294. 5978 akrherz@iastate. edu 7 Nov 2007 Geospatial Interoperability Summit