Grads Py A New Python Interface to Gr

Grads. Py: A New Python Interface to Gr. ADS Jennifer M. Adams ADNET Systems, Inc. Goddard Earth Sciences (GES) Data and Information Services Center (DISC)

Gr. ADS: A Handy Tool for Data Access, Analysis, and Visualization

Keep All Your Tools Sharp

Features of Gr. ADS … it is easy to learn … it is optimized for geophysical data … it handles grids and station observations in all the standard formats … it does fast I/O with file aggregation and indexing … it has a programmable interface for scripting … it can track and respond to user’s mouse operations in the display … it uses smart defaults, but the graphics are highly customizable … it has an active forum with many loyal users … it runs on Unix and Mac OS X (and MS Windows) … it has been under active development since 1988 (30 years!) … it is free open source software, licensed under GPLv 2

Gr. ADS is Used Worldwide

GOES VIS Image with Radar, 500 mb Heights, and SLP 18 z 4 June 2009

OLR and Precipitation Data from Gr. ADS in Google Earth 05 Z 23 May 2009

2016 Daily Surface Temperature Mean, Minimum, and Maximum at Reagan National Airport

The Open. Gr. ADS Project “Seeks to develop advanced interfaces and extensions based on the main Gr. ADS engine” • User-defined commands and functions • Interface to scripting languages Perl and Python • A central repository and collaborative environment to coordinate the development of extensions

The Gr. ADS Data Server (GDS) • An OPe. NDAP server that provides metadata, subsetting, and server-side analysis services • Analysis expressions are evaluated where the data reside on disk; only the analysis result is delivered to the remote client • GDS can host any Gr. ADS-readable dataset • GDS translates all data formats into Net. CDF

A Gr. ADS Schematic

Recent Gr. ADS Developments • • • User-defined plug-in functions replace the old UDFs Logical operators added to expression syntax New Gr. ADS function: if() New scripting language function: sys() Graphics display and printing handled as plug-ins: Ø Ø Ø Cairo library (for display and image/hardcopy output) Traditional X 11 display (with widgets) GD library for image output • Configuration overhaul: all libraries linked dynamically • New Python interface: Grads. Py

User-Defined Plug-Ins Improve performance and flexibility for users who customize Functions load dynamically when invoked by user Data is operated on while sitting in memory Plug-ins must be written in C …but can be linked with FORTRAN libraries • User Defined Plug-in Table (UDPT) describes them for Gr. ADS • • # Type # ---- function gxdisplay gxprint Name Full path to shared object file ------------------ gasort $HOME/grads/plug-ins/gasort. so Cairo ^. . /libgxd. Cairo. so Cairo ^. . /libgxp. Cairo. so X 11 /opt/foo/grads/libgxd. X 11. so GD /opt/foo/grads/libgxp. GD. so Alias -----

Dependent Libraries (The Hidden Cost in Adding Features) readline • ncurses • png • jpeg • zlib • gd • jasper • grib 2 c • udunits • hdf 4 • hdf 5 • curl • netcdf • dap • gadap • shapelib • tiff • geotiff • cairo • freetype • pixman • pkgconfig • Xrender • xml 2 •

What is Grads. Py? Grads. Py is a Python extension module written in C. The software components are: • C program: gradspy. c and gradspy. h • Python script: setup. py • Subroutines in Gr. ADS to pass data to/from Python • Gr. ADS compiled as a dynamically loading library libgradspy. so libgradspy. dylib Grads. Py is present in Gr. ADS version 2. 2. 1+

The Original Grads. Py Use Case Example of a clumsy workflow: 1. 2. 3. 4. 5. Use Gr. ADS I/O to carve model output in GRIB format Write out intermediate results in Net. CDF format Continue data analysis with R and MATLAB Write out intermediate results (again) Create graphics with Gr. ADS Python is the connective tissue that can seamlessly integrate multiple tools: ⋙ import numpy ⋙ import gradspy ⋙ import rpy 2 ⋙ import matlab. engine ⋙ do stuff

Methods of Grads. Py START : CMD : RESULT : PUT : Invoke Gr. ADS with desired switches and arguments Issue a command to Gr. ADS Retrieve a grid using a Gr. ADS expression Create a defined grid object in Gr. ADS ⋙ int = gradspy. start ("arg 1", "arg 2", …) ⋙ string = gradspy. cmd ("any command") ⋙ tuple = gradspy. result ("expression") ⋙ int = gradspy. put (tuple) RESULT tuple = [rank, data, xvals, yvals, zvals, iinfo, dinfo] PUT tuple = ["name", data, xvals, yvals, zvals, iinfo, dinfo]

Sample Python Code for RESULT import numpy as np import gradspy as gp from mpl_toolkits. basemap import Basemap import matplotlib. pyplot as plt gp. start ('-lb') gp. cmd ('open /Users/jmadams 1/data/model. ctl') res = gp. result ('ps') if res[0] < 0 : print 'We have a problem …' else : m = Basemap (projection='cyl', lon_0=180, lat_0=0, resolution='c’) cs = m. pcolormesh (res[2], res[3], res[1]) m. drawcoastlines (linewidth=0. 6) m. drawparallels (np. arange (-80, 81, 40), labels=[1, 0, 0, 0], fontsize=8) m. drawmeridians (np. arange (20, 341, 80), labels=[0, 0, 0, 1], fontsize=8) m. colorbar (cs, location='right', pad='5%') plt. title ('Surface Pressure (mb)') plt. savefig ('result. png', dpi=300)

Sample Python Code for PUT import numpy as np import gradspy as gp import h 5 py datafile='OMCLDO 2 G. he 5' f = h 5 py. File(datafile, "r") var = f["/HDFEOS/Fields/Terrain. Height"] # Array with integer metadata iinfo = np. ndarray (shape=(14), dtype=int) iinfo[0] = Xsize iinfo[1] = Ysize iinfo[2] = Zsize iinfo[3] = Tsize iinfo[4] = Esize iinfo[5] = start. YR iinfo[6] = start. MO iinfo[7] = start. DY iinfo[8] = start. HR iinfo[9] = start. MN iinfo[10] = Tincr iinfo[11] = Ttype iinfo[12] = Tcal iinfo[13] = Estrt # Array with float metadata dinfo = np. ndarray (shape=(6), dtype=float) dinfo[0] = Xstart dinfo[1] = Xincr dinfo[2] = Ystart dinfo[3] = Yincr dinfo[4] = Zstart dinfo[5] = Zincr # Array with variable data (replace Fill. Value with Na. N) data = np. ndarray (shape=var. shape, dtype=float) var. read_direct (data) FV = var. attrs. get("_Fill. Value") np. place (data, data==FV, np. nan) # If XYZ are linear, level values not needed empty = np. ndarray (shape=(1), dtype=float) empty[0] = np. nan # Assemble the tuple tup=["var", data, empty, iinfo, dinfo] # Start Gr. ADS, open a file, pass the data gp. start (‘-l’) gp. cmd ('open OMCLDO 2 G_grid. ctl’) gp. put (tup) # Did it work? gp. cmd ('q define var’) # Plot the variable gp. cmd (‘clear’) <optional commands for graphics control> gp. cmd (‘display var’) gp. cmd ('gxprint terrain. png x 1100 y 850’)

Gr. ADS at GES DISC In Operations: Giovanni (used for Hovmoller diagrams) Gr. ADS Data Servers are hosting these products: • GPM and TRMM • GLDAS and NLDAS • MERRA and MERRA 2 In the Sandbox (s 4 psci): /usr/bin/grads (2. 0. 2) /opt/grads-2. 1. a 2. oga. 1/grads (Open. Gr. ADS) /opt/grads/bin/grads (2. 1. a 3) /usr/local/bin/grads (2. 1. 1. b 0) /home/jmadams 1/bin/grads (2. 2. 1) On My Mac: OS X High Sierra 10. 13. 6

Gr. ADS at GES DISC User Support: How-To’s and data recipes Gr. ADS is a client for all OPe. NDAP data sets Gr. ADS should handle all L 2 G/3/4 products Level 2 swaths ~ Gr. ADS station data Internal Users: Casual visualization and analysis tasks Data management Format translations Quality graphics for publications, presentations, posters, and other interactions with the community

Q: Where is Gr. ADS Going? Gr. ADS is set to become a traditional open source project • Well-established and stable source code • Git workflow with Master, Dev, and Feature Branches • Collaboration and contributions from the community Hooks are in place for: • • • Customized functions Pythonic workflows New/Modified graphics modules Improved autoconfiguration macros Script libraries