Climate Grid An open source gridding tool for

Climate Grid An open source gridding tool for climate monitoring Dan Hollis, Ian Edwards, Mark Mc. Carthy, Mike Kendon, Gareth Dow, Tim Legg and Ian Simpson 11 th European Conference on Applied Climatology, Trieste, Italy 12 -16 September 2016

Climate Grid Aim: Develop a portable, modular and traceable code base following open software standards to provide a tool kit for the generation, exploration and visualisation of UK climate statistics.

What are we replacing? Oracle Arc. View FORTRAN Excel AVENUE Fireworks Grid Archive Dreamweaver & CMS intranet & external web

Climate Grid Components Oracle CSV files Data retrieval Gridding Local files Grid Archive Product Creation Product Delivery Regional Values Database intranet & external web

Open Source Codebase

Development process Test-driven development Iris 1. 8 Python library for analysing and visualising meteorological and oceanographic data sets. http: //scitools. org. uk/iris/ Code refactoring

Data retrieval many dates many variables iris cube (points) Gridding Grid Archive interpolation Product Creation iris cube (raster) Product Delivery net. CDF

Data retrieval Convert to anomalies Regression analysis Gridding Grid Archive IDW interpolation Product Creation Re-combine regn model & residuals Product Delivery Convert to actuals

Data retrieval Gridding Grid Archive Product Creation Product Delivery Provisional Final LTA Pre-QC Test Automated Research Historic CF-net. CDF ESRI Grid

Data retrieval Gridding Product-specific scripts Maps Regional values Grids Point values Grid Archive Product Creation Product Delivery Process-specific scripts Monthly, provisional Daily, final Services Ad hoc products

Data retrieval Product Creation Regional Values Database Gridding Grid Archive Product Creation Product Delivery • New database – still to be designed. . . • Postgre. SQL? • Accessible to all users • Will hold national and regional values • Much easier to add new areas • New code required to create graph and table products

Data retrieval Oracle Data retrieval Gridding Grid Archive Product Creation Extract Validate coordinates Transfor m Filter for blanks Aggregate Filter for missing data Create cube Filter for nonstandard sites Add CF metadata Filter for quality Add station metadata Remove zeroweighted sites Product Delivery

Data retrieval Gridding Grid Archive Product Creation • Have started writing some products (e. g. map images) directly to our intranet web server • Experimenting with writing scripts to send products (e. g. gridded datasets) directly to customers Product Delivery

Typical system architecture /climate-grid-core /climate-grid-uk Python code Station data (CSV files) /resources Metadata, ancillary files, templates /grid_archive Output from the gridding process /product_archive Maps, grids, regional values etc /configuration System configuration files /input_data

System Configuration • Paths to system resources • CF metadata • Grid definition (extent, projection, resolution etc) • Gridding method (by variable, month and run type) • Legends and colours for maps Indexed via ‘short_name’ = a string combining the temporal resolution and variable name • e. g. monthly_maxtemp • e. g. daily_rainfall

Additional components • Command line utility $ climategrid --setup $ climategrid --test $ climategrid --docs • Manual QC using QGIS

Documentation • Work in progress! • Documentation generated using Sphinx • ipython notebooks used to create a ‘user manual’ • Several potential use cases identified: • Standard products from existing grids • Bespoke products from existing grids • Gridding of user’s data using standard methods • Exploration/testing of new gridding methods

Benefits – Automation extract_station_data() csv file gridding() net. CDF file product_creation() png file product_delivery() Scheduling using cron

Benefits – Automation Near-real time monitoring

Benefits – Automation Routine services to customers • We have started routine delivery of gridded datasets to several customers • The script: • exports the grids from the archive • creates a ‘readme’ file from a template • zips up all the files • sends the zip file by email • The process assumes the grids exist! Potential for issues when the upstream process is disrupted. . .

Benefits – Portability Creating grids for the Philippines • Two visiting scientists from PAGASA (1 week) • Created ‘cgph’ (to replace ‘cguk’) • Input data were climate normals for temperature and rainfall for 56 sites • Gridding process was regression against elevation + IDW • Had to define a new output grid (UTM Zone 51) • Had to modify our code to handle different projections • Product scripts still in ‘cguk’ – used GIS to visualise results • Need to research best gridding methods for the Philippines • Installation on Windows?

Benefits – Portability Creating grids for the Philippines Max Temp July 1981 -2010 Rainfall July 1981 -2010

Benefits – Flexibility Evaluating an amateur observing network wow. metoffice. gov. u k

Benefits – Flexibility Evaluating an amateur observing network ~350 Met Office sites ~700 WOW sites

Benefits – Flexibility Evaluating an amateur observing network Climate Grid used to interpolate Met Office stations to WOW sites Mean Bias (deg C) Day-time warm bias RMSE ~2. 5 deg C Temperature Difference (deg C)

Benefits – Flexibility Evaluating an amateur observing network RMSE of interpolating Met Office sites to themselves ~1. 1 deg C

Generic Next Steps products Regional values DB Automated end-to-end Open Scheduling source tools process QC Sample data No more Excel Gridding Direct macros! methods connection to Oracle DB Documentation

Generic Next Steps products Regional values DB Automated end-to-end Open Scheduling source tools process QC Sample data No more Excel Gridding Direct macros! methods connection to Oracle DB Documentation

Summary Successful transition Maintained continuity Greater flexibility Improved skills Work in progress Future collaboration ncic@metoffice. gov. uk