TIGGE Archive Access at NCAR Steven Worley Doug

TIGGE Archive Access at NCAR Steven Worley Doug Schuster Dave Stepaniak Hannah Wilcox

Background on TIGGE WMO World Weather Research Programme THORPEX Ø THe Observing system Research and Predictability EXperiment Ø THORPEX Interactive Global Grand Ensemble (TIGGE) Archive supports research Ø Grand Ensemble = multiple NWP centers ensembles are combined (an ensemble of ensembles) Ø 10 international NWP Centers contributing to TIGGE

Background on TIGGE Three mirrored archive centers Ø NCAR Ø ECMWF Ø CMA {Shared System Development!} Ø Daily Data Flow Metrics Ø 245 GB Ø 1. 6 Million gridded fields as separate data packets

Data Receipt UKMO CMC NCAR ECMWF NCEP NCDC Meteo. France CMA KMA JMA IDD/LDM HTTP FTP Archive Centre Current Data Provider CPTEC Bo. M Unidata IDD/LDM Internet Data Distribution / Local Data Manager Commodity internet application to send and receive data

Major Challenges Insure data receipt, build complete archive Ø Collate data fields into different files types Ø Exchange manifest files as part of IDD/LDM data transmission between Archive centers Ø Verify send, receive Ø Automated resend requests for missing fields Ø Harvest and hold metadata in My. SQL DB’s Ø Identify location of every field in file set Ø Updated often Ø Critical for users interface and background data processing

Major Challenges Ø Limited online storage – 4 TB, ≅ 2. 0 weeks temporal coverage Ø Full archive on NCAR Mass Storage System Ø User registration and metrics required Ø Accept data policy; for research and education only Ø 48 hour delay from forecast initialization time

Major Challenges Ø Access system must accurately display what data are available as users make selections Ø Driven by multi-center research (Grand Ensemble) Ø E. g. , want to experiment with NCEP and JMA forecasts for 500 h. Pa temperatures Ø Maximum common data is determine by JMA Ø 12 Z forecast initialization and 9 -day forecast

Differences between centers

User access demonstration Animation, what you will see Ø Multiple centers Ø (ECMWF, UKMO, NCEP, CMA, CMC, KMA) Ø Fields/Parameters Ø (Geopotential Height, 2 m Temperature) Ø Levels Ø (500 h. Pa, Single Level) Ø Spatial and temporal ranges Ø (Global, 3 -days, 12 Z initialization, 48 hour forecasts) Ø Regridding to common spatial resolution Ø (1. 5°) Ø Output format Ø (net. CDF)

Sample Data Request for an Event

Retrieve Completed Subset

Subset Request Animation

Gustav/Hannah Animation

Lessons Learned Ø Manifest files and automated resend are critical for a complete archive Ø The impact of different contributions from the NWP centers across archive cannot be under estimated Ø There are important design considerations to insure prompt browser interactions Ø Caching data from the DB

Lessons Learned Ø Computational resource requirements ramp up quickly with multi-dimensional problems Ø D’s, center, ensemble member, parameter, forecast length, etc. Ø Archive file structure choices greatly impact subsetting ability Ø TIGGE currently based on synoptic order Ø Time-series by parameter could be better?

End http: //tigge. ucar. edu/

Future Ø Consider adding storage resources to keep a longer period of data online Ø Integrate more computation resources to reduce time to fill requests Ø Create a way to re-stage MSS data to online for internet users

Outline Ø Brief Background on TIGGE Archive Ø Data Receipt Ø Challenges Ø Access Ø Lessons Learn

User registration and metrics required Ø Conditions for use Ø Research and education only Ø 48 -hour delay on availability

Limited online storage – 5 TB, ≅ 2. 5 weeks Ø 1 -2 TB additional storage for user request preparation Ø Supplement Access Ø Direct Archive File download, though interface or Research Data Archive (RDA) Ø All data (200 TB) available through RDA from NCAR Mass Storage System Ø TC data?