NCMP NOAA National Climate Model Portal NOMADS and

NCMP NOAA National Climate Model Portal NOMADS and NCMP: Data Quality Considerations from a Data Service Provider ESIP Summer Meeting Santa Fe, NM July 13 2011 Glenn K. Rutledge NOMADS/NCMP Project Manager NOAA’s National Climatic Data Center Asheville, NC 28801 M. Sutton 1995

NCMP NOAA National Climate Model Portal NOMADS/NCMP Background ØNOMADS is a distributed data access project for access to real-time and retrospective high volume numerical weather prediction and climate models. Conceived in 1999 - operational in 2002. ØNOAA’s NCDC initiated NOMADS with NCEP, GFDL, PCMDI, NCAR. PMEL and others. The collaboration quickly grew under a distributed data access philosophy. Ø Founding member of GO-ESSP. Focus on NWP. NCMP will address climate models, diagnostics, and interoperability for model-to-obs intercomparisons. FY 10 Distinct hosts served: 83, 370 Successful requests: ~125 million 1 day record: 4. 9 TB 2 Original Collaborator’s

NCMP NOAA National Climate Model Portal NOMADS Access Infrastructure NODC NWS Reanalysis. Org GFDL … NOMADS Climate Clearinghouse Community vetted observational database GDS UAF LAS ESG CDAT TDS NOAA Climate Model Portal NCMP - Global Interoperability Program - National Climate Predictions and Projections Center NOAA Cooperative Institute Climate and Satellites CICS-NC Asheville NOAA Climate Assessment Services Collaborations Earth System Grid Framework Distributed Archive Services Users Private or Unpublished results NCDC Rutledge/ 2011

NCMP • NOAA National Climate Model Portal Archive services supported by NCDC is available at http: //nomads. ncdc. noaa. gov/. This server provides – – • Existing NOMADS-NCMP Servers access to most of NCEP's operational data sets a long-term archive for all data sets many other data sets (see http: //nomads. ncdc. noaa. gov/data. php? name=inventory) A non-operational research and development server (and developing ESG node) managed by NCDC is available at http: //nomads 6. ncdc. noaa. gov/ncep_data/index. html A real-time server supported 24 x 7 by NCEP is available at http: //nomads. ncep. noaa. gov/. This server provides – access to NCEP's operational data sets as they are being generated – a short-term archive of up to a month for most data sets – 24 x 7 operational monitoring by NCEP staff – a geographically-diverse backup server to insure operational availability • Four non-operational research and development servers used for customer testing of new products and services prior to operational implementation – These servers are not guaranteed to have current data and their content are supported only during business hours and on the basis of staff availability. Three non-operational research and development servers managed by NCEP are available at • http: //nomad 1. ncep. noaa. gov/ncep_data/index. html • http: //nomad 3. ncep. noaa. gov/ncep_data/index. html • http: //nomad 5. ncep. noaa. gov/ncep_data/index. html • Ocean-NOMADS at http: //edac-dap 2. northerngulfinstitute. org/ocean_nomads/. This server provides most NCEP and some Navy Ocean Models.

NCMP NOAA National Climate Model Portal NOMADS Data Availability NWP Model and Climate Reanalysis Global Forecast System (GFS), 1 and ½ degree March 2004 – Present, October 2006 - Present NCEP Climate Forecast System Reanalysis (CFSRR) Global 32 km January 1979 -Present NCEP Climate Forecast System Reforecast (CFSRR) Global 32 km January 1979 – Present NCEP North American Regional Reanalysis (NARR) 30 years 32 km January 1979 - Present NCEP/NCAR/DOE R 1 & R 2 Global Reanalysis Jan 1948 – Present, Jan 1979 - Present NCEP Global Ensembles (GENS) / TIGGE (w/ NCAR) December 2007 - Present ESRL Twentieth Century Reanalysis Project (20 CR) (2011) January 1850 -Present NCEP Spectral Statistical Interpolation (SSI) Global Data January 2001 - Present Assimilation System (GDAS) w/ model restart February 2002 -Present North American Mesoscale (NAM, formally Eta) 12 km February 2005 - Present Rapid Update Cycle (RUC) 20 km and 13 km January 2006 – Present, March 2007 - Present Climate Data / Coupled AOGCM Limited GFDL CM 2. 0 and CM 2. 1 Climate Experiments Limited AR 4/5 Paleoclimate Model Intercomparison Project (PMIP) POR CMIP 3/5 multi-model suite via ESGF (2011+) AR 4/5 Downscaled CMIP 3/5 (2011) AR 4/5 Observational In-situ NCDC Global Historical Climate Network (GHCN) Temp/Precip Jan 1880 – Present, Jan 1900 - Present NCDC Integrated Global Radiosonde Archive (IGRA) upper air Varies by station NCDC Smith-Reynolds Extended Reconstructed and OI ¼ SST’s Jan 1854 – Present, Jan 1985 – Present Service Records Retention System (SRRS) April 2001 - Present 5

NCMP NOAA National Climate Model Portal Model Archive Growth Exebyte (EB) • Since 1966, model output volume closely mirrors “Moore’s Law”: - a doubling of volume (CPU processing speed) approximately every two years*. • Derived estimates below show existing archive storage requirements without data reduction practices. Volumes approach 1 Exabyte (1 Billion Gigabytes). • Simple data reduction policies to remove forecast products, and outdated reanalysis reforecasts, and Atmospheric-Ocean General Circulation Models (AOGCM’s) at the Archive level greatly reduces archive and stewardship requirements. 6 * unpublished results (Rutledge, Alpert 2004)

NCMP NOAA National Climate Model Portal CFSv 2 Retrospective Forecasts Timeseries of 9 Month Means 2. 5° Verification Data_v 2 Dumps CFSRR Level 2 Data CFSv 1 Huh? T 126 L 64 CFS High-Priority OCNF 1979 -2010 Seasonal Means 1982 -2009 CDAS CFS HIC Reanalysis 1. 0° Retrospective Forecasts Low-Resolution 45 -Day Runs CFSR CFS First-Look CDAS 2 GFS Reforecast 1999 -2010 CFS-R Hindcast Daily Climatologies CFSR-Lite OCNH T 382 L 64 Version 1 PGBF FLXF 1981 -2004 CFS Time Series GDAS 9 -Month Runs 0. 5° 1982 -2010 Monthly Climatologies Level 1 Data 45 Day/1 Season Timeseries CFSv 2 Operational Monthly Means Version 2 Tier 1 Data Calibration Climatologies T 574 CFSv 2 IPVF CFS Operational Monthly Means of 9 Month Reforecasts High Priority Subset 7 High-Resolution set Level 3 Data CFS LIC

NCMP NOAA National Climate Model Portal Architectural Design Functional Areas Project Management Planning & Engineering Climate Science Applications Application Development Sector Engagement Systems Technology Dataset Management: (we work to organize our data for on-line users: e. g. , aggregations) • Model Data Dissemination Support • •

Data Quality for Archives: OAIS Documentation before data goes into Archive Input from Producer Archivist Maintained “What to Archive” process Submission Agreement 9

NCMP NOAA National Climate Model Portal Data Quality Considerations: Where can I find the data I want? • Metadata-based search and discovery • Extending distributed access mechanisms • Technology, Standards & tools such as – – – nc. ISO, ACDD, ISO-19157 ESGF (Grid. FTP) ERDDAP / GI-cat / GI-go UAF “Clean Catalogs” esri Geo. Portal

NCMP NOAA National Climate Model Portal NOMADS/NCMP Data Quality Considerations GIS Sub 1 km modeling issues: - Oblate spheroid (aka, Earth) to sphere At present there are no commonly used transformations when going from one to the other CV CV EARTH MODEL No Datums included in datasets Shapefile defined region to gridded data Shapefiles: Bound Regions 11 Grids defined by Lat & Lon Bounds ?

The Importance of Data Quality and Data Stewardship (When it’s wrong, it’s wrong. . ) A scenario to consider. . Suppose a scientist uses some observations of a biological nature (which were made incorrectly) and used with supplementary external data that has not been well calibrated. The resultant relationships one derives (and which may be used to help drive public policy or millions of dollars in infrastructural investment) from these data however strong the value will be spurious. . (We all lose. . ) 12

Quality Data: Observations and Models (one can not survive without the other) Model representations of the climate system and their development are derived from relationships that are found by examining high quality observational data Theory and then mathematical implementation In order to push modeling forward we will have continuing needs for larger volumes of higher frequency and quality, inter-calibrated and rigorously documented observational data Model validation and use in assimilation systems (NWP) 13

Modeled Data: A wish list When examining model output a scientist wants data that is internally consistent and a good representation of what the model was doing (model sampling must be consistent and documented) ◦ ◦ A large suite of output variables, (one can never have enough) Consistent time models, (360 day year, no leap years, etc. ) Comprehensive Standardized metadata (variable names, etc. ) Time averaged output-(6 hourly, monthly), and is documented on how it was aggregated ◦ Standard output formats (e. g. , CF-netcdf) 14

Data Quality and Model Intercomparison Projects (MIPs) Standardization of data structure facilitates the rapid prototyping of code and pushes model evolution forward-(A good thing) Model to Observation intercomparisons offer unique capabilities to both verify models and assure the quality of the observations employed With new satellite simulators being incorporated in climate models the consistency of these observed measures will be able to be directly addressed-(An apples to apples comparison) 15

NOAA National Climate Model Portal Intercomparison Projects Global Surface Temperature ⁰C Climate of the 20 th century INGV_ECHAM 4 CSIRO 3. 5 GISS_E_H Climate Scenarios R 2 reanalysis This plot is dynamically generated on the fly from CMIP 3 data on NOMADS 16 A 2 Scenario: : Average of 19 different models and average of last ten annual values = 17. 00 ⁰C A 1 B Scenario: : Average of 24 different models and average of last ten annual values = 16. 45 ⁰C B 1 Scenario: : Average of 20 different models and average of last ten annual values = 15. 51 ⁰C

High Quality Model data that adheres to standards allow us to quickly show meaningful model results (source: NOMADS/NCMP) 20 C 3 M Climate of the 20 th century runs (Annual averages): Global variance of surface temperature (C). NCEP/NCAR R 2 reanalysis (1948 -2000) = 0. 064 GFDL 2. 1 (1948 -2000) = 0. 071 GFDL 2. 1 (2000 -2099) A 2 GFDL 2. 1 (2000 -2099) A 1 B GFDL 2. 1 (2000 -2099) B 1 = 0. 841 = 0. 553 = 0. 193 A ratio of variance of annually averaged surface temperature Surface temperature variability would massively increase crop production… 17 SRESA 2

Science Means Collaborations Data Standards (although not glamorous) do help us communicate and work more efficiently with other federal agencies and like efforts including: NOAA, NASA, USGS, NWS, DOE, EPA, private entities and DOD. Also becoming a major player in Climate: Insurance and Risk Management companies These standards also help us reach out internationally as well. A good example is the Program for Climate Model Diagnosis and Intercomparison (PCMDI), which houses all participant IPCC model runs and all data adheres to the CF-netcdf data standard. (because of this the data are widely used within the community) 18

NCMP NOAA National Climate Model Portal Glenn. Rutledge@noaa. gov NCDC Asheville, NC M. Sutton 1995 NCMP http: //nomads. ncdc. noaa. gov tions? NOAA National Climate Model Portal Ques NASA

NCMP NOAA National Climate Model Portal BACKUP NCMP 20 NOAA National Climate Model Portal

NCMP NOAA National Climate Model Portal Select Bibliography Rutledge, G. K. , J. Alpert, and W. Ebisuzaki, 2006: NOMADS, a climate and weather model archive at the National Oceanic and Atmospheric Administration. Bulletin of the American Meteorological Society, 87 (3), 327 -341. Saha, Suranjana, and Coauthors, 2010: The NCEP Climate Forecast System Reanalysis. Bull. Amer. Meteor. Soc. , 91, 1015– 1057. Compo GP, et al. , 2011. The Twentieth Century Reanalysis Project. Q. J. R. Meteorol. Soc. 137. 2010 BAMS Global State of the Climate Report, 2011: (in press) 21

NOMADS Ensemble Probability Tool PDF’s on the fly NOMADS Ensemble Probabilities on the fly: 20 model runs 30 fcst projections 10 days of forecast 22

NCMP NOAA National Climate Model Portal NOMADS / NCMP Technology Enhancements (cont. ) • Incorporating THREDDS-based data into interactive mapping applications • Utilizing OGC components from THREDDS (WMS, WCS, etc. )

NCMP NOAA National Climate Model Portal NOMADS / NCMP Technology Enhancements • Extending NOMADS into interactive and sector-utilized formats and technologies Example: THREDDS Catalog Navigation in Adobe® Flash®

NCMP NOAA National Climate Model Portal Architecture

NCMP NOAA National Climate Model Portal Services • Data Access Services – THREDDS Data Server • OPe. NDAP access form • Net. CDF Subset service • GIS/WCS and WMS capabilities • Raw data file server • Limited aggregation capabilities – Grads Data Server • OPe. NDAP access for Gr. ADS compatible data • Full aggregations to ease access – Web applications • Ensemble probability tool: Easy interpretation of GFS ensembles • SRRS archive / NCEP charts access application 26 • Raw/Scriptable data access – Partial-HTTP subsetting – Anonymous FTP for select datasets – Bulk access through wget scripts • Archive Access – N-HAS offline data request system within NOMADS web interface – Bulk FTP access through N-HAS – OPe. NDAP enabled offline cache. – CLASS bulk and OPe. NDAP access proxy • Requirements and Outreach – User requirements – “Sectoral” engagement (energy, water, transportation, Ag, etc. ) – NCMP Program Plan