Key new features in the research data archive

Key new features in the research data archive Presented by the Data Support Section (DSS) of SCD Introduction ························· Steven Worley ERA-40 Data ························· Joey Comeaux NARR Data ·························· Chi-Fan Shih Near Real-time Data from the IDD ·········· Doug Schuster Metadata: Building Data Discovery and Access ··· Bob Dattore NSF/NCAR/SCD/DSS

Research Data Archive definition • Collection of reference datasets used in atmospheric and related sciences • Over 600 datasets • Add 10 -20 new datasets annually • Maintained by 9 staff • Established 40 years ago • All RDA on the MSS 548 K files 100. 5 TB NSF/NCAR/SCD/DSS

RDA Content • Many categories of data Observations Model Output (NWP & Reanalyses) Climatology Data Station summary data Satellite derived datasets Global topography Much more …. NSF/NCAR/SCD/DSS

Data User’s Functional Diagram for RDA Access MSS All RDA data RDA(DSS) Server Some data All Metadata NCAR User Internet User NSF/NCAR/SCD/DSS CDP Standardized Metadata

Data User’s Functional Diagram for RDA Access MSS All RDA data RDA(DSS) Server Some data All Metadata NCAR User Internet User metadata NSF/NCAR/SCD/DSS CDP Standardized Metadata

Data User’s Functional Diagram for RDA Access MSS All RDA data RDA(DSS) Server Some data All Metadata NCAR User metadata Internet User data NSF/NCAR/SCD/DSS CDP Standardized Metadata

Data User’s Functional Diagram for RDA Access MSS All RDA data RDA(DSS) Server Some data All Metadata NCAR User metadata Internet User future data NSF/NCAR/SCD/DSS CDP Standardized Metadata

User Metrics, January-April 2005 Archive Source Unique Users Data Amount (TB) NCAR MSS 242 19. 1 RDA Server 1604 6. 4 Highlight from RDA Server: Dataset Description Users Amount (GB) 2. 5° Tropospheric Analysis 123 36 1. 0° Tropospheric Analysis 480 1884 Global Upper Air Observations 128 23 Global Surface Observations 154 220 The most recent year of each archive is online NSF/NCAR/SCD/DSS

ECMWF ERA-40 Data from NCAR by Joey Comeaux Data Support Section NSF/NCAR/SCD/DSS

Model Specifics • • • Sept, 1957 – Aug, 2002 T 159 spectral truncation Reduced N 80 (~125 km) 60 Hybrid Levels Ingested one of the largest archives of observational data ever assembled • Atmospheric model coupled to an Ocean Wave model NSF/NCAR/SCD/DSS

PRODUCTS • ANALYSIS TIMES • Daily at 00, 06, 12 and 18 UTC • FORMAT: • GRIB • LOCATION: • All Data on MSS • Analysis Fields on RDA web server NSF/NCAR/SCD/DSS

Products Horizontal Resolution Pressure Surface Level Model Level T 159/N 80 ds 117. 1 ds 117. 2 2. 5 ds 118. 0 ds 118. 1 T 106/N 80 ds 127. 1 T 85/N 64 ds 124. 0 ds 124. 1 ds 124. 2 T 159/N 80 FCST ds 121. 0 ds 121. 1 ds 121. 2 Values in table indicate DSS dataset IDs Web pages available at dss. ucar. edu/datasets/dsnnn. n Monthly Means available for ALL products NSF/NCAR/SCD/DSS

PRODUCTS Variable Lists • Surface – 109 variables – Wind, temp, mslp, sfcp, cloud cover info, precip, fluxes radiation, stresses, Vertical Integrals, & more • Pressure Level – Z, T, W, RH, Q, Vort, Div, ozone mixing • Model Level – Same as pressure level + cloud liquid & ice water content, cloud cover NSF/NCAR/SCD/DSS

PRODUCTS • Other Products available 4 X Daily at Model Resolution Ocean Wave Analysis Fields Isentropic PV ± 2 PVU Ocean Wave Forecast Data ds 123. 0 ds 117. 3 ds 117. 4 ds 123. 1 Chemical Transport Net Tendency Radiative Tendency ds 117. 5 ds 117. 1 ds 117. 6 Feedback Records ? ? ? NSF/NCAR/SCD/DSS

NSF/NCAR/SCD/DSS

More Info Available • DSS ERA-40 web page – http: //dss. ucar. edu/pub/era 40 – WEB links in all DSS talks available in handout NSF/NCAR/SCD/DSS

North American Regional Reanalysis NARR Data Chi-Fan Shih 2/16/2022 NSF/NCAR/SCD/DSS 17

NARR Data Archive at NCAR http: //dss. ucar. edu/pub/narr • Introduction • Products • A Few Details • How to Access the Data • Summary NSF/NCAR/SCD/DSS Chi-Fan Shih

NARR Archive NARR & Eta-212 Domain NSF/NCAR/SCD/DSS

NARR Archive Introduction NARR Eta-212 32 km 29 levels Lambert-Conformal 3 hourly 1979 -2003 Gridded Data: GRIB-1 Analysis Monthly Means Climatologies On-line, Mass Storage 40 km 26 levels same 1995 May-continuing same Init, Anal, Fcst Mass Storage NSF/NCAR/SCD/DSS Chi-Fan Shih

NARR Archive Products 3 -hourly Data Raw Files from NCEP - daily “a” & “b” tar files, 466 MB/day - MSS - Total volume: 4. 2 TB, 1979 -2003 Regrouped Files - subgroups: 3 D, clm, flx, sfc, pbl - file sizes from 16 MB/day to 269 MB/day - tar files, on-line, MSS NSF/NCAR/SCD/DSS Chi-Fan Shih

NARR Archive Products Monthly Means - averages of all data in a month, monthly averages of data every 3 hours - from NCEP - from 1979 to 2003, total volume 144 GB - on-line, MSS Climatologies (Kingtse. Mo@noaa. gov ) - from NCEP - from 1979 to 2001, total volume 20 GB - on-line, MSS - including monthly means NSF/NCAR/SCD/DSS Chi-Fan Shih

NARR Archive Products Regrouped 3 -hourly Data 3 D: 10 variables, 269 MB/day, H, T, U, V, W, RH, . . on 29 pressure levels (from 1000 mb to 100 mb) clm: 12 variables, 16 MB/day, atmospheric column precipitable water, convective cloud, water vapor zonal & meridional fluxes, . . flx: 81 variables, 87 MB/day, U & V at 10 m, soil moisture & temperature, radiation, . . not at surface sfc: 44 variables, 50 MB/day, T, Pres, heat fluxes, radiation, . . at surface pbl: 39 variables, 45 MB/day, T, U, V, W, . . in layers between surface and 180 mb above ground NSF/NCAR/SCD/DSS Chi-Fan Shih

NARR Archive A Few Details • NARR has its own GRIB parameter table • wgrib with NARR GRIB table to get correct parameter names • U, V of NARR are different from U, V from operational Eta output • Constant (or fixed ) fields are on-line NSF/NCAR/SCD/DSS Chi-Fan Shih

NARR Archive How to Access the Data • NCAR Users - msrcp from MSS in /DSS/DS 608. 0/{3 HRLY_TAR, MONTHLY, CLIM ATOLOGIES, UCSD}/… • web – http: //dss. ucar. edu/pub/narr • ftp – to be implemented, contact chifan@ucar. edu • place a data order – tapes, etc. , fees apply • small amount of data – ncdc. noaa. gov NSF/NCAR/SCD/DSS Chi-Fan Shih

NSF/NCAR/SCD/DSS

NSF/NCAR/SCD/DSS

NSF/NCAR/SCD/DSS

NARR Archive Summary § All NARR data files are on-line on DSS RDA server as well as on MSS (in ds 608. 0 dataset) § http: //dss. ucar. edu/pub/narr § ftp download from DSS RDA server will be available § Examples for NCAR users to retrieve MSS files will be added § chifan@ucar. edu 303 -497 -1833 NSF/NCAR/SCD/DSS Chi-Fan Shih

Near Real-time Data from the IDD by Doug Schuster Data Support Section NSF/NCAR/SCD/DSS

Near Real-time Data from the IDD http: //dss. ucar. edu/datasets/ds 336. 0 http: //dss. ucar. edu/datasets/ds 335. 0 • Dataset Background • Observational Data Products • Model Data Products • IDD Data Access NSF/NCAR/SCD/DSS

Unidata’s Internet Data Distribution Service • • IDD Distributes Meteorological Data to private and public institutions subscribing to the service via internet nodes. • Data is captured by subscribers and used for real-time weather analysis, forecasts, model initialization, misc. research, etc… NSF/NCAR/SCD/DSS

Near Real-time Data from the IDD • Contents - Includes real-time observational and model data captured and archived from the IDD. • Purpose - Provide a resource to examine events in a near real-time setting (e. g. examine last week’s tornado outbreak). - Provide a permanent backup of the selected data to complement Unidata’s distribution service. NSF/NCAR/SCD/DSS

Observational Data, May 2003 - current http: //dss. ucar. edu/datasets/ds 336. 0 • Original reports, transmitted across the Global Telecommunication System (GTS), are captured and decoded by Unidata software. • NO QC of observations is performed. • The decoded reports are stored and archived in net. CDF format. • Used to supplement DSS’s ADP and ISH (DATSAV) dataset archives. NSF/NCAR/SCD/DSS

Available Obs Data, May 2003 - current http: //dss. ucar. edu/datasets/ds 336. 0 COVERAGE REPORT TYPE FREQUENCY Land Surface SYNOP 3 -Hourly Land Surface METAR Hourly Marine Surface BUOY Hourly Upper Air RAOB 3 -Hourly NSF/NCAR/SCD/DSS

00 UTC Upper Air Station Coverage Sample 00 Z IDD Upper Air Stations NSF/NCAR/SCD/DSS

Gridded Model Data, Dec 2002 - Current http: //dss. ucar. edu/datasets/ds 335. 0 • Stored in GRIB format on standard NCEP transmission grids. • Used to supplement gridded and reanalysis datasets. • Includes initialization and forecast grids. NSF/NCAR/SCD/DSS

Available Model Data, Dec 2002 - Current Model Init Times Forecast Length RUC 00 -21 Z @ 3 12 -hrs @ 3 - CONUS -hr int 81 km LC Grid 211 ETA/NAM 00 Z, 12 Z 60 hrs @ 6 -hr int 81 km LC Grid 211 GFSNH 00 Z-18 Z @ 6 -hr int 120 hrs @ 6 N. H. -hr int 381 km PS Grid 201 GFSEXT 00 Z 84 -240 hrs Global @ 12 -hr int 5˚ Lat/Lon NCEP ENS 00 Z, 12 Z 84 hrs @ 6 -hr int. 0 -90˚ N 1. 5˚ Grids 150 -330˚ E 39 and 40 ECMWF 12 Z 168 hrs @ 24 -hr int. Global NSF/NCAR/SCD/DSS Coverage Grid Res CONUS 5˚ Mercator

IDD Data Access • Most Recent 3 -months of data are available on RDA server. • RDA server archive populated daily. • RDA server archive copied to MSS weekly. • IDD Obs Data (May 2003 - current, RDA server & MSS) http: //dss. ucar. edu/datasets/ds 336. 0/data • IDD Model Data (Dec 2002 - current, RDA server & MSS) http: //dss. ucar. edu/datasets/ds 335. 0/data • NCEP Grid Definitions http: //www. nco. ncep. noaa. gov/pmb/docs/on 388/tableb. html • Schuster@ucar. edu, 303. 497. 1216 NSF/NCAR/SCD/DSS

Metadata: Building Data Discovery and Access Bob Dattore dattore@ucar. edu NSF/NCAR/SCD/DSS

History of Metadata • DSS has been compiling metadata in digital form for 20+ years • Dataset-level metadata for all datasets in a quasi-standard format • File-level metadata for all data files on the NCAR MSS • Format of this information varies by the dataset specialist who creates/enters it NSF/NCAR/SCD/DSS

Dataset-Level Metadata • Identifies and locates datasets having required data • Allows searches by “key” fields (e. g. date, parameter, project, platform, etc. ) • Drives main methods of discovery: • • • Search Engines Dataset Catalogs (precipitation, MM 5) Interaction with DSS Staff NSF/NCAR/SCD/DSS

File-Level Metadata • Identifies and locates files containing needed data • Gives information about the data (format, obs or gridded? , etc. ) • Gives information about the files (COSblocking, volume, etc. ) • Drives access methods: • Read directly from MSS (msrcp) • Download from web servers (DSS, CDP) NSF/NCAR/SCD/DSS

Goals for Improvement • Standardize all metadata • • DSS standards (XML) Controlled keyword lists Common formats for describing data files Easier to build software to process the metadata • Capture all the metadata necessary to completely describe our datasets and files • Allows us to map our metadata into other standards (THREDDS, DIF, etc. ) NSF/NCAR/SCD/DSS

Goals for Improvement • Use databases for quicker access • Faster searches for data discovery • Faster access to metadata by software (e. g. - program to subset a large dataset) • Create the ability to access information and data files over aggregations of data NSF/NCAR/SCD/DSS

NSF/NCAR/SCD/DSS

NSF/NCAR/SCD/DSS

NSF/NCAR/SCD/DSS

NSF/NCAR/SCD/DSS

NSF/NCAR/SCD/DSS

NSF/NCAR/SCD/DSS

NSF/NCAR/SCD/DSS

Improvements to the MSS File Lists • Provide more information about each data file • Allow users to create customized lists (e. g. - show only the files for 2004, only files containing temperature, etc. ) NSF/NCAR/SCD/DSS

The RDA on the CDP • The Community Data Portal is a collection of datasets from participating organizations • The next session, “New data portals for the community”, will talk about the CDP in more detail • All of our datasets are represented on the CDP NSF/NCAR/SCD/DSS

The RDA on the CDP • Datasets can be searched alongside those of other participants • As our metadata improves, so will the THREDDS catalogs • We are working to build MSS file lists for all datasets • Will allow NCAR users to access our MSS files through the CDP NSF/NCAR/SCD/DSS

NSF/NCAR/SCD/DSS

• Comments? • Questions? NSF/NCAR/SCD/DSS