NARR User Issues Wesley Ebisuzaki CPCNCEP Wesley Ebisuzakinoaa

NARR User Issues Wesley Ebisuzaki CPC/NCEP Wesley. Ebisuzaki@noaa. gov http: wwwt. emc. ncep. noaa. gov/mmb/rreanl/

Topics ● ● ● Files: contents, availability, formats Tools: BUFR, GRIB Obtaining the data: NCEP, NCAR, NCDC Scripting downloads What is reality? Is reanalysis truth? An insider's viewpoint.

GRIB Files ● ● For grids (horizontal fields) such as 500 mb Z Lambert conformal grid: 32 km eta grid (#221) – ● Model grid: not available except at NCEP – ● ● Have grids of lat/lon, display programs No plans to make available, display programs? Analyses are on-line and forecasts ? 99% of people will use the GRIB files

BUFR ● ● ● For all input observations except SST, snow, ice, observed precipitation (GRIB) Input obs BUFR files have QC marks Analyses/forecasts at station locations (MOLTS) – ● Much smaller than GRIB files, limited locations Distribution (informal) – Includes restricted data – Few users (experience from global reanalysis) – Often other sources of data (in easier formats)

Misc Files ● ● ● Restart files: huge, full precision, NCEP-only Statistics: – Fit of obs to NARR, fit of obs to R 2 – Small, can be made available on-line Observation counts: – ● Small, should be put on-line PNG (monitoring plots) – Small, should be put on-line

Want more files? Don't expect too much. NARR has zero funding.

Despair? ● ● ● Global Reanalysis has been running this way for years! CPC has been running global reanalysis for its own operations. CPC and the outside community have shared interest in maintaining a stable assimilation system. NCEP has an interest in others using its products. However, limited resources, need to be selective, work with partners

Status ● ● ● Real-time is running (4 days late) Started building 1. 5 -2 year on-line rotating archive of latest analyses at NCEP 1979 -2003 merged files at NCAR, NCDC has 5 TB merged files on-line, trying to support the on-line community by NOMADS EMC people are working on other projects

Tools: BUFR ● ● Situation improving Gr. ADS can read bufr directly

Tools: GRIB ● Gr. ADS: display and computations – ● ● ● grib 2 ctl. pl: make Gr. ADS control files wgrib: inventory, decode GRIB files, remove records from a file (save space) lcgrib: regional subset (save space) copygb: change grid (save space or neurons) grbpoint. pl: grib -> point value printout (slow) Gempak (? ), NCL (? )

Example: Gr. ADS ● ● Make control file (analyses) – grib 2 ctl. pl -rr cpc_rr_mon. 197901 >my. ctl – grib 2 ctl. pl -rr cpc_rr_mon. %y 4%m 2 >my. ctl Make index file (analyses) – ● gribmap -0 -i my. ctl Run Gr. ADS – grads – open my. ctl

Example: wgrib ● ● Use: inventories, make files smaller, decode data fortran and C programs See contents of a grib file – ● wgrib (options) cpc_mon_197901 To extract records, decode records, see wgrib home page.

Example: lcgrib ● ● ● Regional subset of Lambert conformal grid *** MAKE SMALLER FILES *** lcgrib in-grib out-grib west south east north – West, south, east and north are in degrees – N and E are +ve, S and W are -ve

Obtaining Data: NCEP ● Rotating archive: latest 1. 5 – 2 years of analyses – ● ● http: //nomad 7. ncep. noaa. gov/ncep_data Other files: TBD NCEP not set up to deal with the public requests – Go through the web – Go through NCDC, NCAR

Obtaining Data: NCAR ● ● ● Have 1979 -2003 merged data Sure they will get more of the data Plans, availability?

Obtaining Data: NCDC ● ● NCDC has the merged data 1979 -2003 and want 2004 ASAP NCDC will use NOMADS to distribute the data to the on-line community. NOMADS is project of many institutions including NCDC and NCEP for distributing data over the internet. NOMADS-NCDC is taking software from NCEP and COLA to distribute GRIB data.

Data Access at NCDC ● ● Custom plots ftp 4 u: access by selected time, field, region http access to raw data (like FTP) GDS (Gr. ADS Data Server): Ope. NDAP server (formerly DODS), on-line access to data

Difficulties in on-line distribution ● ● ● Downloading a 5 TB data set is difficult Need ways to minimize data transfer Plots (maps) work but limited utility GDS needs to be optimized for large datasets ftp 4 u is a solution

FTP 4 U ● Only download data that you need – Select levels – Select variables – Select geographic regions – Select times

Making FTP 4 U Work ● ● Don't request more data than needed Make one big request, not many small requests of the same file; i. e. , make one request to get the winds, temp and heights rather than 3 requests. Do not request long time series. Shared resource (1 job at a time) Be prepared to script/automate the downloads – May be impossible to download in a few hours, may require days, weeks and multiple

Scripting FTP 4 U Downloads ● ● FTP 4 U is normally accessed by a browser Anything a browser can do, a script can also do Experience at NCEP is that most plot/ftp 4 u access is scripted (automated) NCDC's modifications to the NCEP provided programs make it more difficult to script the download but nothing that a c. URL user couldn't handle. NCDC wants to change!

Scripting Downloads cont'd ● ● Use c. URL, a unix/windows program, to “emulate” a browser Each browser page becomes a call to c. URL. With a little RTFM was able get script working sample program will be put on NARR home page

Scripting Downloads cont'd ● ● ● Downside: steep learning curve, may have no local experts for help Upside: can make large data transfers possible, can be used in other projects (ex. getting fcst data automatically, skewing online opinion polls) Given that we get more data off the web, it's a skill that someone in a large group should eventually have.

Obtaining Data: NCDC ● ● ● It's a learning situation for NCDC Give feedback (complain), it's the only way to improve the situation. Lessons learned here will help the next time you try getting a different dataset from NCDC's nomad server. Current issues: full disk (new RAID is will be installed soon), obtaining time series, scripting support

What is Reality? ● ● Some people consider reanalysis to be truth NARR is new, bound to be warts, already found some.

What is Reality? ● Accuracy of reanalysis depends on many factors – Data going into the analyses ● Types, counts, quality of observations – Accuracy of previous analysis – Accuracy of atmospheric/land model – Spatial and temporal filtering – NO EASY ANSWER

What is Reality? cont'd ● Accuracy – check to the appropriate data counts and biases ● Need to add our counts to web ● Need to add fit to data to web – check other reanalyses – Fit to observations – Consider sensitivity to modeling assumptions – Sensitivity experiments (data, resolution, model), only available to limited # of people

What is Reality? cont'd “Trust but verify” Ronald Reagan

URLs ● copygb: ftp: //ftpprd. ncep. noaa. gov/pub/cpc/wd 51 we/copygb – ● change grid type of grib files c. URL: http: //curl. haxx. se – ● change grid type of grib files download URLs, handles post data GEMPAK: http: //my. unidata. ucar. edu/content/software/gempak/index. html – ● graphics, can handle NARR files with some effort Gr. ADS: http: //grads. iges. org/grads/

URLs ● grbpoint. pl: ftp: //ftpprd. ncep. noaa. gov/pub/cpc/wd 51 we/wgrib. scripts – ● grib 2 ctl. pl: http: //www. cpc. ncep. noaa. gov/products/wesley/grib 2 ctl. html – ● f 95 compiler for copygb HIPHOP: http: //iacweb. etzh. ch/stall/dominik/hihop/ – ● makes control files for Gr. ADS g 95: http: //www. g 95. org/ – ● simple program to extract data from lat/lon grib support for idl lcgrib: http: //www. cpc. ncep. noaa. gov/products/wesley/gribw. html – regional subset of Lambert conformal grib files

URLs ● NARR: http: //wwwt. emc. ncep. noaa. gov/mmb/rreanl/index. html – home page for NARR ● NARR rotating archive: http: //nomad 7. ncep. noaa. gov/ncep_data ● NCAR Command Language (NCL): http: //ngwww. ucar. edu/ – ● NCDC data: http: //nomads. ncdc. noaa. gov/#narr_datasets – ● graphics, computations package that supports grib on-line supplier of NARR data wgrib: http: //www. cpc. ncep. noaa. gov/products/wesley/wgrib. html – inventory, slice & dice grib files