Archiving of UERRA data in MARS and related

Archiving of UERRA data in MARS and related data services Richard Mládek, Manuel Fuentes, Sebastien Villaume, Shahram Najm, Matthew Manoussakis UERRA GA Tarragona 28 -30. 11. 2017

Highlights since the previous GA Final list of 9 UERRA datasets Ø MESAN dataset split to 2 versions v. New type=oi implemented in MARS & ECCODES(former GRIB-API) Ø COSMO v. Deterministic dataset out of scope v. Might be still archived during 2018 5 of 9 UERRA datasets fully archived Ø Grand total: 762, 937, 000, 463, 558 (693. 887 Tbytes) Ø Number of fields: 3, 637, 248, 971 3 datasets still being archived Ø UM datasets - completion expected by January 2018 UERRA GA Tarragona 28 -30. 11. 2017

UERRA datasets - Progress status https: //software. ecmwf. int/wiki/display/UER/Progress+status The former MESAN(SMHI) dataset renamed & split to 2 datasets (V 1/V 2) 3 temporary datasets (short period; expver=test in MARS) not needed any more Ø MESCAN-SURFEX already archived (less parameters; fc only +6 H; 2008) Ø UM & UM/En (full data but not the final model version; 2008) UERRA GA Tarragona 28 -30. 11. 2017

UERRA datasets (MARS set up) MARS keys (class=ur) origin stream type number remarks Production datasets (expver=prod) 1 COSMO edzw oper an/fc - Out of scope 2 COSMO/En edzw enda an/fc 1. . 20 Archived 3 HARMONIE/V 1 eswi oper an/fc - Archived 4 HARMONIE/V 2 eswi enda an/fc 1 Archived; not real eps 5 MESAN /V 1 eswi oper oi - Archived; only tcc 6 MESAN /V 2 eswi enda oi 1 Xmas 2017; only tcc; not real eps 7 MESCAN-SURFEX lfpw oper an/fc - Xmas 2017 8 MESCAN-SURFEX/En lfpw enda an/fc 1. . 8 Archived 9 UM/4 DVAR egrr oper an/fc - Jan 2018 10 UM/En 4 DVAR egrr enda an/fc 1. . 20 Jan 2018 New type=oi created for MESAN data in MARS (using external non-ECMWF GRIB exploring tools it will look like type=an) UERRA GA Tarragona 28 -30. 11. 2017

MARS achievements MARS & ECCODES design for UERRA Ø Fully WMO compliant GRIB 2 definitions for all UERRA parameters Ø Any UERRA data could be archived (some peculiarities because of late information) Ø New level type=hl (height levels) Ø New soil level type=sol v. Allowing varying grid dependant soil level depths – Not crucial for UERRA at the end but is ready for future demands Ø Parameters “since previous post-processing” (varying validity periods) v. Surface air maximum temperature since previous post-processing v. Surface air minimum temperature since previous post-processing v 10 metre wind gust since previous post-processing UERRA GA Tarragona 28 -30. 11. 2017

MARS achievements New parameters defined for UERRA & approved by WMO Ø Percolation Ø Downward short-wave radiation flux, clear sky Ø Upward short-wave radiation flux, clear sky Ø Downward long-wave radiation flux, clear sky Ø Soil heat flux (updated version) Data services Ø Dedicated data portal v. Data discovery and small sample retrievals Ø Web-API for large retrievals v. UERRA data usage statistics will be available UERRA GA Tarragona 28 -30. 11. 2017

UERRA web site at ECMWF Static pages Ø Data portal (not fully updated for datasets with ongoing archiving) Ø Issues with data (identified problems) Ø UERRA retrieval efficiency (important) Ø Support v. Contacts – providers’ contacts will be needed for post UERRA as well – Prefereably dedicated mailing list like uerra. support@. . . v. FAQ, Forum. . Ø Description of project, parameters, datasets. . v. The latest final models’ description required Ø Resources (info from development phase; info about data interpolation/visualization etc) UERRA GA Tarragona 28 -30. 11. 2017

UERRA web site at ECMWF Tracking actual state Ø Progress status (with links to the model specific log pages) Ø Parameter availability (with links to MARS Catalogue for each dataset) Ø Data production and archiving schedule (overview of completed and still expected periods) UERRA GA Tarragona 28 -30. 11. 2017

UERRA web pages at ECMWF UERRA GA Tarragona 28 -30. 11. 2017

UERRA data portals 1. Public access Ø Self registration still required before accessing the dedicated UERRA data portal 2. Users with ECMWF account Ø Full access including MARS catalogue UERRA GA Tarragona 28 -30. 11. 2017

UERRA data portals Public access UERRA GA Tarragona 28 -30. 11. 2017

UERRA data portals Users with ECMWF account Ø Full access to MARS catalogue Ø Direct links to MARS catalogue for each UERRA dataset from the page Parameters availability UERRA GA Tarragona 28 -30. 11. 2017

Model specific features Model levels Ø COSMO: 1. . 40 Ø HARMONIE: 1. . 65 Ø UM: 1. . 63 Soil levels Ø COSMO: 8 soil layers level borders at 0, 0. 01, 0. 03, 0. 09, 0. 27, 0. 81, 2. 43, 7. 29, 21. 87 m Ø HARMONIE: 3 soil levels/layers (sot on levels, vsw on layers) level depths are grid dependent Ø MESCAN-SURFEX: 14 soil layers (only 6 layers for preliminary shorter reanalysis runs) level borders at 0, 0. 01, 0. 04, 0. 1, 0. 2, 0. 4, 0. 6 m Ø UM: 4 soil layers level borders at 0, 0. 1, 0. 35, 1 and 3 m UERRA GA Tarragona 28 -30. 11. 2017

Model specific features Surface level parameters output frequencies https: //software. ecmwf. int/wiki/display/UER/Description Analysis: Ø UM, HARMONIE, MESCAN-SURFEX v analysis output in six hourly intervals (at 00/06/12/18 UTC) Ø MESCAN-SURFEX v precipitation analysis (24 h accumulation from 06 UTC to 06 UTC of next day) output at 06 TC Ø COSMO, MESAN v hourly analysis outputs for 1. . 23 UTC runs Forecast: Ø UM, COSMO v forecast output at T+1, 2, 3, 4, 5, 6, 9, 12, 15, 18, 21, 24, 27, 30 started at 00 UTC and 12 UTC v forecast output at T+1, 2, 3, 4, 5, 6 started at 06 UTC and 18 UTC Ø HARMONIE (started at 00/06/12/18 UTC): v sd, sr, al, rsn, slhf, sshf parameters (derived from ALADIN model driving SURFEX) at T+1, 2, 3, 4, 5, 6 only v all other parameters at T+1, 2, 3, 4, 5, 6, 9, 12, 15, 18, 21, 24, 27, 30 Ø MESCAN-SURFEX (started at 00/06/12/18 UTC): v parameters sp, tp, 2 r, 2 t, 10 si, 10 wdir, tidirswrf from forecast output at T+6 only (fields from ALADIN model driving SURFEX) v all other parameters from forecast output at T+1, 2, 3, 4, 5, 6 UERRA GA Tarragona 28 -30. 11. 2017

Model specific features COSMO Ø Model levels above 100 h. Pa not available Ø Step 0 not available in forecasts ("fc") because of nudging MESCAN-SURFEX Ø Contains analysis of total precipitation accumulated between 6 H of the previous day and 6 H of the day encoded in the GRIB 2 Ø 7 parameters sp, tp, 2 r, 2 t, 10 si, 10 wdir, tidirswrf from forecast output archived at T+6 only (fields from ALADIN model driving SURFEX) HARMONIE Ø Steps only up to +6 H available for parameters derived from ALADIN model driving SURFEX UM Ø Model level increases with height MO (opposite to other models) UERRA GA Tarragona 28 -30. 11. 2017

HIRLAM datasets HARMONIE/V 1 Ø Full 55 years archived (1961 -2015) HARMONIE/V 2 Ø Full 5 years period 2006 -2010 archived Ø Archived under stream=enda and number=1 but it is not ensemble MESAN/V 1 Ø Full 6 year period 2006 -2010 archived Ø New type=oi used in MARS due to technical reasons MESAN/V 2 Ø Expected by Xmas 2017; period 1993 -2006; the work in progress (new satellite data etc) Ø Will be archived under stream=enda and number=1 but it is not ensemble UERRA GA Tarragona 28 -30. 11. 2017

HIRLAM datasets 2 identified issues Ø Wrong 0% values in relative humidity, cloud cover and albedo fields v. There are some isolated spots of one or a few grid points with wrong zero values in the fields of relative humidity (both at 2 m and on pressure levels). It happened because of a wrong algorithm used for re -computation of the values from 0 -1 interval to 0 -100% (the supersaturation was set to 100% instead of 0% value). v. The same issue can be probably found in cloud cover and albedo fields as well but its extent there is not known. Ø Missing values in surface sensible heat flux and surface latent heat flux fields v. The sensible heat flux and surface latent heat flux fields appear to have missing values over the land because of that should not be used. UERRA GA Tarragona 28 -30. 11. 2017

COSMO datasets COSMO/En Ø Full 5 years period 2006 -2010 archived COSMO Ø Out of scope Ø Still might be archived during 2018 UERRA GA Tarragona 28 -30. 11. 2017

Met Office datasets UM/4 DVAR and UM/En 4 DVAR datasets Ø Simultaneous archiving (2 streams) Ø Completed periods: v 1979 -1989 v 2000 -2011 Ø In progress: v 1990 -1999 v 2012 -2016 UERRA GA Tarragona 28 -30. 11. 2017

Météo-France datasets MESCAN-SURFEX/En Ø Full 5 years period 2006 -2010 archived v. Re-archived from test to prod mode in MARS – Done quickly thanks to new MARS consolidation suite (@Sebastien Villaume) otherwise it would take many weeks(months) MESCAN-SURFEX/En Ø 40 years period 1961 -2000 archived Ø 2001 -2015 in progress (expected finish before Xmas 2017) UERRA GA Tarragona 28 -30. 11. 2017

ECMWF links for UERRA at ECMWF (high level page): http: //www. ecmwf. int/en/research/projects/uerra UERRA at ECMWF: https: //software. ecmwf. int/wiki/display/UER UERRA data portal: http: //apps. ecmwf. int/datasets/data/uerra UERRA retrieval efficiency : https: //software. ecmwf. int/wiki/display/UERRA+retrieval+efficiency Issues with data : https: //software. ecmwf. int/wiki/display/UER/Issues+with+data Parameter list: https: //software. ecmwf. int/wiki/display/UER/Parameters Parameter availability: https: //software. ecmwf. int/wiki/display/UER/Parameter+availability Expected archiving schedule: https: //software. ecmwf. int/wiki/display/UER/Data+production+and+archiving+schedule Data interpolation and visualization: https: //software. ecmwf. int/wiki/display/UER/Data+interpolation+and+visualization UERRA GA Tarragona 28 -30. 11. 2017