Applying a metadata standard for international weather information

Applying a metadata standard for international weather information Gil Ross Met Office NESC Workshop September 2005 © Crown copyright Gil Ross NESC Workhop 6 th-8 th Sept 05 Page 1

Contents § WMO - previously introduced § Why a metadata standard for WMO? § Issues § Concepts and extensions § WMO Core © Crown copyright Gil Ross NESC Workhop 6 th-8 th Sept 05 Page 2

WMO - Weather Forecasting § Rapid exchange §Observations §Forecasts § Exchange formats §Brevity important §Unambiguous § language-independent §character §binary § “Obscure” © Crown copyright Gil Ross NESC Workhop 6 th-8 th Sept 05 Page 3

Surface Observations 0000 Z 3 rd September http: //www. metoffice. gov. uk/research/nwp/observations/data_coverage/index. html © Crown copyright Gil Ross NESC Workhop 6 th-8 th Sept 05 Page 4

Aircraft observations 000 Z 3 rd September © Crown copyright Gil Ross NESC Workhop 6 th-8 th Sept 05 Page 5

Numerical Weather Prediction Models : ~ Now (Q 3 2005) Ø Global ~40 km ~50 Level Ø North Atlantic/Europe 12 km ~38 level (50? ) Ø UK 4 km ~38 Level Ø Re-locatable Defence and Civilian Ø Trial Ensemble (global & regional) at half horizontal resolution ? © Crown copyright Gil Ross NESC Workhop 6 th-8 th Sept 05 Page 6

© Crown copyright Gil Ross NESC Workhop 6 th-8 th Sept 05 Page 7

WMO - climate § Observations § Model simulations § Interdisciplinary §meteorology §oceanography §biology § land § marine §hydrology §chemistry §………. . © Crown copyright Gil Ross NESC Workhop 6 th-8 th Sept 05 Page 8

Why metadata? § Expanding science § data not known § data not exploited § observations repeated § Exploitation § decision systems § non-experts § Need to: § discover § exchange © Crown copyright SMFW 01 NWBB 270000 AAXX 27004 91753 32481 51008 10331 20259 40078 58017 83202 333 20263 59018 83816 84078= 91754 01581 51812 10287 20245 40092 58017 60034 70182 85200 333 20256 59016 60017 85820= NNNN Gil Ross NESC Workhop 6 th-8 th Sept 05 Page 9

ISO 19115 and WMO § WMO has observer status at ISO § ISO - choose a “community profile” § not the whole lot (40+ standards) § WMO tasked an “Expert Team” to look at a “Core Profile” which would suit all WMO Commissions but which might be extended for particular themes. § approach was not Normative ISO § did not have the experience of ISO § did have WMO processes and protocols § ISO 19115 inadequate in important ways and over-the-top in others § so took shortcuts. © Crown copyright Gil Ross NESC Workhop 6 th-8 th Sept 05 Page 10

Shortcuts § First § Using ISO 19115 Data Dictionary, created an XML Schema and tried writing metadata § Then § Removed sections “not relevant” § Flattened tag structure § Added WMO relevant terms § Tried writing metadata § Before GML and 19139 § Did not fully use UML § Did not use extension mechanism (clunky) © Crown copyright Gil Ross NESC Workhop 6 th-8 th Sept 05 Page 11

Issues for meteorology § MD_Topic. Category. Code § Climatology/Meteorology/Atmosphere and Oceanography § The whole of our discipline! § Subdivision would help § Keyword structure inadequate § Probably need a taxonomy © Crown copyright Gil Ross NESC Workhop 6 th-8 th Sept 05 Page 12

Issues – Data Extent – Temporal Extent § Forecasts § Need absolute minimum of 3 times to define § probably 5 or more primary metadata times § Observation time (span) § Time forecast run § Validity time of forecast § Accumulation interval for, eg, rainfall § etc § Climate § use 360 day year © Crown copyright Gil Ross NESC Workhop 6 th-8 th Sept 05 Page 13

Issues – Coordinate Reference Systems § Irregular grids § Different co-ordinates §ECMWF store: § Spherical Harmonics (SH) § Gaussian Grid (GG) § Latitude/Longitude (LL) Coastal ocean model http: //oceanworld. tamu. edu/resources/ocng_textbook/chapter 15/Images/Fig 15 -3. htm German weather model (http: //www. dwd. de) © Crown copyright Gil Ross NESC Workhop 6 th-8 th Sept 05 Page 14

Non-”standard” grids On the left is the tripolar grid (Murray 1996, Griffies et al 2004) used by MOM 4 for the current IPCC model CM 2. In the middle is the cubed sphere (Rancic and Purser 1990) planned for the Finite-Volume atmosphere dynamical core for the next-generation GFDL models AM 3 and CM 3. On the right is another promising grid, the yin-yang grid (Kageyama et al 2004). © Crown copyright Gil Ross NESC Workhop 6 th-8 th Sept 05 Page 15

Issues – Data Extent – Vertical Extent § NWP models have high dimensionality § Temps, humidity, winds and flow variables, water and ice fractions, particulates, chemical abundances and rates defined at every point in 4 D § These can be used as coordinates other than linear distance § Field of temperature at a fixed pressure § Field of pressure at a fixed temperature © Crown copyright Gil Ross NESC Workhop 6 th-8 th Sept 05 Page 16

Issues – Data Extent – Vertical Extent § Physical co-ordinates § One parameter (eg “potential vorticity”) on a constant value of another (eg “potential temperature L R L © Crown copyright Gil Ross NESC Workhop 6 th-8 th Sept 05 R Page 17

Issues – Features and Coverages § ISO descriptions of features and coverages § Not relevant – not understood § threw it out! § First real test metadata § Decided we did need features and coverages § However STILL didn’t understand ISO description § Particularly coverages – § 3 dimensional fields § restrictive definition § Developed our own Feature Catalogue § Our coverages § flexible grids § one variable using other as co-ordinate © Crown copyright Gil Ross NESC Workhop 6 th-8 th Sept 05 Page 18

Data Quality and Lineage § ISO MD Data. Quality has 20 subclasses! § Did not understand § Truncated considerably – too much § Now appreciate that e. g. hydrography has to specify precision to metre, and different quality measures at different parts of the chart § Meteorological forecast quality § At 3 day forecast – if 6 hours and 100 km in error § We are pleased! © Crown copyright Gil Ross NESC Workhop 6 th-8 th Sept 05 Page 19

Issues - implementation § Travelling Metadata § What is it? Metadata that ‘travels’ with the data! § Unlike most data portal initiatives our ‘operational’ data is actively distributed throughout the global community of NMS’s; currently pushed around on the GTS § WMO already has travelling metadata of sorts … § the AHL of bulletins § What’s wrong with this? § Identifier Overloaded an identifier with semantics § Expert or programmatic parser … metadata should be open & easy to resolve § WMO is committed to providing ISO 191 xx “compliant” metadata © Crown copyright Gil Ross NESC Workhop 6 th-8 th Sept 05 Page 20

About ‘travelling metadata’ § What are the constraints on travelling metadata? § It must not ‘abuse’ network bandwidth § It must relate to the WMO Core Profile § Why not just use the standard WMO Core Profile? § XML is not particularly concise even at the best of times § XML schemas derived from ISO 19115 are definitely NOT concise § Travelling metadata must exhibit the following characteristics: § Concise (to minimise telecoms costs) § Provide enough information to conveniently identify the data it is paired with … e. g. UID, time, location, context, content (possibly implied by the context) © Crown copyright Gil Ross NESC Workhop 6 th-8 th Sept 05 Page 21

About ‘travelling metadata’ (3) § But if we can’t use WMO Core Profile how can it be ‘standard’? § Although travelling metadata does not CONFORM to the WMO Core Profile of ISO 19115 (because of its flattened structure) but should be DERIVED from the profile … using the same [global] elements § Furthermore, it is intended for use within a closed community, therefore does not NEED to conform to an open standard § However, travelling metadata should ALWAYS point to a service that can transform the travelling metadata into the standards compliant metadata for sharing outside our community © Crown copyright Gil Ross NESC Workhop 6 th-8 th Sept 05 Page 22

WMO Core § Subset of ISO 19115 § with extensions and interpretations § Testing § climate observations § http: //cliware. meteo. ru/wdccatalog/index_en. html § Arctic studies § http: //acsys. npolar. no/adis. php#Datasets § automated extraction from routine weather data § SIMDAT © Crown copyright Gil Ross NESC Workhop 6 th-8 th Sept 05 Page 23