WIGOS Metadata Representation XML Schema Jrg Klausen Meteo

WIGOS Metadata Representation & XML Schema Jörg Klausen, Meteo. Swiss Dominic Lowe Tom Kralidis, Environment Canada

Reference documents • WIGOS Metadata Standard –… Need URL here • WIGOS Metadata Schema – https: //wmo. projecthut. com/svn/wmdr/ – User: reviewer / password: Ican. Review 2 • WIGOS Guide – http: //www. wmo. int/pages/prog/www/wigos/WGM. html • OSCAR/Surface – http: //oscar. wmo. int/surface

Outline • • • Introduction Formalizing WIGOS metadata Understanding the WIGOS metadata model XML schema definition (XSD) & Schematron Examples?

INTRODUCTION

WIGOS Metadata Standard 1. Observed variable 2. Purpose of observation 3. Station/ platform 4. Environment 5. Instruments & methods of observation 6. Sampling 7. Data processing and reporting 8. Data Quality 9. Ownership and Data Policy 10. Contact

WMDS is a descriptive standard • • WMDS describes concepts and principles 10 categories Mix of general and specific metadata items Ambiguous without further specification Need formal specification of metadata items Need cardinalities Need «best practice» guidance material

OSCAR/Surface metadata sources • NMHSs are primary source of information • Integrate existing metadata automatically to reduce burden

FORMALIZING WIGOS METADATA

Formalizing WIGOS Metdata From Standard to Use validate Conceptual Principles Formal Specification WIGOS Metadata Standard WIGOS Metadata Model Application Rules WIGOS Metadata Schema • Templates • «Best practice» • Custom tools Packaged Content WIGOS Metadata XML Files OSCAR/Surface WIGOS Metadata Archive API submit generate WIGOS Metadata Use API extract

What is a formal specification? • List allowed elements • Specify cardinalities – 0. . 1 (optional, at most one) – 0. . * (optional, many allowed) – 1 (mandatory, exactly one) – 1. . * (mandatory, at least one) • Specify hierarchy between elements – «A» depends on «B» • More documentation

UNDERSTANDING THE WIGOS METADATA MODEL

Formal WIGOS Metadata Model class High. Level. Overview Information about Facilities and Equipment WMD Record «Feature. Type» Abstract. Environmental. Monitoring. Facility «Feature. Type» WIGOSMetadata. Record + + + + + deployment : Deployment [0. . *] equipment : Equipment [0. . *] equipment. Log : Equipment. Log [0. . *] extension : Any [0. . *] facility : Observing. Facility [0. . *] facility. Log : Facility. Log [0. . *] facility. Set : Facility. Set [0. . *] header. Information : Header observations : Observation [0. . *] + + + «Feature. Ty. . . Facility. Set INSPIRE or other properties can be included in WIGOS via the extension property. additional. Description : Character. String [0. . 1] extension : Any [0. . *] geopositioning. Method : Geoposistioning. Method. Type [0. . 1] geospatial. Location : Timestamped. Location [0. . *] online. Resource : CI_Online. Resource [0. . *] responsible. Party : CI_Responsible. Party «Data. Type» Timestamped. Location 0. . * +facility + + 1. . * «Data. Type» Header location : GM_Object valid. Time. Period : TM_Period «Feature. Type» Observing. Facility + file. Date. Time : Date. Time [0. . 1] + record. Owner : CI_Responsible. Party [0. . 1] + + + «Data. Type» Observation belongs. To. Set : Character. String date. Established : Date. Time observations : Observation [1. . *] «Feature. Type» Equipment «Phase 3» + altitude. Or. Depth : Altitude. Or. Depth. Type [0. . 1] + climate. Zone : Climate. Zone. Type [0. . 1] + local. Topography : Local. Topography. Type [0. . 1] + relative. Elevation : Relative. Elevation. Type [0. . 1] + surface. Cover : Surface. Cover. Type [0. . 1] + surface. Cover. Classification : Surface. Cover. Classification. Type [0. . 1] + surface. Roughness : Surface. Roughness. Type [0. . 1] + topographic. Context : Topographic. Context. Type [0. . 1] + observation : OM_Observation + program. Affiliation : Program. Affiliation [1. . *] «Data. Type» Program. Affiliation + + 0. . * + +equipment. Host +hosted. Equipment 0. . 1 «voidable» + territory. Name : Territory. Type + wmo. Region : WMORegion. Type +facility. Log «Phase 2» «Data. Type» Instrument. Operating. Status + instrument. Operating. Status : Instrument. Operating. Status. Type + valid. Period : TM_Period +sub. Equipment 0. . * drift. Per. Unit. Time : Character. String [0. . 1] observable. Range : Character. String [0. . 1] observing. Method : Observing. Method. Type observing. Method. Details : Character. String [0. . 1] specification. Link : URI [0. . 1] specified. Absolute. Uncertainty : Character. String [0. . 1] specified. Relative. Uncertainty : Character. String [0. . 1] Observing facility «Phase 3» + firmware. Version : Character. String [0. . 1] + manufacturer : Character. String [0. . 1] + model. Number : Character. String [0. . 1] + serial. Number : Character. String [0. . 1] «Phase 2» + facility. Type : Observing. Facility. Type [0. . 1] + program. Affiliation : Program. Or. Network. Affiliation. Type + program. Specific. Facility. Id : Character. String + valid. Period : TM_Period [0. . 1] «WIS» -type metadata Extensions «Informative» Observing. Capability 1 +equipment 1 +deployed. Equipment 1 +equipment. Log «Phase 2» 0. . 1 Log «Feature. Type» Facility. Log Equipment «Feature. Type» Equipment. Log Details of the Observations, the Deployments and the Observing Process Identified. Type the observed. Property is a reference to a definition of the observed property. «Metaclass» Property. Type 1 +observed. Property Phenomenon +related. Observation 0. . * Any. Feature 0. . * Observation «Feature. Type» OM_Observation + + + parameter : Named. Value [0. . *] phenomenon. Time : TM_Object result. Quality : DQ_Element [0. . *] +generated. Observation result. Time : TM_Instant Process. Used 0. . * valid. Time : TM_Period [0. . 1] +property. Value. Provider 0. . * Domain +feature. Of. Interest 1 +procedure alternative. Metadata. Reference : CI_Citation [0. . *] contact : CI_Responsibility [1. . *] date. Info : CI_Date [1. . *] default. Locale : PT_Locale [0. . 1] metadata. Identifier : MD_Identifier [0. . 1] metadata. Linkage : CI_Online. Resource [0. . *] metadata. Profile : CI_Citation [0. . *] metadata. Standard : CI_Citation [0. . *] other. Locale : PT_Locale [0. . *] parent. Metadata : CI_Citation [0. . 1] extension : Any [0. . *] + + + «Feature. Type» Process + Deployment control. Schedule : Character. String [0. . 1] instrument. Operating. Status : Instrument. Operating. Status [0. . *] maintenance. Schedule : Character. String [0. . 1] documentation. Ref : Generic. Name [0. . 1] +data. Generation +result «Data. Type» Schedule «type» Any {root} 1. . * Intention «Feature. Type» Result. Set + distribution. Info : MD_Distribution [0. . *] «Feature. Type» SF_Sampling. Feature + + + + + positional. Accuracy : DQ_Positional. Accuracy [0. . 2] SF_Spatial. Sampling. Feature should be used as the Feature Of Interest to provide the spatial geometry/domain of the Observation. 1. . * «Data. Type» Data. Generation diurnal. Base. Time : Date. Time +schedule end. Hour : int end. Minute : int 1 end. Month : int end. Weekday : int start. Hour : int start. Minute : int start. Month : int start. Weekday : int + valid. Period : TM_Period +reporting lineage : LI_Lineage [0. . 1] parameter : Named. Value [0. . *] «Feature. Type» SF_Spatial. Sampling. Feature + «Feature. Type» Process + «Feature. Ty. . . OM_Process 1 «informative» + + + application. Area : Application. Area. Type [1. . *] + height. Above. Local. Reference. Surface : Measure [0. . 1] + local. Reference. Surface : Local. Reference. Surface. Type [0. . 1] + source. Of. Observation : Source. Of. Observation. Type +deployment + valid. Period : TM_Period «Phase 2» 1. . * + communication. Method : Data. Communication. Method. Type [0. . 1] + exposure : Exposure. Type [0. . 1] + representativeness : Representativeness. Type [0. . 1] METCE «Phase 3» Process + configuration : Character. String [0. . 1] Range «Feature. Ty. . . GFI_Feature +sampled. Feature «Feature. Type» Deployment MD_Metadata + + +metadata + + Metadata 0. . 1 + + + 1 «Data. Type» Reporting «Data. Type» Sampling + + diurnal. Base. Time : TM_Clock. Time sampling. Strategy : Sampling. Strategy. Type [0. . 1] «Phase 3» + sample. Treatment : Sample. Treatment. Type [0. . 1] + sampling. Procedure : Sampling. Procedure. Type [0. . 1] + sampling. Procedure. Description : Character. String [0. . 1] + temporal. Sampling. Interval : TM_Period. Duration [0. . 1] «Phase 3, voidable» + sampling. Time. Period : TM_Duration [0. . 1] «Phase 2» + spatial. Sampling. Resolution : Measure [0. . 1] + spatial. Sampling. Resolution. Details : Character. String [0. . 1] +sampling 1 +processing 0. . 1 «Data. Type» Processing «Phase 2» + aggregation. Period : TM_Period. Duration [0. . 1] + processing. Centre : Character. String [0. . 1] «Phase 3» + data. Processing : Character. String [0. . 1] + software. Details : Character. String [0. . 1] + software. URL : URI [0. . 1] + + + data. Use. Constraints : Data. Policy. Type [0. . 1] reference. Datum : CD_Vertical. Datum [0. . 1] spatial. Reporting. Interval : Measure [0. . 1] temporal. Reporting. Interval : TM_Period. Duration time. Stamp. Meaning : Time. Stamp. Meaning. Type [0. . 1] uom : Measurement. Unit. Type «Phase 3» + data. Format : Data. Format. Type [0. . 1] + data. Format. Version : Character. String [0. . 1] + latency : TM_Period. Duration [0. . 1] + numerical. Resolution : Measure [0. . 1] + official. Status : Boolean [0. . 1] «Phase 2» + level. Of. Data : Level. Of. Data. Type [0. . 1] + reference. Time. Source : Reference. Time. Type [0. . 1] Data. Generation • Schedule • Sampling • Processing • Reporting

«Feature. Type» WIGOSMetadata. Record «Feature. Type» WIGOSMetadata. Record + + + + + deployment : Deployment [0. . *] equipment : Equipment [0. . *] equipment. Log : Equipment. Log [0. . *] extension : Any [0. . *] facility : Observing. Facility [0. . *] facility. Log : Facility. Log [0. . *] facility. Set : Facility. Set [0. . *] header. Information : Header observations : Observation [0. . *] • A container for various sections • All sections are optional • Enables documentation of partial WMD records, e. g. – a list of observing facilities only – a subset of observations at a facility

«Feature. Type» «Feature. Ty. . . Facility. Set + + + Observing. Facility «Feature. Type» INSPIRE or other Abstract. Environmental. Monitoring. Facility properties can be additional. Description : Character. String [0. . 1] included in WIGOS via the extension property. extension : Any [0. . *] geopositioning. Method : Geoposistioning. Method. Type [0. . 1] geospatial. Location : Timestamped. Location [0. . *] online. Resource : CI_Online. Resource [0. . *] responsible. Party : CI_Responsible. Party 0. . * +facility 1. . * «Feature. Type» Observing. Facility + belongs. To. Set : Character. String + date. Established : Date. Time + observations : Observation [1. . *] «Phase 3» + altitude. Or. Depth : Altitude. Or. Depth. Type [0. . 1] + climate. Zone : Climate. Zone. Type [0. . 1] + local. Topography : Local. Topography. Type [0. . 1] + relative. Elevation : Relative. Elevation. Type [0. . 1] + surface. Cover : Surface. Cover. Type [0. . 1] + surface. Cover. Classification : Surface. Cover. Classification. Type [0. . 1] + surface. Roughness : Surface. Roughness. Type [0. . 1] + topographic. Context : Topographic. Context. Type [0. . 1] «Phase 2» + facility. Type : Observing. Facility. Type [0. . 1] «voidable» + territory. Name : Territory. Type + wmo. Region : WMORegion. Type • Observing. Facility describes a measurement station / site / platform /observatory / … • Facilities can be grouped into sets – Different WIGOS IDs – Different sites • Documentation can be extended to serve specific community needs – Interpreters will likely ignore extensions

«Feature. Type» Identified. Type «Metaclass» Property. Type 1 +observed. Property OM_Observation • OM_Observation used to describe (a time series of) observations of an observed. Property obtained using an OM_Process • OM_Observations can be related, e. g. – Several variables observed by same instrument Phenomenon +related. Observation 0. . * Any. Feature 0. . * «Feature. Type» OM_Observation + + + parameter : Named. Value [0. . *] phenomenon. Time : TM_Object result. Quality : DQ_Element [0. . *] +generated. Observation result. Time : TM_Instant Process. Used 0. . * valid. Time : TM_Period [0. . 1] «Feature. Ty. . . OM_Process +procedure 1

«Feature. Type» Equipment, Deployment «Feature. Type» Abstract. Environmental. Monitoring. Facility + + + • Deployments describe additional. Description : Character. String [0. . 1] extension : Any [0. . *] geopositioning. Method : Geoposistioning. Method. Type [0. . 1] geospatial. Location : Timestamped. Location [0. . *] online. Resource : CI_Online. Resource [0. . *] responsible. Party : CI_Responsible. Party «Feature. Type» Equipment +sub. Equipment 0. . * + drift. Per. Unit. Time : Character. String [0. . 1] + observable. Range : Character. String [0. . 1] + observing. Method : Observing. Method. Type + observing. Method. Details : Character. String [0. . 1] + specification. Link : URI [0. . 1] + specified. Absolute. Uncertainty : Character. String [0. . 1] + specified. Relative. Uncertainty : Character. String [0. . 1] «Phase 3» + firmware. Version : Character. String [0. . 1] + manufacturer : Character. String [0. . 1] + model. Number : Character. String [0. . 1] + serial. Number : Character. String [0. . 1] 1 +deployed. Equipment «Feature. Type» Deployment + application. Area : Application. Area. Type [1. . *] + height. Above. Local. Reference. Surface : Measure [0. . 1] + local. Reference. Surface : Local. Reference. Surface. Type [0. . 1] + source. Of. Observation : Source. Of. Observation. Type + valid. Period : TM_Period «Phase 2» + communication. Method : Data. Communication. Method. Type [0. . 1] + exposure : Exposure. Type [0. . 1] + representativeness : Representativeness. Type [0. . 1] «Phase 3» + configuration : Character. String [0. . 1] + control. Schedule : Character. String [0. . 1] + instrument. Operating. Status : Instrument. Operating. Status [0. . *] + maintenance. Schedule : Character. String [0. . 1] – when, where, why (application. Area), what Equipment has been used – configurations, maintenance and calibration routines – instrument operating status as a fxn of time • Deployments can be parallel or consecutive • Deployments also describe the Data. Generation processes

«Data. Type» Data. Generation + + + + + • Data. Generation involves «Data. Type» Data. Generation «Data. Type» Schedule diurnal. Base. Time : Date. Time +schedule end. Hour : int end. Minute : int 1 end. Month : int end. Weekday : int start. Hour : int start. Minute : int start. Month : int start. Weekday : int + valid. Period : TM_Period +reporting 1 «Data. Type» Reporting «Data. Type» Sampling +sampling + diurnal. Base. Time : TM_Clock. Time + sampling. Strategy : Sampling. Strategy. Type [0. . 1] «Phase 3» + sample. Treatment : Sample. Treatment. Type [0. . 1] + sampling. Procedure : Sampling. Procedure. Type [0. . 1] + sampling. Procedure. Description : Character. String [0. . 1] + temporal. Sampling. Interval : TM_Period. Duration [0. . 1] «Phase 3, voidable» + sampling. Time. Period : TM_Duration [0. . 1] «Phase 2» + spatial. Sampling. Resolution : Measure [0. . 1] + spatial. Sampling. Resolution. Details : Character. String [0. . 1] +processing 0. . 1 «Data. Type» Processing «Phase 2» + aggregation. Period : TM_Period. Duration [0. . 1] + processing. Centre : Character. String [0. . 1] «Phase 3» + data. Processing : Character. String [0. . 1] + software. Details : Character. String [0. . 1] + software. URL : URI [0. . 1] + data. Use. Constraints : Data. Policy. Type [0. . 1] + reference. Datum : CD_Vertical. Datum [0. . 1] + spatial. Reporting. Interval : Measure [0. . 1] + temporal. Reporting. Interval : TM_Period. Duration + time. Stamp. Meaning : Time. Stamp. Meaning. Type [0. . 1] + uom : Measurement. Unit. Type «Phase 3» + data. Format : Data. Format. Type [0. . 1] + data. Format. Version : Character. String [0. . 1] + latency : TM_Period. Duration [0. . 1] + numerical. Resolution : Measure [0. . 1] + official. Status : Boolean [0. . 1] «Phase 2» + level. Of. Data : Level. Of. Data. Type [0. . 1] + reference. Time. Source : Reference. Time. Type [0. . 1] – Sampling – Processing – Reporting according to a – Schedule • Multiple schedules can be defined, e. g. – Working days vs weekend – Winter vs summer

XML SCHEMA DEFINITION (XSD) & SCHEMATRON

What is what? • XSD – Formal validation rules • Schematron – Additional formal constraints that cannot be expressed in XSD • On-line validators exist for XSD – https: //www. freeformatter. com/xml-validatorxsd. html – http: //www. xmlvalidation. com/ • Stand-alone tools – XMLSpy

WMDR • Development version of schema – https: //wmo. projecthut. com/svn/wmdr/branches /development/xsd/wmdr. xsd • Development version of schematron – https: //wmo. projecthut. com/svn/wmdr/branches /development/xsd/schematron/ • Credentials – User: reviewer – Password: Ican. Review 2

EXAMPLES

Example 1 • https: //wmo. projecthut. com/svn/wmdr/branc hes/development/examples/WIGOSMetadata Record. Example. xml

How can you help? • Think about what information you would like to send to OSCAR/Surface – Full records of a single station? – Incremental changes of your entire network? – …? • Prepare «real-world» XML example files • Validate, share and discuss with the OSCAR/Surface team what works for you and what doesn’t

Thank you Merci ﺷﻜﺮﺍ ! i on i t za o Yo y & u ur an g r o • Financial support. Swiss Federal Office of Foreign Affairs, Meteo. Swiss, WMO, Met Norway • Project Team at Meteo. Swiss. (current) J Klausen, L Cappelletti, B Calpini, M Musa, M Brändli, L Koppa, C Walder, E Grüter, S Sandmeier, M Schäfer, A Rubli, Tom Hager, Attila Loos; (past) J Mannes, S Spreitzer, M Leutenegger, C Sigg, M Abbt, W Brunelli, J Mettler • Project Team at WMO (current). F Belda, LP Riishojgaard, T Pröscholdt • Project Team at European Dynamics (current). T Galousis, M Ulmann, L Christou, N Pappa, S Sklavos, … • • • ICG-WIGOS. S Barrell, B Calpini, … TT-WMD. (current) K Monnik, J Klausen, J Swaykos, T Boston, U Looser, E Büyükbas, Zhao Licheng, T Oakley, S Foreman, D Lockett, L Nunes IPET-MDRD. D Lowe, J Tandy, … • JCOMMOPS, GAW WDCs, ET-WDC, …