6 Metadata what are they SDMX metadata implementation

6. Metadata, what are they ? SDMX metadata implementation: The ESS Metadata Handler (ESS-MH) Fernando H MORENTE ORIA Eurostat, Unit B. 5 Data and metadata services and standards European Statistical Training Programme (ESTP) SDMX Basics course, 6 -8 March 2018, Luxembourg Eurostat 1

What’s metadata? • A set of data that describes and gives information about other data (Oxford dictionaries) • information that is given to describe or help you use other information (Cambridge dictionary) 2 Eurostat

Types of metadata Structural metadata • acting as identifiers and descriptors of the data, such as: • dimensions of statistical cubes • variables • titles of tables • nomenclatures (code lists) • always associated with the data to allow their identification, retrieval and browsing 3 Eurostat

Example for structural metadata 4 Eurostat

Types of metadata Reference metadata • acting only as descriptors of the data, they don’t help to actually identify the data • They can be of different kinds: • conceptual metadata • methodological metadata • quality metadata (process and output) • can be exchanged independently from the data they are related to, but are however often linked to them 5 Eurostat

Example for reference metadata 6 Eurostat

ESS standardisation of reference metadata based on SDMX • The SDMX Glossary (2016) • The SDMX Technical Standards: • The information model for creation of the Metadata Structure Definitions (MSDs) • The SDMX-ML for documenting the XML format • The Euro SDMX Registry for storing the MSDs etc. 7

Standardisation of structural metadata • Code lists describe dimensions in data tables, giving a meaning to the data • Code lists are based on: • official statistical classifications such as NACE, NUTS, ISCO… • the SDMX Content Oriented Guidelines • Domain specific codifications • A standard code list is a code list already harmonised • Standard code lists should be used all along the statistical business process: data design, collection, aggregation, dissemination, archiving… 8 Eurostat

Example of a harmonised code list (NACE Rev. 1. 1) Old version (before harmonisation) New version (after harmonisation) Domains Old codes Old label_en New codes New label_en hrst, htec MA_TOTAL Manufacturing sector fats MAN Manufacturing industries theme 3 RD Manufacturing industry theme 4 B 0200 Manufacturing industry D Manufacturing theme 8 SE 0_4 Manufacturing industry theme 9 TOT_MANUF Manufacturing industry ds, hrst, htec MA_LOW_TEC Low technology manufacturing sector LOT Low Technology (incl. following NACE codes: 15 -22; 36, 37) D_LTC Low-technology manufacturing inn I_LOW_TEC Low tech industries: NACE Rev. 1 codes 15 to 22, 36 and 37 hrst, htec SE_TOTAL Services: NACE Rev. 1. 1 sections G to Q = 50 to 99 G-Q Services fats SER Services sector fats / inn Eurostat 9

Impact on the statistical business processes • Better comparability: same codes for the same concepts • Increase efficiency: less transcoding; less code lists; clean lists • Improve accuracy: facilitate data management and exchange and reduce the number of errors • Re-usability and integration of the data: data warehouse are only possible if codes corresponding to the same concept are the same • SDMX implementation: it is essential for the implementation of a SDMX data/metadata exchange process • The ESS standard code lists will also be made available in the Euro SDMX Registry (currently RAMON) 10 Eurostat

RAMON http: //ec. europa. eu/eurostat/ramon 11 Eurostat

Standard Code Lists in RAMON 12 Eurostat

The ESS Reference Metadata Standards ESMS • Euro SDMX Metadata Structure ESQRS EPMS • ESS Standard for Quality Reports Structure • Eurostat Process Metadata Structure 13 Eurostat

The Euro SDMX Metadata Structure (ESMS) 14 Eurostat

The ESS Standard for Quality Reports Structure (ESQRS) 15 Eurostat

Integration of ESMS & ESQRS under SIMS 16 Eurostat

The Euro Process Metadata Structure (EPMS) Concept name 1 1. 2 1. 3 1. 4 1. 5 1. 6 1. 7 1. 8 2 3 4 4. 1 4. 2. 2 4. 3. 1 Contact organisation unit Contact name Contact person function Contact mail address Contact email address Contact phone number Contact fax number Summary process description Workflow Statistical processing Data collection Source data - integration Source data - coding Data validation in Member States 4. 3. 2 4. 3. 3 4. 3. 4 Validation rules agreed with Member States Data validation - detection (Eurostat) Data validation - correction (Eurostat) Concept name 4. 4. 1 4. 4. 2 4. 4. 3 4. 4. 4. 5. 1 4. 5. 2 5 5. 1 6 6. 1 7 7. 1 7. 2 7. 3 7. 4 8 8. 1 8. 2 8. 3 8. 4 8. 5 8. 6 Eurostat Data compilation - variables Data compilation - weights Data compilation - aggregates Data compilation - finalisation Data compilation - draftoutput Data validation - final Data validation final - output Data validation final - explanation Confidentiality - data treatment Release policy User access Dissemination format Publications On-line database Micro-data access Other IT applications for data reception/collection IT applications for data processing IT applications for data validation IT applications for data confidentiality IT applications for metadata Other IT applications 17

Dissemination of reference metadata 18 Eurostat

Dissemination of national reference metadata 19 Eurostat

The ESS Metadata Handler The business process Input from national metadata Metadata from the Eurostat Domain manager Eurostat as main administrator Euro SDMX Registry ESS – Metadata Handler ESS-MH IT application Common user Interface Output produced for the Eurostat Web Other output for Eurostat or external users Eurostat 20

What is the ESS Metadata Handler? • The ESS Metadata Handler is a web based application for reference metadata production, exchange and dissemination in the ESS • It implements the ESS metadata standards (ESMS, ESQRS and EPMS, etc. ) • It replaces EMIS (used in Eurostat) and NRME (for countries) • It contains many improvements based on users' feedback (in terms of business process and functionalities) • It’s been in production since 31 January 2014 21

National Statistical Institute EUROSTAT ESS Metadata Handler (ESS MH) ESS MH Database Eurostat Website National Metadata File EDAMIS National Metadata File PRODUCTION TREATMENT & ANALYSIS DISSEMINATION 22

The business process for using the ESS MH for national metadata • Mapping of the existing national reference metadata files to the ESMS and/or ESQRS formats • Conversion of existing national reference metadata files into standard structure • Insertion of these files into the ESS MH application • The NSI’s are asked to complete, enhance their converted files, directly in the ESS MH • The responsible Domain Managers in Eurostat are asked to validate these ESMS / ESQRS files • The national metadata are finally disseminated on Eurostat Web site (if decided so) 23

Result • National and European reference metadata files that include under a standard structure, sources and summary information regarding data quality and the production process in general 24

Future developments • Update of the SDMX Glossary • Creation of a global MSD 25

Practical Example Updating a national reference file 26

Hands on exercise • Your line supervisor has asked you to update the HICP reference metadata file available in the ESS MH, knowing that: • - The index reference period has been modified to 2016=100 • - The reference year should also be updated (use your pc number as reference year) • Note: Since the other metadata information remains up-todate, you may copy the latest available file and implement the changes 27

Questions?