ESS Standards for reference metadata and quality reporting

ESS Standards for reference metadata and quality reporting (ESMS, ESQRS, SIMS) Giorgia Simeoni, Istat, simeoni@istat. it “Methods and tools for reference metadata documentation and communication” Project “Methods, quality and metadata” Division ESTP Training Course “Information standards and technologies for describing, exchanging and disseminating data and metadata” Rome, 19 -22 June 2018 THE CONTRACTOR IS ACTING UNDER A FRAMEWORK CONTRACT CONCLUDED WITH THE COMMISSION Eurostat

Metadata is data that defines and describes other data. Source: OECD Glossary of statistical terms Metadata: information that is needed to be able to use and interpret statistics. Metadata describe data by giving definitions of populations, objects, variables, the methodology and quality. Source: Eurostat's Concepts and Definitions Database Eurostat

Metadata can be classified as Eurostat

Eurostat

Eurostat http: //ec. europa. eu/eurostat/cache/metadata/en/tour_occ_esms. htm

Reference metadata are metadata describing the contents and the quality of the statistical data. Reference metadata should include all of the following: - "conceptual" metadata, describing the concepts used and their practical implementation, allowing users to understand what the statistics are measuring and, thus, their fitness for use; - "methodological" metadata, describing methods used for the generation of the data (e. g. sampling, collection methods, editing processes); - "quality" metadata, describing the different quality dimensions of the resulting statistics (e. g. timeliness, accuracy). Source: SDMX Metadata Common vocabulary Eurostat

Reference metadata and quality reporting Information on output quality is included in the concept of reference metadata. Information on methodologies applied during the statistical process are related to process quality. Information on concepts are connected to relevance, comparability and coherence of statistics, that are quality dimensions. Quality reporting and reference metadata are strictly connected Eurostat

Definition of Quality Reporting Quality reporting is the preparation and dissemination, on a regular or irregular basis, of reports conveying information about the quality of a statistical product or survey (Eurostat, 2003 b). A quality report provides information on the main quality characteristics of a product so that the user should be able to assess product quality. In the optimal case quality reports are based on quality indicators (Bergdahl et al. , 2007) Eurostat

Main features of a quality report • Object: a statistical product or statistical process • Target recipients: users or producers of official statistics • Purposes: to accompany official statistics dissemination, to support quality assessment • Contents: quality dimensions, quality indicators • Further characteristics: standard structure, different levels of detailored on recipient and purpose Eurostat

User oriented quality reporting • To communicate quality to users • To report on strengths and limits in order to support a proper use of statistics • To enhance transparency Concise, not much technical and focussed on the statistical output Eurostat

Producer oriented quality reporting • To support management monitoring • To identify points for further improvements • To produce reference documentation for quality assessment (e. g. : audit and self-assessment) Detailed, comprehensive, supplemented with quality indicators and focussed more on the statistical process, even if following product quality dimensions Eurostat

ESS Quality reporting: commitment European Statistical Law Regulation (EU) 2015/759 of 29 April 2015 amending Regulation (EC) No 223/2009 on European statistics. Art. 12 (3): “Member States shall provide the Commission (Eurostat) with reports on the quality of data transmitted, including any concerns they have regarding the accuracy of the data. The Commission (Eurostat) shall assess the quality of the data transmitted, on the basis of appropriate analysis, and shall prepare and publish reports and communications on the quality of European statistics” Eurostat

ESS Quality reporting: commitment The Single Integrated Metadata Structure (SIMS) version 2. 0 is approved as a ESS standard by the ESSC Definition of ESS standard: A normative document, established by consensus among ESS members and approved by a recognised body according to the procedure of ESS standardisation, that provides for common and repeated use by several actors in the ESS, rules, guidelines or characteristics for the development, production and dissemination of European Statistics, aimed at the achievement of the optimum degree of order in the context of the implementation of the mission and vision of the ESS. Eurostat

ESS Quality reporting: commitment European Statistical Law Regulation (EU) 2015/759 of 29 April 2015 amending Regulation (EC) No 223/2009 on European statistics. The Single Integrated Metadata Structure (SIMS) version 2. 0 is approved as a ESS standard by the ESSC SIMS can be considered the standard for quality reporting according to Article 12 of Regulation 223/2009 on European statistics Eurostat

The Single Integrated Metadata Structure (SIMS) SIMS is a dynamic and unique inventory of ESS standard reference metadata statistical concepts with definitions and reporting guidelines. Eurostat

SIMS 2. 0 Eurostat

ESMS and ESQRS are 2 structures (templates) that can be derived from SIMS • Euro SDMX Metadata Structure (ESMS) recommended for useroriented quality reporting • ESS Standard for Quality Reports Structure (ESQRS) recommended for producer-oriented quality reporting All the concepts used for ESMS/ESQRS/SIMS files are based on SDMX cross-domain concepts Eurostat asks NSIs and ONAs to provide reference metadata and quality reports according to ESMS and ESQRS for European Statistics Eurostat

ESMS 2. 0 Eurostat

ESQRS 2. 0 Eurostat

When ESMS and when ESQRS? • Eurostat is implementing gradually quality reporting domain by domain. • When approaching a new domain, the available documentation is analysed and it is decided which template should be implemented. • In the long term, ESMS should be implemented for all domains and ESQRS for a subset, but often, if there is already some quality documentation available, ESQRS in introduced first. • Only a few domains implement both ESMS and ESQRS. • In the ESS has been decided that by the end of 2018 reference metadata according to the standard should be provided by NSI to Eurostat for all European statistics. 20 Eurostat

NSI implementation of ESMS and ESQRS • In the ESS ESMS and ESQRS implementation is someway “mandatory” to transmit reference metadata to Eurostat • ESMS and ESQRS implementation is also highly recommended at NSI level • However, different NSIs adopted different approaches to reference metadata and quality reporting implementation 21 Eurostat

Examples of quality reporting at NSI level Information on quality and methodology Quality reports Standardised Methodological Report Metadata and Methodology Eurostat

Manuals Further methodological indications are needed to fill-in quality reports… • ESS Handbook for Quality Reports 2014 is the main reference, with its annexes • Technical manual of the SIMS (+ SIMS v. 2. 0) • ESS Guidelines for the Implementation of Quality and Performance Indicators Eurostat

ESS Handbook for quality reports The new version of the Handbook should be available in 2019. Eurostat

Annex I: ESS Quality and Performance Indicators Data completeness – rate* Sampling error – indicators* Over-coverage - rate Common units - proportion Unit non-response - rate* Item non-response - rate* Data revision - average size* Imputation - rate Time lag - first results Time lag - final results* Punctuality - delivery and publication* Asymmetry for mirror flows statistics – coefficient Length of comparable time series* Data tables – consultations Metadata - consultations Metadata completeness - rate Eurostat

Annex II: Technical manual of the SIMS 1. Introduction 2. Use and general characteristics of the SIMS 3. Distinction between the short user- (U) and the detailed producer- (P) oriented quality reports 4. Integration of the Quality and Performance Indicators in the SIMS 5. Implementation of the "once for all purposes" quality reporting strategy Annex 1 Creation of thev. Single Structure from the– ESMS and ESQRS SIMS 2. 0 Integrated – ESMSMetadata 2. 0 and ESQRS 2. 0 2015 Annex 2 ESS guidelines for SIMS Annex 3 New ESMS guidelines as derived from SIMS Annex 4 New ESQRS guidelines as derived from SIMS Annex 5 How to create and submit a report in NRMEHandler ESS Metadata Eurostat

ESS quality reporting: standards and tools Eurostat

What is ESS Metadata Handler (ESS-MH)? • The ESS-MH is an IT application provided by Eurostat allowing the compilation, management and transmission of reference metadata files and quality reports following ESS standards • SDMX compliant: it produces metadatasets and it is able to import in SDMX-ml for the statistical domains included in a category scheme for which one ore metadataflow(s) has been opened referring to specific MSDs Eurostat

Interoperability between national metadata system and ESS-MH Possible scenarios 1. Only ESS-MH is used to produce national metadata files * Nielsen et al. (2015) ESS reference metadata: implementation of standards and interoperability of national and European metadata systems Workshop of the Modernisation Committee on Standards: International Collaboration for Standards. Based Modernisation Meeting, 5 – 7 May 2015 Eurostat

Interoperability between national metadata system and ESS-MH 2. National metadata system is used to produce the metadata file and then it is manually imported in ESS-MH * Nielsen et al. (2015) ESS reference metadata: implementation of standards and interoperability of national and European metadata systems Workshop of the Modernisation Committee on Standards: International Collaboration for Standards. Based Modernisation Meeting, 5 – 7 May 2015 Eurostat

Access to ESS-MH • At national level there are 2 main user profiles that access to ESS-MH: • The Super Provider (ex-National Administrator): • manage the metadata files from all the statistical domains and from all the national Institutions • manage national user profiles • The Provider (ex-National User): • manage the metadata files for specific statistical domain(s) • To access ESS-MH an European Commission Authentication Service (ECAS) account is needed Eurostat

Reference metadata transmission Reference metadata in the ESS are mainly exchanged in SDMX format via the ESS-MH. Thus, what is implemented in the ESS-MH defines the rules for the exchange. E. g. : - At the moment, the version of SDMX used for reference metadata in ESS-MH is 2. 0 - For IT experts: there are 2 types of SDMX messages for metadatasets 1. the generic metadata set 2. the metadata structure specific metadata set • The generic metadata set is required in the ESS-MH Eurostat

Interoperability between national metadata system and ESS-MH • Technical specification for file to be produced by the national metadata systems and imported in the ESS-MH has been produced by Eurostat • E. g. : - only the SDMX-ML file with the metadata set or a zip file including the metadataset, a directory with the annexes, a directory with the images, and the related MSD (optional) can be imported - the name of the metadataset as well as the tag on the header should follow specific rules - the «annotations» tag can be used to transmit information on annexes and on the restriction for publication Eurostat

ESS-MH • Production environment https: //webgate. ec. europa. eu/estat/spe/metaconv/ • Acceptance environment https: //webgate. acceptance. ec. europa. eu/estat/spe/metaconv/ • Webinar on ESS-MH are periodically organised by Eurostat

References Eurostat (2015) Single Integrated Metadata Structure v 2. 0 – ESMS 2. 0 and ESQRS 2. 0 – 2015 http: //ec. europa. eu/eurostat/documents/64157/4373903/SIMS-2 -0 -Revised-standards-November-2015 -ESSCfinal. pdf/47 c 0 b 80 d-0 e 19 -4777 -8 f 9 e-28 f 89 f 82 ce 18 Eurostat (2015) Single Integrated Metadata Structure and its Technical Manual – 2014 http: //ec. europa. eu/eurostat/documents/64157/4373903/03 -Single-Integrated-Metadata-Structure-and-its-Technical -Manual. pdf/6013 a 162 -e 8 e 2 -4 a 8 a-8219 -83 e 3318 cbb 39 Eurostat (2014) ESS Handbook for Quality reports 2014 http: //ec. europa. eu/eurostat/documents/64157/4373903/01 ESS-Handbook-for-Quality-Reports-2014. pdf/d 6152567 -a 007 -4949 -a 169 -251 e 0 ac 7 c 655 Eurostat (2014) ESS Guidelines for the implementation of the ESS quality and performance indicators (QPI) http: //ec. europa. eu/eurostat/documents/64157/4373903/02 -ESS-Quality-and-performance-Indicators 2014. pdf/5 c 996003 -b 770 -4 a 7 c-9 c 2 f-bf 733 e 6 b 1 f 31 ESS Standard for Quality Reports Structure (release 1, October 2010) Euro SDMX Metadata Structure (release 3, March 2009) Eurostat (2009). Eurostat Methodologies and Working papers “ESS Standards for Quality Reports”. http: //epp. eurostat. ec. europa. eu/portal/page/portal/ver-1/quality/documents/ESQR_FINAL. pdf Eurostat (2003 a), Definition of Quality in Statistics. Eurostat Working Group on Assessment of Quality in Statistics, Luxembourg, October 2 -3. Eurostat (2003 b), Glossary. Eurostat Working Group on Assessment of Quality in Statistics, Luxembourg, October 2 -3. Eurostat