SDMX IT Tools SDMX use in practice in

SDMX IT Tools SDMX use in practice in NA National Account use case Jean-Francois LEBLANC Christian SEBASTIAN 11 -13 May 2016 Eurostat, Unit B 3 – IT solutions for statistical production Eurostat

Table of contents 1. Objectives 2. National Accounts (NA) 1. 2. 3. 4. 5. NA NA NA business needs Domains Technical Integration Data Validation Process Project Cycle 2 Eurostat

Table of contents 2. SDMX Tools in practice 1. 2. 3. 4. 5. Data Structure Wizard SDMX Converter SDMX Reference Infrastructure Tools relationship SDMX Tools – Summary 3. SDMX Tools for Data Providers 1. 2. 3. 4. Possibilities for Data Providers IT Architectures for Data Providers How to use the tools Conclusion 3 Eurostat

1 Objectives • To reflect how National Accounts work • To present the SMDX IT tools • To expose how to work with the data and SDMX in real situations • To give a base understanding of the tools and their relationship that will be of help for further explanations 4 Eurostat

2 National Accounts (NA) National Accounts BOP FDI Excel questionnaires GESMES files GFS Main Aggregates Sector Accounts Supply & Use Input / Output 5 Eurostat

2 Why SDMX for NA? Within Eurostat Internationa lly Within the ESS 6 Eurostat

2. 1 NA business needs Within Eurostat Internationa lly • Main trigger: new methodology • SNA 93 SNA 2008 • ESA 95 ESA 2010 • Business process reengineering • Corporate production systems • Integration of production chains Within the ESS • Harmonised coding 7 Eurostat

2. 1 NA business needs Cross-domain integration • Balance of Payments • Foreign Direct Investment Within Eurostat Internationa lly Within the ESS Business process reengineering • Standardization • Improved validation • Reuse of tools 8 Eurostat

2. 1 NA business needs Data quality • 1 country 1 year 1 GDP • Shared validation rules • Across all organisations Within Eurostat Lowering reporting burden • 1 data flow from national to international International cooperation Internationa lly Within the ESS • Inter-Agency Group on Economic and Financial Statistics = SDMX Sponsors • Example: PGI Website http: //www. principalglobalindicators. org 9 Eurostat

2. 2 NA Domains National Accounts Shared Concepts and Code Lists Cross Domain Concepts and Code Lists SNA 2008 ESA 210 Other related domains Balance of Payment FDI statistics Statistics Other domains 10 Eurostat

2. 3 NA Technical Integration Sourc e Nationa l DBs Mappin g Sourc e National data sources National statistical organizations SDMX DSDs & Data Flows International organizations Metadata driven process based on SDMX standards and tools Full automation of the data exchange possible 11 Eurostat

2. 4 NA Data Validation Process Before/during transmission “First Level” After Transmission “Second Level” Currently covered by SDMX • Format Check (SDMX-ML) • Code Check (SDMX DSD) • Code used correctly (SDMX Dataflow & Constraint) Not covered by SDMX • Detailed value check • Validation expressions • … 12 Eurostat

2. 5 NA Project Cycle 13 Eurostat

3. SDMX Tools in practice Registry • Find DSDs • Edit/Create DSDs DSW • Data files -> SDMX files Converter • Database -> SDMX WS/files SDMX-RI 14 Eurostat

3. 1 Data Structure Wizard • Java tool (open source and platform independent) • Supports • Maintenance of SDMX v 2. 0 and v 2. 1 structures • Import and export of artefacts from/to SDMX-ML v 2. 0, v 2. 1 and GESMES format • Interaction with Euro SDMX Registry (retrieval, modification and search of artefacts) • Sample dataset creation 15 Eurostat

3. 2 SDMX Converter • Java tool (open source and platform independent) • File based conversion • Different ways of using it: • • Graphical user interface Command Line interface Web Service interface Common API interface • Formats*: SDMX-ML 2. 0, SDMX-ML 2. 1, SDMX-EDI (Gesmes), FLR, CSV, Google DSPL, predefined Excel templates *Exceptions may apply 16 Eurostat

Mapping Store DSD Local environment Is my dataflow Is my WS working? Test Client (Desktop Software) NSI Client Mapping Web Service Dissemin ation DB Data Consumer (external) 3. 3 SDMX Reference Infrastructure 17 Eurostat

3. 4 Tools relationship produces uses stores DSD SDMX Registry DSW MSD produces DSD uses SDMX Converter produces CSV or other datafile produces SDMX Dataset ESS-MH MSD uses SDMX-RI exposes uses Mapping Assistant SDMX Meta. Dataset produces Mapping Census Hub National DB Eurostat 18

3. 4. 1 Tools relationship – Example Eurostat produces NA DSDs DSW Eurostat stores NA DSDs SDMX Registry DSD SDMX-RI Mapping Assistant SDMX-ML Dataset NSI uses NA DSDs NSI maps NA DSDs Mapping National DB NSI produces SDMX Converter Eurostat NSI uses NA DSDs NSI exports NA data (CSV, XLS) 19

3. 5 SDMX Tools – Summary • Eurostat tools Eurostat SDMX Info Space • SDMX Converter • SDMX Reference Infrastructure & Mapping Assistant • SDMX Registry & Data Structure Wizard 20 Eurostat

4. SDMX Tools for Data Providers Tools offered Use Action Web Forms (EU) Excel-like templates Transmission of low volumes of data No costs for EU organisations Manual work for senders (type, copy/paste) SDMX Converter Converts data files from different file formats to SDMX format Local installation on sender’s PC Installation on server SDMX Reference Infrastructure A set of tools that allows Installation on server to transform directly your Mapping of database to DSDs data from a database to SDMX Common API Supports the development of local SDMX systems Local development in organisations 21 Eurostat

4. 1 Possibilities for Data Providers Webforms EDAMIS Single Entry Point SDMX-ML file Excel sheets SDMX Converter SDMX-ML file Database export SDMX Converter SDMX-ML file Database SDMX Reference Infrastructure Eurostat Same DSD Same Format SDMX-ML file SDMX Web Service 22

4. 1. 1 Possibilities for Data Providers. NA. Not applicable in National Webforms Accounts Will be phased Excel out in National sheets Accounts Database export SDMX Converter SDMX-ML file Database SDMX Reference Infrastructure EDAMIS Single Entry Point Eurostat SDMX-ML file Same DSD Same Format SDMX-ML file SDMX Web Service 23

4. 2 IT Architectures for Data Providers Push mode (EDAMIS) Webforms Web-based Excel sheets SDMX Converter Local conversion Push mode (EDAMIS) Database export Local conversion Push mode (EDAMIS) Database SDMX Reference Infrastructure Pull mode SDMX Converter 24 Eurostat

4. 3 How to use the tools If you are a Data Collector Design your DSD using the DSW Make the DSD available in the SDMX Registry Inform your data providers for the available DSD Eurostat 25

4. 3 How to use the tools If you are a Data Provider Download/pull the DSD from the SDMX Registry If you have a DDB Data stores as files Install and use the SDMX-RI Install and use the Converter Expose the data to be pulled (WS, HUB) Push via EDAMIS Eurostat 26

4. 4 Conclusion SDMX statistical and technical standards & SDMX IT tools and IT infrastructure provide the "toolkit" for ü the harmonisation and integration of data structures ü the rationalisation of national/international data flows ü finally improving the data quality of official statistics. 27 Eurostat

SDMX in practice in NA 28 Eurostat