SDMX Tools Overview and architecture ESTP Workshop on

SDMX Tools Overview and architecture ESTP Workshop on SDMX in National Accounts Luxembourg, 16 -18 March 2016 Eurostat

Technical Integration Source National data sources SDMX(Mapping) Implementation Challenge National databases National statistical organisations SDMX Data Structure Defintions & Data Flows International organisations Metadata driven process based on SDMX standards and tools Full automation of the data exchange possible

THE SDMX COMPONENTS SDMX Informati on Model Content Oriented Guideline s IT Infrastructure for exchange and sharing 3 3 Eurostat

THE SDMX COMPONENTS SDMX Informati on Model Content Oriented Guideline s IT Infrastructure for exchange and sharing 4 4 Eurostat

SDMX Tools for National Accounts – decision tree Do you have a database for ESA? No Do you plan to create one? Yes Good start 1) RI + Mapping Yes RI+Loader 2) CSV export with Converter No Really not? SDMX Converter 3) Own development Eurostat 5

Getting there: One goal – different possibilities Phase out in Excel sheets National Accounts SDMX Converter SDMX-ML file Database & Export SDMX Converter SDMX-ML file Database & Mapping SDMX Reference Infrastructure Database & Loader SDMX Reference Infrastructure Eurostat SDMX-ML file SDMX Web Service Same DSD Same Format

SDMX Converter https: //webgate. ec. europa. eu/fpfis/mwikis/sdmx/index. php/SDMX_Converter • File based conversion • Open source and platform independend (Java) • Different ways of using it: (will be demonstrated) • Graphical user interface • Batch file (client side or server side) • Web Service interface • Reusing source code in your own Java application • Formats: SDMX-ML 2. 0/2. 1, SDMX-EDI (Gesmes), FLR, CSV, Google DSPL XLS, XLSX (read only with additional preparation) Eurostat

SDMX Dataset DSD convert Eurostat

Demonstration • SDMX Converter & ESA Excel template • Open the SDMX-NA maintenance page on SDMX. org (https: //sdmx. org/? page_id=1498) • Download ESA table 0101 template and the correct DSD • Convert it to SDMX-ML with the SDMX Converter 9 Eurostat

Getting there: One goal – different possibilities Phase out in Excel sheets National Accounts SDMX Converter SDMX-ML file Database & Export SDMX Converter SDMX-ML file Database & Mapping SDMX Reference Infrastructure Database & Loader SDMX Reference Infrastructure Eurostat SDMX-ML file SDMX Web Service Same DSD Same Format

SDMX Reference Infrastructure https: //webgate. ec. europa. eu/fpfis/mwikis/sdmx/index. php/SDMX_Reference_Infrastr ucture_SDMX-RI Data Provider Data Collector SDMX Registry Mapping Assistant DSD Web Svc Non-SDMX local database Test Client SDMX-RI Eurostat Web Client SDMX data set

Getting there: One goal – different possibilities Phase out in Excel sheets National Accounts SDMX Converter SDMX-ML file Database & Export SDMX Converter SDMX-ML file Database & Mapping SDMX Reference Infrastructure Database & Loader SDMX Reference Infrastructure Eurostat SDMX-ML file SDMX Web Service Same DSD Same Format

National Source (e. g. CSV) ISTAT Loader & SDMX-RI http: //sdmx 2. istat. it/sodidownload/Download. aspx Data Provider ISTAT Loader Data Collector SDMX Registry Mapping Assistant DSD Web Svc SDMX compliant local database Test Client SDMX-RI Eurostat Web Client SDMX data set

Functional characteristics • Allows the creation of a dissemination databased on the SDMX-IM • It is integrated with the SDMX-RI. • Loads data files having CSV, FLR, GESMES-TS, SDMX formats • Allows the user managing at three different levels: Administrator, DSDs and dataflows user. • Uses three DBSM types (Oracle, SQL Server, My. Sql)

Architecture SDMX, flr, gesmes. TS, csv Loader database Mapping assistant Structure files Mapping Store database Structural tables Data tables Mapping set tables Web Service Loader

User manager Publish dataflows Import Data files Definition of the mapping Creation of Data Tables Import dataflows from MS Flow of the application

SDMX Tools for National Accounts – decision tree Do you have a database for ESA? No Do you plan to create one? Yes Good start 1) RI + Mapping Yes RI+Loader 2) CSV export with Converter No Really not? SDMX Converter 3) Own development Eurostat 17