7 SDMX A nontechnical overview of the SDMX

  • Slides: 44
Download presentation
7. SDMX: A non-technical overview of the SDMX architecture and IT tools Raynald PALMIERI

7. SDMX: A non-technical overview of the SDMX architecture and IT tools Raynald PALMIERI Eurostat, Unit B. 5 Data and metadata services and standards European Statistical Training Programme (ESTP) SDMX Basics course, 6 -8 March 2018, Luxembourg Eurostat 1

WHERE TO FIND THE TOOLS 2

WHERE TO FIND THE TOOLS 2

http: //ec. europa. eu/eurostat/web/sdmx-infospace/sdmx-it-tools 3

http: //ec. europa. eu/eurostat/web/sdmx-infospace/sdmx-it-tools 3

Type of tools • Eurostat SDMX tools for designers • Shared tools • SDMX

Type of tools • Eurostat SDMX tools for designers • Shared tools • SDMX tools for data providers 4

EUROSTAT SDMX TOOLS FOR DESIGNERS 5 Eurostat

EUROSTAT SDMX TOOLS FOR DESIGNERS 5 Eurostat

Eurostat SDMX tools for designers 6 Tools offered Use Data Structure Wizard Used to

Eurostat SDMX tools for designers 6 Tools offered Use Data Structure Wizard Used to create, edit and test SDMX objects Maintenance of SDMX registry Euro SDMX Registry A central repository for storing and sharing SDMX objects Eurostat

Eurostat SDMX tools for designers Data Structure Wizard Web service Create/Edit 7 Eurostat Euro

Eurostat SDMX tools for designers Data Structure Wizard Web service Create/Edit 7 Eurostat Euro SDMX Registry Store/Edit

Data Structure Wizard • Creation / edition of DSDs • Validation of DSD •

Data Structure Wizard • Creation / edition of DSDs • Validation of DSD • Local repository • Artefacts are stored in a local repository • Remote repository • Can be used for retrieving existing artefacts 8

DATA STRUCTURE DEFINITION - Design Data Structure Wizard • • Java desktop application Graphical

DATA STRUCTURE DEFINITION - Design Data Structure Wizard • • Java desktop application Graphical Interface For DSD designers Maintenance of SDMX v 2. 0/2. 1 data and meta data structures • Web service to query/submit SDMX registries

DSW: Management of SDMX artefacts 10

DSW: Management of SDMX artefacts 10

DSW: Artefact example: the codelists 11

DSW: Artefact example: the codelists 11

SHARED TOOLS FOR DESIGNERS AND DATA PROVIDERS 12

SHARED TOOLS FOR DESIGNERS AND DATA PROVIDERS 12

Work roles and process SDMX Registry Interpret SDMX messages Consult and get DSD COLLECTING

Work roles and process SDMX Registry Interpret SDMX messages Consult and get DSD COLLECTING ORGANISATION Publish DSD Consult and get DSD Create DSD 13 Send SDMX messages Format / structure SDMX messages

SDMX Registry • Metadata registry which implements the SDMX registry specifications • Tasked with

SDMX Registry • Metadata registry which implements the SDMX registry specifications • Tasked with providing structure, organisation, maintenance and query interfaces for the SDMX components required to support the data sharing vision • Provides a web-based user interface as well as web services for interacting with the SDMX structural metadata objects in use within the collecting organisation and with statistical partners. 14 Eurostat

SDMX Registry • Only users concerned with providing information need to access the data

SDMX Registry • Only users concerned with providing information need to access the data and reference metadata sets. • An application which wants a particular data or metadata set would then query the registry for the structure, and then go and retrieve the data or metadata set directly from the provider's web server. • Example of existing SDMX registries: • Euro SDMX Registry (2. 1) • SDMX Global Registry (2. 1) 15 Eurostat

New Euro SDMX Registry (2. 1) § Eurostat's implementation of the SDMX Registry §

New Euro SDMX Registry (2. 1) § Eurostat's implementation of the SDMX Registry § Based on SDMX 2. 1 (as published on April 2011) Also capable of importing and exporting SDMX 2. 0 artefacts. § Allows browsing, searching, editing and subscribing to artefacts. § Advanced access control mechanism of artefacts controlling their visibility (Public/Private). Eurostat

New Euro SDMX Registry (2. 1) Home page Access to the content of the

New Euro SDMX Registry (2. 1) Home page Access to the content of the Registry via text search Access to the content of the Registry by type Most recent items Eurostat

New Euro SDMX Registry (2. 1) Registry Content Filtering list Number of objects of

New Euro SDMX Registry (2. 1) Registry Content Filtering list Number of objects of type selected Eurostat

SDMX Global Registry (SGR) • Launched in 2011 by the SDMX Sponsoring institutions •

SDMX Global Registry (SGR) • Launched in 2011 by the SDMX Sponsoring institutions • Originally to support the world-wide technical implementation of Balance of Payments and National Account implementations • To make metadata material (DSDs, concept schemes, MSDs, code lists, etc. ) publicly and centrally available used and maintained by national and international organisations.

Who should use the SDMX Global Registry (SGR)? • Any national or international agency

Who should use the SDMX Global Registry (SGR)? • Any national or international agency that wants to use SDMX standards for data management or for exchanging data with other organisations using SDMX standard. • Any end-user of statistical data can access the data structures posted on the SGR.

How to use the SDMX Global Registry (SGR)? • All information published in the

How to use the SDMX Global Registry (SGR)? • All information published in the SGR is publicly available and freely viewable and downloadable. • No need to log in to view or to download information - SDMX Global Registry Content. • Users can view the full content of the Global Registry

Exploring by agency Available SDMX objects Click on a type Click on Export to

Exploring by agency Available SDMX objects Click on a type Click on Export to choose a SDMX object to download Enter a text to be found in the Registry Access to the web service

Codelist Content 2. Select an object

Codelist Content 2. Select an object

1. Choose a DSD in the list Dimensions, measure and attributes Corresponding list of

1. Choose a DSD in the list Dimensions, measure and attributes Corresponding list of codes

Example of result for "NA-MAIN" 5 versions found

Example of result for "NA-MAIN" 5 versions found

Web Service Query Select the SDMX version, the Object type and the identifiers of

Web Service Query Select the SDMX version, the Object type and the identifiers of the SDMX object (ID, agency, version) Select which referenced objects (e. g. Code lists) you want to include View the xml file in the browser or Download the file The URL generated is the SDMX query to be sent to the web service to get the xml file corresponding to the selected object

View the xml file in your browser (DSD NA_MAIN)

View the xml file in your browser (DSD NA_MAIN)

SDMX TOOLS FOR DATA PROVIDERS 28 Eurostat

SDMX TOOLS FOR DATA PROVIDERS 28 Eurostat

SDMX Tools for Data Providers Tools offered Use Action Web Forms (EU) Excel-like templates

SDMX Tools for Data Providers Tools offered Use Action Web Forms (EU) Excel-like templates Transmission of low volumes of data No costs for EU organisations Manual work for senders (type, copy/paste) SDMX Converter (sender’s PC) Converts data files between SDMX formats and other file formats Installation on sender’s PC Manual work for senders (convert) SDMX Converter (batch mode) SDMX Reference Infrastructure Installation on server A set of tools that Installation on server allows to connect your Mapping of database to IT systems to the SDMX DSDs world Own SDMX Implementation 29 Local development in organisations Eurostat

Data provider view: One goal – different possibilities Webforms EDAMIS Single Entry Point SDMX-ML

Data provider view: One goal – different possibilities Webforms EDAMIS Single Entry Point SDMX-ML file Excel sheets SDMX Converter SDMX-ML file Database export SDMX Converter SDMX-ML file Database SDMX Reference Infrastructure Eurostat SDMX-ML file SDMX Web Service Same DSD Same Format

Data provider view: One goal – different IT Architectures Push mode Webforms Web-based Excel

Data provider view: One goal – different IT Architectures Push mode Webforms Web-based Excel sheets SDMX Converter Push mode Local conversion Database export Local conversion Database SDMX Reference Infrastructure SDMX Converter Push mode Push or Pull mode Eurostat

SDMX Converter http: //ec. europa. eu/eurostat/web/sdmx-infospace/sdmx-it-tools • File based conversion • Open source and

SDMX Converter http: //ec. europa. eu/eurostat/web/sdmx-infospace/sdmx-it-tools • File based conversion • Open source and platform independent (Java) • Different ways of using it: • • Graphical user interface Batch file (server or client side) Web Service interface Reusing source code in your own Java application • Formats: SDMX-ML 2. 0 & 2. 1, SDMX-EDI (Gesmes), FLR, CSV, Google DSPL, predefined Excel templates Eurostat

GUI (Converter 5. 4. 4) 1. Selection of the input/output files and their format

GUI (Converter 5. 4. 4) 1. Selection of the input/output files and their format 2. a Select the DSD in the local drive 2. b Identify a DSD to download from the SDMX Registry (configuration required) 2. If the local DSD includes multiple versions, we can specify the one desired derired 2. c Identify a dataflow linked to the DSD to download from the SDMX Registry (configuration required) 3. Excel parameter file 3. SDMX header (. prop file) CSV parameters Only for flat and excel files 4. Mapping and Transcoding 6. SDMX (output) validation 5. CSV quotation XML parameters for SDMX output formats Eurostat

New Converter Interface Eurostat (6. 11. 4)

New Converter Interface Eurostat (6. 11. 4)

Web Interface http: //wlts 0018. cc. cec. eu. int: 2061/converter-webapp/convert (Internal Eurostat URL) Eurostat

Web Interface http: //wlts 0018. cc. cec. eu. int: 2061/converter-webapp/convert (Internal Eurostat URL) Eurostat

36 Eurostat

36 Eurostat

37 Eurostat

37 Eurostat

Exercice: Converter • Installation • From CSV to SDMX Compact 38 Eurostat

Exercice: Converter • Installation • From CSV to SDMX Compact 38 Eurostat

What is SDMX-Reference Infrastructure • Framework for SDMX data provision • Set of pick-and-choose

What is SDMX-Reference Infrastructure • Framework for SDMX data provision • Set of pick-and-choose reusable building blocks allowing a statistical office to disseminate SDMX data to the external world • Designed to provide data and structural metadata based on mappings to each organization's dissemination data warehouse • Uses SDMX standards including the one for Web Services 39 Eurostat

Data provider view: SDMX Reference Infrastructure Database SDMX Reference Infrastructure Push or Pull mode

Data provider view: SDMX Reference Infrastructure Database SDMX Reference Infrastructure Push or Pull mode Eurostat

Tools Relationships – Example Store Produce DSW SDMX Registry DSD Produce use SDMX Converter

Tools Relationships – Example Store Produce DSW SDMX Registry DSD Produce use SDMX Converter SDMX-RI SDMX-ML Dataset Mapping Assistant Produce CSV File 41 use Eurostat Mapping National DB

SDMX Tools – Summary • Eurostat tools Eurostat SDMX Info Space • http: //ec.

SDMX Tools – Summary • Eurostat tools Eurostat SDMX Info Space • http: //ec. europa. eu/eurostat/web/sdmx-infospace/sdmx-it-tools • SDMX Converter • SDMX Reference Infrastructure & Mapping Assistant • SDMX Registry & Data Structure Wizard Eurostat

Support for data providers • SDMX Reference Infrastructure ü Installed in ~32 countries for

Support for data providers • SDMX Reference Infrastructure ü Installed in ~32 countries for Census Hub ü Can be extended to other domains • 1 st contact point - Local Transmission Coordinators ü In National Statistical Institutes ü Experience with EDAMIS and SDMX • 2 nd contact point - Eurostat support teams Eurostat

Thank you for your attention! Questions Eurostat

Thank you for your attention! Questions Eurostat