SDMX Reference Infrastructure 27 October 2015 SDMX Tools

SDMX Reference Infrastructure 27 October, 2015 SDMX Tools Task Force Eurostat 1

Objectives of this presentation SDMX-RI Usage & Functionality (Do you need more information? ) What is new (Do you know what is and how it works? ) Current and upcoming releases Enhancement and functionalities Show some of the future changes Participants to comment on: Usage of the SDMX-RI future 27 October, 2015 SDMX Tools Task Force Eurostat 2

Functionality Set of building blocks Allowing an organization to expose data to third parties (via Web Service) Supporting mapping of dissemination databases to given structural metadata (via Mapping Assistant) Testing mappings and web services and exporting data in SDMX format (via Test Client) Browsing statistical data (via Web Client) Supports SDMX v 2. 0 (and v 2. 1) WS guidelines Java and. NET Oracle, SQL Server, My. SQL 27 October, 2015 SDMX Tools Task Force Eurostat 3

SDMX-RI architecture overview 27 October, 2015 SDMX Tools Task Force Eurostat 4

SDMX – RI - Mapping Assistant Stores the SDMX structures agreed for the data exchange process Allows users to define subsets of data to be disseminated Creates and stores mappings between the internal data structure and SDMX concepts (e. g. My_column_A = AGE) Creates and stores mappings between the internal classifications and SDMX codelists (e. g. My_code_AB = Y_LT 15) 27 October, 2015 Result: Control the exposed data Preview the data in SDMX format Identify errors SDMX Tools Task Force Eurostat 5

SDMX – RI - Test Client Allows users to view and extract data in SDMX format, using the mappings defined in the Mapping Assistant tool extract data directly from the dissemination database extract data using a web address (web service) Result: Allows to test the dissemination process and SDMX compliance Create custom extraction Identify errors Extract data in different formats 27 October, 2015 SDMX Tools Task Force Eurostat 6

SDMX – RI - Web Client Allows users to view and extract data in SDMX format, using the mappings defined in the Mapping Assistant tool Provides user friendly interface for even not experienced users Can extract data using a web address (web service) Result: Allows to test the data dissemination process and SDMX compliance Create custom extraction Identify errors Extract data in different formats 27 October, 2015 SDMX Tools Task Force Eurostat 7

SDMX – RI – NSI/Web service No graphical user interface Invisible for the user modules controlling the incoming data requests Retrieving SDMX structure and mappings Retrieving data from the dissemination database Generating data response messages Sending data in SDMX format 27 October, 2015 Result: Data are made available to different data consumers via internet SDMX Tools Task Force Eurostat 8

SDMX Reference infrastructure - Evolution SDMX 2. 0 National accounts SDMX 2. 0 Agriculture Component based architecture Creation for the CENSUS 2011 Reused in other statistical domains by some countries Installations in ESS NSIs for CENSUS 2011 SDMX 2. 1 Architecture based on common API New version of the SDMX RI Integrated by other organisations 2015 9 27 October, 2015 Eurostat

Hub scenario SDMX Data Provider = NSI (STANDARD, EXCHANGE, METADATA, REPOSITORY) DSD Mapping Assistant Web Client Test Client Metadata repository WEB SERVICES Mapping store Data Collector DSD SDMX query SDMX response Non-SDMX local database 27 October, 2015 Census Hub 10 Eurostat

SDMX-RI - identified advantages Facilitate the SDMX compliance phase Supports all modes of data transmission Limited/no change of the existing environment, production or dissemination systems Limited/no need for adaptation Open source Reusable modules/installations through well-defined APIs 27 October, 2015 SDMX Tools Task Force Eurostat 11

SDMX-RI - identified advantages (cont. ) Facilitate the SDMX compliance phase Reusable modules/installations through well-defined APIs High performance Minimal hardware and software requirements SDMX-RI in the Hub concept (pull mode) No data transmission No replication of the data In case of update, the new data are available immediately 27 October, 2015 SDMX Tools Task Force Eurostat 12

Prerequisites for SDMX-RI Technical Statistical Supported databases – Oracle DB, SQL Server, My. SQL*. SDMX compliance Required development framework: Java and. NET Perform mapping activities Deployment requirements – Tomcat, Web. Logic, IIS Update of SDMX structures and mappings (if necessary) (Minimal) hardware and software requirements Aggregated data are produced NSIs should maintain the system 27 October, 2015 SDMX Tools Task Force Eurostat 13

What is new? What is new in the last public release What is not released yet? Ongoing activities Show some of the future changes 27 October, 2015 SDMX Tools Task Force Eurostat 14

Current and upcoming releases Public release Dec 2014 (Feb 2015 Java) Sdmx. Source. NET v 0. 5. 1 as part of the NSI Web Service package Sdmx. Source JAVA v 1. 1. 4 b as part of the NSI Web Service package CAPI IData. Reader. Engine implementation Extension of the Mapping Store (included other artefacts) Mapping Assistant supports Db, Query and Initialization plug-ins Non-public release March 2015 Support of: Agency (Scheme), Data Provider/Consumer (Scheme), Organization Unit (Scheme), Structure Set (Codelist and Structure Maps only) Content Constraint Coded TIME dimension 27 October, 2015 SDMX Tools Task Force Eurostat 15

Current and upcoming releases Non-public release June 2015 and October 2015 Annotation support in SDMX-RI Attachment measure support (SDMX v 2. 0) Dataflow logging for the Wsand NSI WS needs read-only user TNS support in the MA, better error feedback and allow “substring” and other methods in Custom Query Category tree in Test Client as a re-usable component and TC shows SDMX names in available languages TAC shows dataflow names in available languages Planned public release Dec 2015 Transcoding of non-mandatory attributes without loss of records External usage flag for NSI WS 27 October, 2015 SDMX Tools Task Force Eurostat 16

Ongoing activities Work concentrated only on the ultimate solution Testing of latest development release Analysis of requirements Continuous enhancements Improvements (GUI, performance) Integration of 3 rd party software Bug fixes 27 October, 2015 SDMX Tools Task Force Eurostat 17

Planned future enhancements Better Data Query support Limit and error reporting Handle SDMX XML queries better Handle Complex Queries not implemented in the Sdmx. Source More SDMX arterfacts supported by the SDMX-RI and included in the Mapping Store database User support in NSI Client Save queries on server side (on hold) 27 October, 2015 SDMX Tools Task Force Eurostat 18

SDMX Reference infrastructure - Future Incorporate new file formats JSON New use cases Collaboration in the development and maintenance of the infrastructure with other organisations 19 27 October, 2015 Eurostat

SDMX Reference infrastructure future – New use cases SDMX-RI at the core of the new dissemination chain of Eurostat SDMX-RI to be used for the exchange of confidential data Extension of the pull method for data-sharing initiatives 20 27 October, 2015 Eurostat

SDMX Reference infrastructure future – Enhance collaboration SDMX-RI developed by Eurostat but used and extended by other organisations (ISTAT, INEGI, OECD) Eurostat and interested organisations working together to share development activities Collaboration platform being set-up by Eurostat (Source code repository, wiki, issue tracking tool …) 21 27 October, 2015 Eurostat

Usage Mainly developed to support data exchange via web services in SDMX-ML format (Pull mode) Employed for the European Census Hub Included in ESTAT IT strategy Growing number of use cases Eurostat's Dissemination Web Service SDMX data file creation by certain statistical offices (push mode) Adoption by Member States and international organisations (pull and push mode) Integrated (and adapted) in national SDMX related systems 27 October, 2015 SDMX Tools Task Force Eurostat 22

SDMX Infrastructure is …. SDMX infrastructure is mature and it is being used in different production processes (Census HUB, National accounts …) New uses cases are developing, further developments are expected Collaboration between different organisations is key to enhance the infrastructure and extend its usage 27 October, 2015 SDMX Tools Task Force Eurostat 23

Thank you! Questions? Discussion (Part 1) – SDMX-RI in general How the SDMX RI can help you to improve the data collection and dissemination? Do you see additional use cases for the SDMX RI? 27 October, 2015 SDMX Tools Task Force Eurostat 24

Current and upcoming releases SDMX-RI. NET 30 June 2015 (with improvements to be released Q 4 2015) (Mapping Store v 4. 0) Mapping Assistant v 5. 5. 0 Test Client v 4. 5. 0 NSI Web Service v 4. 4. 0 NSI Web Client v 3. 5. 0 Test. Auth. Config v 3. 4. 0 Sdmx. Source. NET v 0. 7. 0 31 March 2015 (non-public release) (Mapping Store v 3. 3) Mapping Assistant v 5. 4. 0 Test Client v 4. 4. 1 NSI Web Service v 4. 3. 0 NSI Web Client v 3. 4. 1 Test. Auth. Config v 3. 3. 0 Sdmx. Source. NET v 0. 6. 0 (Mapping Store v 3. 3) Mapping Assistant v 5. 3. 1 Test Client v 4. 3. 1 NSI Web Service v 4. 2. 1 NSI Web Client v 3. 3. 1 Test. Auth. Config v 3. 2. 1 Sdmx. Source. NET v 0. 5. 1 21 November 2014 (public release) 27 October, 2015 SDMX Tools Task Force Eurostat Annotation support in SDMX-RI Attachment measure support (SDMX v 2. 0) Dataflow logging for the WS TNS support in the MA Internationalisation in the TC Agency (Scheme) Data Provider/Consumer (Scheme) Organization Unit (Scheme) Structure Set (Codelist and Structure Maps only) Content Constraint Coded TIME dimension Sdmx. Source. NET v 0. 5. 1 as part of the NSI Web Service package CAPI IData. Reader. Engine implementation Extension of the Mapping Store (included other artefacts) 25

Current and upcoming releases SDMX-RI JAVA 2 January 2015 (public release) (Mapping Store v 3. 3) NSI Web Service v 5. 2. 3 NSI Web Client v 2. 3. 1 Sdmx. Source. JAVA v 1. 1. 4 b Coded TIME dimension 2 March 2015 (non-public release) (Mapping Store v 3. 3) NSI Web Service v 5. 2. 4 NSI Web Client v 2. 3. 2 Sdmx. Source. JAVA v 1. 1. 4 b (Mapping Store v 4. 0) NSI Web Service v 5. 3. 0 NSI Web Client v 2. 3. 3 Sdmx. Source. JAVA v 1. 1. 4 b 8 June 2015 (with improvements to be released Q 4 2015) 27 October, 2015 SDMX Tools Task Force Eurostat Annotation support in SDMX-RI Attachment measure support (SDMX v 2. 0) 26