SDMX RI Progress report June 2019 SDMX Workshop

SDMX RI Progress report June, 2019 SDMX Workshop, Rome Eurostat

Overview � Latest developments � Why SDMX-RI � SDMX – RI WS and Mapping Assistant � SDMX-RI new architecture � Mapping Assistant API and WS � Mapping Assistant as web application � Status and future development � Discussion June, 2019 SDMX Workshop, Rome Eurostat

Objectives of this presentation � Eurostat to present the: � Overview of the SDMX-RI architecture � Work done during 2018 -2019 � Status and future developments � Participants: � Acknowledge the progress and status of the SDMX-RI. � Provide their comments on the proposed improvements June, 2019 SDMX Workshop, Rome Eurostat

Eurostat SDMX tools Where is SDMX? SDMX tools Metadata design and build DSW; SDMX Registry Data compliance SDMX Converter; SDMX-RI Collection EDAMIS; SDMX-RI; Census/Data Hub; ESS-MH Validation STRUVAL; EDIT Data loading and storage DLBB Disseminate DSWS; SDMX-RI; Census Hub More information here June, 2019 SDMX Workshop, Rome Eurostat Input Used in

Eurostat SDMX tools � OSS, SDMX tools DSW, SDMX Registry SDMX Converter; SDMXRI SDMX-RI, Census/Data Hub, ESS-MH STRUVAL, EDIT DLBB DSWS, SDMX-RI, Census Hub June, 2019 supported by Eurostat � For data collectors and data providers � Cover different use case � Based on SDMX Standard: Registry, WS, Reference Architecture, IM � Re-usable and shared building blocks � Java mainly (SDMX-RI also in. NET) � GUI, API and/or WS � Based on Common API (CAPI) � Extensible SDMX Workshop, Rome Eurostat

What is SDMX-RI � Framework for SDMX data provision � Set of pick-and-choose reusable building blocks allowing a statistical office to disseminate SDMX data to the external world � Designed to provide data and structural metadata based on mappings to each organization's dissemination data warehouse � Uses SDMX standards including the one for Web Services June, 2019 SDMX Workshop, Rome Eurostat

Why SDMX-RI � Developed to simplify the exchange of data � Provides standard software and components, allowing individual statistical organizations to interact and exchange their data using the same software and methodology � Modular approach, users can �use part or the entire infrastructure, �extend it by adding new modules or modify it in any other way to suit their own purposes � Developed June, 2019 in both Java and. NET SDMX Workshop, Rome Eurostat

SDMX-RI components � Mapping Assistant �Graphical tool to create the mapping between the DSD and the dissemination database. � Test Client �Used � NSI to test your dataflow locally Web Service �Allows � NSI Client �Web June, 2019 you to share your dataflow interface to interact with the Web Service SDMX Workshop, Rome Eurostat

SDMX-RI vs SDMX Converter Eurostat tools NSI developed software Process workflow SDMX codes Extract files Transform file SDMX file Dissemina tion/Trans mission SDMX Converter NSI software SDMX Converter Processing for sending EDAMIS SDMX-RI Non-SDMX local data Test client Web client Mapping Assistant NSI development NSI software EDAMIS Processing for sending NSI Web service HUB EDAMIS June, 2019 SDMX Workshop, Rome Back

SDMX – RI - Mapping Assistant Stores the SDMX structures agreed for the data exchange process � Allows users to define subsets of data to be disseminated � Creates and stores mappings between the internal data structure and SDMX concepts � (e. g. My_column_A = AGE) � Creates and stores mappings between the internal classifications and SDMX codelists � (e. g. My_code_AB = Y_LT 15) � June, 2019 Result: � Control the exposed data � Preview the data in SDMX format � Identify errors SDMX Workshop, Rome Eurostat

SDMX – RI - Test Client Allows users to view and extract data in SDMX format, using the mappings defined in the Mapping Assistant tool � extract data directly from the dissemination database � extract data using a web address (web service) � Result: � Allows to test the dissemination process and SDMX compliance � Create custom extraction � Identify errors � Extract data in different formats June, 2019 SDMX Workshop, Rome Eurostat

SDMX – RI - Web Client Allows users to view and extract data in SDMX format, using the mappings defined in the Mapping Assistant tool � Provides user friendly interface for even not experienced users � Can extract data using a web address (web service) � Result: � � June, 2019 Allows to test the data dissemination process and SDMX compliance Create custom extraction Identify errors Extract data in different formats SDMX Workshop, Rome Eurostat

SDMX – RI – NSI/Web service No graphical user interface � Invisible for the user modules controlling the incoming data requests � Retrieving SDMX structure and mappings � Retrieving data from the dissemination database � Generating data response messages � Sending data in SDMX format � Result: � Data are made available to different data consumers via internet June, 2019 SDMX Workshop, Rome Eurostat

Architecture - modular, compliant, reusable � Common components for SDMX software (CAPI) � Derived from the SDMX IM � Version agnostic (the model tends to favor SDMX v 2. 1) � Extensible Common/shared implementation (Sdmx. Source) � Developed in Java (Metadata Technology) � Ported to. NET (Eurostat) � Additional APIs and implementations (Sdmx. Source_ext) � � Fully fletched environment for data reporting and dissemination June, 2019 SDMX Workshop, Rome Eurostat

Governance � Developed � OSS � User and supported by Eurostat (EUPL 1. 1, LGPL v 3) needs driven development � Employed in national and international organisations � Governed by ex-SDMX Tools Task Force and its members now called Shared Tools Expert Group June, 2019 SDMX Workshop, Rome Eurostat

Release cycle � Public release – once per year; � Test releases - approx. 2 per month � Sdmx. org � Eurostat SDMX Info. Space � Nexus (Java) � Nuget (. NET) � Git/Stash (development) June, 2019 SDMX Workshop, Rome Eurostat

Tools usage � Used inside and outside ESS � � ''As IS'' users ''Fit for purpose'' users ''Add new'' users Environment for collaborative development and sharing of software (CITnet) � � � � � Guidelines Wiki Source Version Control Software Issue tracker Code explorer and Collaborative Code Review Software code repository management Continuous integration Agile Iteration Management and Planning Collaborative documentation June, 2019 SDMX Workshop, Rome Eurostat

Why we collaborate � Same/similar � Use cases � Requirements � Users � But � Different implementations and/or behavior � Duplications of effort/resources � Double work for the end users June, 2019 SDMX Workshop, Rome Eurostat

Collaboration levels Started in 2016 under the umbrella of SDMX Tools Task Force • Pushing the standard forward • Seeking common solutions to common problems SD MX SERVICES E FRONT ND • Implementation of SDMX • Modular and plug-able architecture June, 2019 • Common frameworks • Component re-use SDMX Workshop, Rome Eurostat

Services � SDMX standard implementation � Extended coverage � Including experimental or not yet in SDMX features � Having � Using � With a working example helps establish better SDMX features the same NSI Web Service different plugins and data/structure stores � Enhancing the common SDMX library � Sdmx. Source June, 2019 SDMX Workshop, Rome Eurostat

2018 -2019 Overview � Continuous � SDMX-RI improvements Web Service, Mapping Assistant and MSDB � Implementation of the new SDMX-RI � Design and planning finished (end 2017) � New architecture � Mapping Assistant from a desktop to Web Application with new GUI � Mapping Assistant Web Service and API � SDMX-RI Web Service � Analysis and integration of new functionalities/components/tools June, 2019 SDMX Workshop, Rome Eurostat

SDMX-RI Architecture June, 2019 SDMX Workshop, Rome Eurostat

SDMX-RI to be Architecture Mapping Assistant DB Auth DB Dissemination Auth DB (DDB) Authentication DB Libraries MAAPI Output Formats Authentication Data Persistance Data Retriever Server Mapping Assistant Web Service Users PCs June, 2019 Structure Persistance Structure Retriever SDMX RI Web Service Mapping Assistant Web Application SDMX Workshop, Rome Eurostat

Sdmx. Source � Common API for SDMX development � Abstracts most of the SDMX complexity from the developers � Available for Java and. NET � Java 7 � Maven �. NET packages Standard 2. 0 � Nu. Get � Eurostat packages actively maintains fork for Java and. NET port � Initial implementation in Java Metadata Technologies � Many contributions from OECD (SDMX-JSON, SDMX-CSV) June, 2019 SDMX Workshop, Rome Eurostat

Sdmx. Source Implementations in Sdmx. Source Example Implementations in other tools SDMX 2. 0 Generic Data Custom Excel format SDMX 2. 0 Compact Data Software uses One streaming API for reading data SDMX 2. 0 Cross Sectional Data SDMX 2. 1 Generic Data SDMX 2. 1 Structure Specific Data SDMX CSV Data SDMX JSON Data SDMX EDI Data June, 2019 SDMX Workshop, Rome Eurostat Custom CSV formats

SDMX RI Web Service � Also known as NSI WS � Available for Java 7 Tomcat, Web. Logic � ASP. NET Core 2. 2, IIS on Windows, Kestrel on Windows/Linux � � SDMX � Compliant Experimental features � Implemented � and actively maintained by Eurostat . NET many contributions from OECD and ISTAT � Performance � Streams data for data requests June, 2019 SDMX Workshop, Rome Eurostat

SDMX RI Web Service Authentication Policy authorization SOAP SDMX v 2. 1 SDMX REST SOAP SDMX v 2. 0 Output Formats Data / Structure Store Authorization Other plugins SDMX XML, CSV, JSON formats, GESMES/TS SDMX RI Data/Structure Retriever Plugins Web Service and Persistence for Mapping Assistant Database. MSDB Dataflow Authorization SDMX Header, Dissemination DB connection RDF from ISTAT Sdmx. Source June, 2019 SDMX Workshop, Rome

How to deploy June, 2019 SDMX Workshop, Rome Eurostat

SDMX-RI June, 2019 new features SDMX Workshop, Rome

2018: Work already done under the SDMXRI �Improve SDMX standard coverage & new features � Available Data constraint support via REST � Support for reference partial detail (. NET implemented by OECD) � SDMX v 2. 1 Time Range & Time Zone off-sets in. NET � SDMX CSV data (. NET implemented by OECD) � SDMX JSON data/structure v 1. 0. 0 support (. NET implemented by OECD) � Submit Structure Append/Replace behaviour alignment June, 2019 SDMX Workshop, Rome Eurostat

2018: Work already done under the SDMXRI �SDMX Experimental features � SDMX REST Paging data and structures � Specify format/version in the URL �Authentication/Authorization � Move authentication to a new DB schema � De-couple access rules/roles authorization from NSI WS June, 2019 SDMX Workshop, Rome Eurostat

2018: Work already done under the SDMXRI � Data Registration � Custom URL (to be done) � Set up for missing values (Na. N vs empty) � DENODO support � Transcoding improvement: � Description matching (Jaccard or Damerau) � Transcoding for non calendar years June, 2019 SDMX Workshop, Rome Eurostat

2018: Work already done under the SDMXRI � Initialisation of the Auth. DB using a simple HTML page through the MA WS � Documentation improvement � Categorisation implementation � MA API and MA API WS initial implementation � NSI WS bug fixes � Packaging improvements in. NET finished (Include in the MA WS war an instance of NSI WS to simplify deployment). Java to be done June, 2019 SDMX Workshop, Rome Eurostat

2019: Work already done under the SDMXRI �Improve SDMX standard coverage & new features � REST query for items in item scheme � SDMX-CSV Accept header parameters (. NET implemented by OECD) � SDMX v 2. 1 Time Range & Time Zone off-sets in Java � Support for frequencies with multiplier e. g. A 10 � Extend support for Content Constraints � REST query for Actual and Allowed constraints � Support for content constraints attaching to other maintainable artefacts June, 2019 SDMX Workshop, Rome Eurostat

2019: Work already done under the SDMXRI � SDMX RI. NET � WS and MAWS migrated to. NET Core 2. 2 � Libraries migrated to. NET Standard 2. 0 � WS. NET on Linux (Implemented by OECD) � Mapping Assistant, NSI Client and Test Client target. NET Framework 4. 6. 2 � SDMX RI Java � Reduced feature gap with. NET Improved testing � Plug-in subsystem in NSI WS Java � � Testing � � Automated end to end tests Use of Docker June, 2019 SDMX Workshop, Rome Eurostat

2019: Work already done under the SDMXRI �Mapping Assistant compatible with the new SDMXRI Web Service and Mapping Store �Targets. NET v 4. 6. 2 �NSI Client June, 2019 SDMX Workshop, Rome Eurostat

New features covered under the MA API/WS � Import structural metadata from an SDMX Registry � Transcoding and Mapping enhancements � Support for multiple Mapping Assistant DB � Select Mapping Assistant DB per request � Authentication/Authorization changes (NSI/MA WS) � Roles/Access attributes � Use of single authentication store for both NSI WS and the new Mapping Assistant WS June, 2019 SDMX Workshop, Rome Eurostat

New features covered under the MA API/WS � Upgrading a Dataflow � Using Structure Set � Copy mapping set to a newer version of a dataflow/DSD � Copy Mapping Assistant DB to another DB (import/export) � SDMX and entities � Or just an entity with parents � On the Fly DSD creation � From June, 2019 a dataset SDMX Workshop, Rome Eurostat

Developed (Under testing) � Mapping Assistant Web application � MSDB v 6. 6 � Proxy settings � Content Constraints attachment June, 2019 SDMX Workshop, Rome Eurostat

Ongoing � Alignment �Mapping (. NET vs Java ) ongoing Assistant Web Application � Tuning � Integration �Proxy settings in MSDB �SDMX REST Include. History support �Enhance DDB vendor support June, 2019 SDMX Workshop, Rome Eurostat

Planned after June � On the Fly DSD (with a template) � Tasks under the collaborative development June, 2019 SDMX Workshop, Rome Eurostat

Communication � Product requirements https: //webgate. ec. europa. eu/CITnet/confluence/display/SD MXRI/Requirements � How to pages https: //webgate. ec. europa. eu/CITnet/confluence/display/SD MXRI/02+How-to+articles � Development news https: //webgate. ec. europa. eu/CITnet/confluence/display/SD MXRI/Development June, 2019 SDMX Workshop, Rome Eurostat

Technology Stack/Requirements � OS . NET (Windows, Linux on beta) � Java (Linux or Windows) � � Hardware � RAM (Memory) � Minimum 1 GB � Recommended 2 GB+ � CPU � Intel/AMD 64 bit (recent) � Hard disk � Minimum 100 MB � Recommended 512ΜΒ – 1 GB June, 2019 SDMX Workshop, Rome Eurostat

Technology Stack/Requirements � Java 7 (compatible with Java 8) � JAX-WS, JAX-RS � Web Logic 12. 1. 3+ and Tomcat 8+ � �. NET � . NET Core � NSI � WS and MAWS . NET v 4. 6. 2 � Desktop � Mapping Assistant, NSI Client, Test Client . NET standard v. 2 � Sdmx. Source, � MAAPI, Data Retriever , Structure Retriever IIS 7. 5 or later (Optional for NSI WS!) June, 2019 SDMX Workshop, Rome Eurostat

Technology Stack/Requirements � DB � Maria DB 10. 2. x or later � Oracle 11 g or later � SQL Server 2014 or later � My. SQL works but not recommended � Java script � Browsers � Internet � React � ES June, 2019 Explorer 11, Edge, current Firefox and Chrome 15 16 SDMX Workshop, Rome Eurostat

Migration to. NET Core Sdmx. Source. NET and SDMX RI. NET June, 2019 SDMX Workshop, Rome Eurostat

Overview � Migrate Sdmx. Source/SDMX RI libraries to. NET Standard 2. 0 � When an API is missing, migrate to. NET Core 2. 2 � Migrate Web Services to ASP. NET Core 2. 2 � Migrate NUnit tests to. NET Core or. NET Framework � Upgrade 3 rd June, 2019 party dependencies SDMX Workshop, Rome Eurostat

Libraries � Sdmx. Source � Estat Sdmx. Source Extension � MAAPI � DR � SR June, 2019 SDMX Workshop, Rome Eurostat

Web Services � NSI WS WCF and ASMX based � REST endpoints � � Migrate � to. NET Core compatible REST Web Service stack SOAP endpoints � Implement SOAP parser/writer � Over REST � Using. NET Core compatible REST Web Service stack � MA WS Web Api 2 based � Try to use Web Api 2. NET Core or migrate to other Web Service stack � June, 2019 SDMX Workshop, Rome Eurostat

Tests � Based � Use � If on NUnit 2. x or NUnit 3. x . NET Framework 4. 6. 2 in first iteration there is time � Migrate to NUnit 3. x � Use. NET Core NUnit adapter nuget package June, 2019 SDMX Workshop, Rome Eurostat

Constraints � Need to migrate projects in specific order � If a dependency is not in. NET Standard/Core then the project wont compile � Everything depends on Sdmx. Source � So it needs to build first � Then everything except Sdmx. Source depends on Estat Sdmx. Source � So June, 2019 on… SDMX Workshop, Rome Eurostat

Issues � Behavior changes in. NET Core APIs � Standard API behaved differently in. NET Core � Workarounds / Documentation in github tickets � Missing functionality � Parts of Database vendor independent API was removed � Configurable settings for Web Services/HTTP modules � Tooling June, 2019 support issues SDMX Workshop, Rome Eurostat

1. Sdmx. Source DR, SR Migration order chart 2. Estat Sdmx. Source Extension 5. NSI WS 3. MAAPI 6. MA WS 4. DR 4. SR June, 2019 SDMX Workshop, Rome Eurostat

Thank you! Questions June, 2019 SDMX Workshop, Rome Eurostat