7 SDMX A nontechnical overview of the SDMX
- Slides: 44
7. SDMX: A non-technical overview of the SDMX architecture and IT tools Raynald PALMIERI Eurostat, Unit B. 5 Data and metadata services and standards European Statistical Training Programme (ESTP) SDMX Basics course, 6 -8 March 2018, Luxembourg Eurostat 1
WHERE TO FIND THE TOOLS 2
http: //ec. europa. eu/eurostat/web/sdmx-infospace/sdmx-it-tools 3
Type of tools • Eurostat SDMX tools for designers • Shared tools • SDMX tools for data providers 4
EUROSTAT SDMX TOOLS FOR DESIGNERS 5 Eurostat
Eurostat SDMX tools for designers 6 Tools offered Use Data Structure Wizard Used to create, edit and test SDMX objects Maintenance of SDMX registry Euro SDMX Registry A central repository for storing and sharing SDMX objects Eurostat
Eurostat SDMX tools for designers Data Structure Wizard Web service Create/Edit 7 Eurostat Euro SDMX Registry Store/Edit
Data Structure Wizard • Creation / edition of DSDs • Validation of DSD • Local repository • Artefacts are stored in a local repository • Remote repository • Can be used for retrieving existing artefacts 8
DATA STRUCTURE DEFINITION - Design Data Structure Wizard • • Java desktop application Graphical Interface For DSD designers Maintenance of SDMX v 2. 0/2. 1 data and meta data structures • Web service to query/submit SDMX registries
DSW: Management of SDMX artefacts 10
DSW: Artefact example: the codelists 11
SHARED TOOLS FOR DESIGNERS AND DATA PROVIDERS 12
Work roles and process SDMX Registry Interpret SDMX messages Consult and get DSD COLLECTING ORGANISATION Publish DSD Consult and get DSD Create DSD 13 Send SDMX messages Format / structure SDMX messages
SDMX Registry • Metadata registry which implements the SDMX registry specifications • Tasked with providing structure, organisation, maintenance and query interfaces for the SDMX components required to support the data sharing vision • Provides a web-based user interface as well as web services for interacting with the SDMX structural metadata objects in use within the collecting organisation and with statistical partners. 14 Eurostat
SDMX Registry • Only users concerned with providing information need to access the data and reference metadata sets. • An application which wants a particular data or metadata set would then query the registry for the structure, and then go and retrieve the data or metadata set directly from the provider's web server. • Example of existing SDMX registries: • Euro SDMX Registry (2. 1) • SDMX Global Registry (2. 1) 15 Eurostat
New Euro SDMX Registry (2. 1) § Eurostat's implementation of the SDMX Registry § Based on SDMX 2. 1 (as published on April 2011) Also capable of importing and exporting SDMX 2. 0 artefacts. § Allows browsing, searching, editing and subscribing to artefacts. § Advanced access control mechanism of artefacts controlling their visibility (Public/Private). Eurostat
New Euro SDMX Registry (2. 1) Home page Access to the content of the Registry via text search Access to the content of the Registry by type Most recent items Eurostat
New Euro SDMX Registry (2. 1) Registry Content Filtering list Number of objects of type selected Eurostat
SDMX Global Registry (SGR) • Launched in 2011 by the SDMX Sponsoring institutions • Originally to support the world-wide technical implementation of Balance of Payments and National Account implementations • To make metadata material (DSDs, concept schemes, MSDs, code lists, etc. ) publicly and centrally available used and maintained by national and international organisations.
Who should use the SDMX Global Registry (SGR)? • Any national or international agency that wants to use SDMX standards for data management or for exchanging data with other organisations using SDMX standard. • Any end-user of statistical data can access the data structures posted on the SGR.
How to use the SDMX Global Registry (SGR)? • All information published in the SGR is publicly available and freely viewable and downloadable. • No need to log in to view or to download information - SDMX Global Registry Content. • Users can view the full content of the Global Registry
Exploring by agency Available SDMX objects Click on a type Click on Export to choose a SDMX object to download Enter a text to be found in the Registry Access to the web service
Codelist Content 2. Select an object
1. Choose a DSD in the list Dimensions, measure and attributes Corresponding list of codes
Example of result for "NA-MAIN" 5 versions found
Web Service Query Select the SDMX version, the Object type and the identifiers of the SDMX object (ID, agency, version) Select which referenced objects (e. g. Code lists) you want to include View the xml file in the browser or Download the file The URL generated is the SDMX query to be sent to the web service to get the xml file corresponding to the selected object
View the xml file in your browser (DSD NA_MAIN)
SDMX TOOLS FOR DATA PROVIDERS 28 Eurostat
SDMX Tools for Data Providers Tools offered Use Action Web Forms (EU) Excel-like templates Transmission of low volumes of data No costs for EU organisations Manual work for senders (type, copy/paste) SDMX Converter (sender’s PC) Converts data files between SDMX formats and other file formats Installation on sender’s PC Manual work for senders (convert) SDMX Converter (batch mode) SDMX Reference Infrastructure Installation on server A set of tools that Installation on server allows to connect your Mapping of database to IT systems to the SDMX DSDs world Own SDMX Implementation 29 Local development in organisations Eurostat
Data provider view: One goal – different possibilities Webforms EDAMIS Single Entry Point SDMX-ML file Excel sheets SDMX Converter SDMX-ML file Database export SDMX Converter SDMX-ML file Database SDMX Reference Infrastructure Eurostat SDMX-ML file SDMX Web Service Same DSD Same Format
Data provider view: One goal – different IT Architectures Push mode Webforms Web-based Excel sheets SDMX Converter Push mode Local conversion Database export Local conversion Database SDMX Reference Infrastructure SDMX Converter Push mode Push or Pull mode Eurostat
SDMX Converter http: //ec. europa. eu/eurostat/web/sdmx-infospace/sdmx-it-tools • File based conversion • Open source and platform independent (Java) • Different ways of using it: • • Graphical user interface Batch file (server or client side) Web Service interface Reusing source code in your own Java application • Formats: SDMX-ML 2. 0 & 2. 1, SDMX-EDI (Gesmes), FLR, CSV, Google DSPL, predefined Excel templates Eurostat
GUI (Converter 5. 4. 4) 1. Selection of the input/output files and their format 2. a Select the DSD in the local drive 2. b Identify a DSD to download from the SDMX Registry (configuration required) 2. If the local DSD includes multiple versions, we can specify the one desired derired 2. c Identify a dataflow linked to the DSD to download from the SDMX Registry (configuration required) 3. Excel parameter file 3. SDMX header (. prop file) CSV parameters Only for flat and excel files 4. Mapping and Transcoding 6. SDMX (output) validation 5. CSV quotation XML parameters for SDMX output formats Eurostat
New Converter Interface Eurostat (6. 11. 4)
Web Interface http: //wlts 0018. cc. cec. eu. int: 2061/converter-webapp/convert (Internal Eurostat URL) Eurostat
36 Eurostat
37 Eurostat
Exercice: Converter • Installation • From CSV to SDMX Compact 38 Eurostat
What is SDMX-Reference Infrastructure • Framework for SDMX data provision • Set of pick-and-choose reusable building blocks allowing a statistical office to disseminate SDMX data to the external world • Designed to provide data and structural metadata based on mappings to each organization's dissemination data warehouse • Uses SDMX standards including the one for Web Services 39 Eurostat
Data provider view: SDMX Reference Infrastructure Database SDMX Reference Infrastructure Push or Pull mode Eurostat
Tools Relationships – Example Store Produce DSW SDMX Registry DSD Produce use SDMX Converter SDMX-RI SDMX-ML Dataset Mapping Assistant Produce CSV File 41 use Eurostat Mapping National DB
SDMX Tools – Summary • Eurostat tools Eurostat SDMX Info Space • http: //ec. europa. eu/eurostat/web/sdmx-infospace/sdmx-it-tools • SDMX Converter • SDMX Reference Infrastructure & Mapping Assistant • SDMX Registry & Data Structure Wizard Eurostat
Support for data providers • SDMX Reference Infrastructure ü Installed in ~32 countries for Census Hub ü Can be extended to other domains • 1 st contact point - Local Transmission Coordinators ü In National Statistical Institutes ü Experience with EDAMIS and SDMX • 2 nd contact point - Eurostat support teams Eurostat
Thank you for your attention! Questions Eurostat
- Sdmx converter
- Ecb sdmx
- Sdmx excel
- Sdmx istat
- Http://sdmx.store:8080
- Sdmx converter
- Lời thề hippocrates
- đại từ thay thế
- Quá trình desamine hóa có thể tạo ra
- Môn thể thao bắt đầu bằng chữ đua
- Cong thức tính động năng
- Thế nào là mạng điện lắp đặt kiểu nổi
- Hát kết hợp bộ gõ cơ thể
- Khi nào hổ con có thể sống độc lập
- Các loại đột biến cấu trúc nhiễm sắc thể
- Thế nào là sự mỏi cơ
- Vẽ hình chiếu đứng bằng cạnh của vật thể
- Phản ứng thế ankan
- Chó sói
- Thiếu nhi thế giới liên hoan
- điện thế nghỉ
- Một số thể thơ truyền thống
- Thế nào là hệ số cao nhất
- Trời xanh đây là của chúng ta thể thơ
- Ng-html
- So nguyen to
- đặc điểm cơ thể của người tối cổ
- Phối cảnh
- Các châu lục và đại dương trên thế giới
- Chụp phim tư thế worms-breton
- ưu thế lai là gì
- Hệ hô hấp
- Tư thế ngồi viết
- Cái miệng nó xinh thế
- Mật thư anh em như thể tay chân
- Bổ thể
- Tư thế ngồi viết
- Thứ tự các dấu thăng giáng ở hóa biểu
- Thẻ vin
- Thể thơ truyền thống
- Alleluia hat len nguoi oi
- Các châu lục và đại dương trên thế giới
- Khi nào hổ con có thể sống độc lập
- Từ ngữ thể hiện lòng nhân hậu
- Diễn thế sinh thái là