Presentation plan Accessing and Retrieving SDMX data Visualising

Presentation plan • Accessing and Retrieving SDMX data • Visualising and analysing SDMX data Excel R and Python Modern BI tools (example with Qlik Sense)

Accessing and Retrieving SDMX data SDMX defines a set of web services to retrieve SDMX objects and data. Available both via SOAP and REST APIs

SDMX REST API – SDMX objects http: //Root. URL/Object. Type/Agency. ID/Object. ID/version • Root. URL: location of the REST API. For Eurostat data, it is http: //ec. europa. eu/eurostat/SDMX/diss-web/rest/ • Object. Type: type of object requested (e. g. dataflow or datastructure) • Agency ID: The maintenance agency for the object requested (e. g. ESTAT) • Object. ID: ID of the object requested • Version: version number of object requested (e. g, 1. 0 or latest)

Example: Retrieving information about Eurostat data Each table in the Eurostat database corresponds to an SDMX Dataflow ID

Example: Retrieving information about Eurostat data Query for retrieving the SDMX dataflow: http: //ec. europa. eu/eurostat/SDMX/dissweb/rest/dataflow/ESTAT/une_rt_m/latest Query for retrieving the corresponding SDMX DSD: http: //ec. europa. eu/eurostat/SDMX/dissweb/rest/datastructure/ESTAT/DSD_une_rt_m /latest

SDMX REST API – SDMX data http: //Root. URL/data/Flow. Ref/Key/parameters • Root. URL: location of the REST API. For Eurostat data, it is http: //ec. europa. eu/eurostat/SDMX/diss-web/rest/ • Flowref: The ID of the desired dataflow • Key: The subset of the data requested, as specified by a key (i. e. a set of values for the dimensions in the DSD). Elements of the key are separated by dots. • Parameters: Optional parameters that can further identify a subset of the data (e. g. startperiod, endperiod or last. NObservations). Parameters are always introduced by a "? " sign and separated by a "&" sign.

Example: Retrieving Eurostat data I want monthly unemployment rate data from Austria from 2010 to 2013 http: //Root. URL/data/Flow. Ref/Key/parameters • Root. URL: http: //ec. europa. eu/eurostat/SDMX/diss-web/rest/ • Flowref: une_rt_m • Key: …. . AT • Parameters: ? startperiod=2015&endperiod=2017

Retrieving SDMX data: caveats In order to correctly input the key parameter, I need to know the correct codes and the correct order of the dimensions. I must look at the DSD! If I do not want to look for a specific key, I can just use "all" instead of the dots SDMX data comes in many formats. The default format for the REST API is the "SDMX Generic format". If I want another format, I need to modify the query header. But for that I need a REST client…

Visualising and Analysing SDMX data SDMX-ML great for data processing. But not great for understanding the data at first glance…

Visualising and Analysing SDMX data Options: • Using the SDMX Converter to convert data from SDMX-ML to CSV • Opening the SDMX-ML file directly with Excel • Taking advantages of analytical libraries for parsing XML data or SDMX-ML data specifically

Using Excel to visualise SDMX-ML • Import the SDMX-ML file directly in Excel. Select the "As XML Table" option and ignore the pop-up warnings. • Excel will flatten the nested SDMX structure. We can then easily build a pivot table to analyse and visualise the data

SDMX data using R or Python • There are SDMX libraries available for both R and Python • These libraries can directly connect to SDMX Web Services import and transform SDMX data into native formats

SDMX data using R or Python • For R, two very good libraries are available: rsdmx and RJSDMX (the latter developed by Banca d'Italia) • • For Python, there is a library called panda. SDMX (with very good online documentation) • These libraries can directly connect to SDMX Web Services import and transform SDMX data into native formats

SDMX data using R • rsdmx can be used to download SDMX objects and SDMX data • • • Advantage: unified and flexible interface (only one method to learn) • • Disadvantage: unified and flexible interface (you need to know what you’re doing)

SDMX data using R • RJSDMX can be used to download SDMX data, but not objects • • • Advantage: “single-minded” approach, less SDMX knowledge expected from users • • Disadvantage: fewer functionalities

SDMX with modern BI tools • All modern BI tools (Tableau, Qlik, Spotfire etc…) have built-in REST connectors and XML parsers • Creating a dashboard based on SDMX data is usually a matter of clicks

Key messages • SDMX data can be retrieved via standardised web services • There are several options to visualise and analyse SDMX data