Standardizing and industrializing a business process the dissemination

Standardizing and industrializing a business process – the dissemination use case Alessio Cardacino - (alcardac@istat. it) ESTP Course “Information standards and technologies for describing, exchanging and disseminating data and metadata” THE CONTRACTOR IS ACTING UNDER A FRAMEWORK CONTRACT CONCLUDED WITH THE COMMISSION Eurostat

SDMX architectures and tools supporting data dissemination and reporting processes 2 Eurostat

SDMX database-driven Architecture 3 Eurostat

SDMX Hub Architecture National Statistical Institute Eurostat Census Hub National Statistical Institute 4 Eurostat

Eurostat SDMX Reference Architecture Data user Data Collectors Test Client Web Service Provider Mapping Assistant Web Client Auth. DB NSI Mapping Store DB Test. Auth. Config Dissemination PC-AXIS Disseminatio n RDBMS 5

SDMX Istat toolkit 6 Eurostat

Implementing the data dissemination process through the «SDMX Datamanager » web application 7 Eurostat

SDMX Modelling of the primary hypercube Hypercube definition: A single hypercube containing data on: • Industrial turnover index, Italy (Base 2010=100) • Industrial turnover index, Italy (Base 2015=100) 8 Eurostat

Industrial turnover index table 9 Eurostat

Creation of the primary data hypercube from the STS+IT 1+1. 0 DSD through the «SDMX Datamanager» web application 10 Eurostat

Access to the SDMX Datamanager Username: admin Password: admin 11 Eurostat

The starting window (Builder) Click here to load structural metadata Here Cubes are shown Here DSDs are shown 12 Eurostat

Load structure metadata: Category schemes The first action that must be performed is to upload two or more SDMX Category schemes: 1 will be used to categorize the cubes, the others to categorize SDMX Data flows: q Click on the [Import SDMX Structure] button q Select the file “BL_DW+IT 1+1. 0. xml” (used for categorizing the cubes) q Click on the [Import SDMX Structure] button q Select the file “DDB_DW+IT 1+1. 0. xml” (used for categorizing the Dataflows) (1) (2) (3) (4) Eurostat 13

Import of the DSD In the «Manager Structure» interface of the «Builder» section, click on the «Import Structure» button. In the pop-up window, click on the «Add File» button, select the STS+IT 1+1. 0, containing the DSD and the referenced artefacts, and click on the «Upload» button. 14 Eurostat

Selection of the categoryscheme for categorizing the cubes When the first cube is created, after having selected the DSD to be used for defining its structure, is shown a popup window containing the list of the available categoryschemes, from which has to be selected that to be used for categorizing the cubes 15 Eurostat

Primary hypercube creation In the «Manager Structure» interface of the «Builder» section: 1. select the STS+IT 1+1. 0 DSD (that has to be final) 2. Click on the «Select Category» button and in the popup window select the «Industry and Construction» category 16 Eurostat

Primary hypercube creation 3. Fill in the «Datacube unique code» textbox the INDTURN code 4. Click on the «Add Name» button, and in the popup windows fill in the italian and english name of the hypercube: • Industrial turnover index 17 Eurostat

Primary hypercube creation Select the conditional Attributes to be included in the hypercube structure (Dimensions and mandatory Attributes are selected as a default) and click on 18 the «Create Datacube» Eurostat

Primary hypercube creation 19 Eurostat

Csv data upload on the primary hypercube through the «SDMX Datamanager» web application 20 Eurostat

Mapping of the csv data structure The columns of the csv file must be mapped against the data structure of the hypercube in order to allow the data import. 21 Eurostat

Mapping of the csv data structure : steps 1. In the «Manager mapping» interface of the «Builder» section, click on the «New Mapping. Set» button. 2. Select the hypercube to be mapped: BL_INDTURN 22 Eurostat

Mapping of the csv data structure : steps 3. Select the csv file IND_TURNOVER. csv to be mapped: 23 Eurostat

Mapping of the csv data structure : steps 4. Perform the mapping: 5. Save mapping 24 Eurostat

Load the csv data file 1. In the «Data import» interface of the «Loader» section, select the mapping previously created: 2. Select the file to be uploaded: IND_TURNOVER. csv 25 Eurostat

Load the csv data file 3. Upload the file by clicking on the button «Upload» 4. Preview the data to be imported 26 Eurostat

Load the csv data file 5. Click on the «Import button» 6. At the end of the «Import» task, the system returns a report indicating the state of the upload (successful or not) and the eventual issues with the indication of the affected records 27 Eurostat

View the imported data 7. View the imported records in the «Manager structure» interface of the Builder section, by clicking on the «Cube Details» button 28 Eurostat

Definition and publication of the Dataflows/views to be disseminated through the « «SDMX Datamanager» web application 29 Eurostat

Hypercubes to be disseminated Starting from the primary hypercube ( «Industrial Turnover index» ), have to be disseminated in SDMX format and via web service the following hypercubes, without performing any duplication of data in the definition of the hypercubes: • Industrial turnover Index, base 2010=100 • Industrial turnover Index, base 2015=100 30 Eurostat

SDMX Dissemination though the ISTAT’s SDMX Datamanager and the SDMX-RI’s Nsi. Web. Service 1. SDMX Datamanager has been developed by ISTAT using standard SDMX libraries provided by Eurostat (Sdmx. Source. NET) 2. In order to provide the data through the SDMX web service are automatically generated by the Datamanager SDMX mappings of the local data structures (DDB hypercubes) against the DSDs referenced by the published dataflows that are derived by the DSD through which as been 31 generated the primary cube Eurostat

Definition of the Dataflows Under the «Industry and Construction» category, have to be created the following dataflows: Dataflow Id Dataflow name (en) INDTURN_2010 Industrial turnover Index, base 2010=100 INDTURN_2015 Industrial turnover Index, base 2015=100 32 Eurostat

Dataflow creation of INDTURN_2010 1. In the «Query builder» section, click on the «New Cube View» button: 33 Eurostat

Dataflow creation of INDTURN_2010 2. In the «Dataflow» tab, insert the following information: 34 Eurostat

Dataflow creation of INDTURN_2010 3. In the «Query» tab, select the BL_INDTURN hypercube 4. Select the following cube columns: 35 Eurostat

Dataflow creation of INDTURN_2010 5. Insert the following filter conditions: 36 Eurostat

Dataflow creation of INDTURN_2010 6. Query preview: 37 Eurostat

Dataflow creation of INDTURN_2010 7. Dataflow categorization: 8. Save Dataflow by clicking on the «Save View» button 38 Eurostat

Dataflow creation of INDTURN_2010 9. View in the list and Set in production: 39 Eurostat

Put in production Dataflow/Views From the Dataflow/Views list, click on Then insert eventually header information about sender and/or receiver for the selected dataflow 40 Eurostat

Dataflow INDTURN_2010 in production 41 Eurostat

Dataflow creation of INDTURN_2015 42 Eurostat

Dataflow creation of INDTURN_2015 Selected cube columns: 43 Eurostat

Dataflow creation of INDTURN_2015 Filter conditions: 44 Eurostat

Dataflow creation of INDTURN_2015 Dataflow categorization: Save Dataflow by clicking on the «Save View» button 45 Eurostat

Dataflow creation List of created dataflows: 46 Eurostat

Browse the SDMX WEB Service content (the created dataflows) through the Eurostat’s NSI Client 47 Eurostat

Eurostat’s NSI Client data browsing Selection of a dataflow (INDTURN_2015) and dimension filtering: 48 Eurostat

Eurostat’s NSI Client data browsing Tabular view of selected data 49 Eurostat

Browse the SDMX Web Service content (the created dataflows) through the ISTAT’s SDMX Data Browser 50 Eurostat

ISTAT’s Data Browser Selection of a dataflow (LOMB_16_FGIUR) and dimension filtering: 51 Eurostat

ISTAT’s Data Browser Tabular view of selected data 52 Eurostat