Standardizing and industrializing a business process the dissemination
Standardizing and industrializing a business process – the dissemination use case Alessio Cardacino - (alcardac@istat. it) ESTP Course “Information standards and technologies for describing, exchanging and disseminating data and metadata” THE CONTRACTOR IS ACTING UNDER A FRAMEWORK CONTRACT CONCLUDED WITH THE COMMISSION Eurostat
SDMX architectures and tools supporting data dissemination and reporting processes 2 Eurostat
SDMX database-driven Architecture 3 Eurostat
SDMX Hub Architecture National Statistical Institute Eurostat Census Hub National Statistical Institute 4 Eurostat
Eurostat SDMX Reference Architecture Data user Data Collectors Test Client Web Service Provider Mapping Assistant Web Client Auth. DB NSI Mapping Store DB Test. Auth. Config Dissemination PC-AXIS Disseminatio n RDBMS 5
SDMX Istat toolkit 6 Eurostat
Implementing the data dissemination process through the «SDMX Datamanager » web application 7 Eurostat
SDMX Modelling of the primary hypercube Hypercube definition: A single hypercube containing data on: • Industrial turnover index, Italy (Base 2010=100) • Industrial turnover index, Italy (Base 2015=100) 8 Eurostat
Industrial turnover index table 9 Eurostat
Creation of the primary data hypercube from the STS+IT 1+1. 0 DSD through the «SDMX Datamanager» web application 10 Eurostat
Access to the SDMX Datamanager Username: admin Password: admin 11 Eurostat
The starting window (Builder) Click here to load structural metadata Here Cubes are shown Here DSDs are shown 12 Eurostat
Load structure metadata: Category schemes The first action that must be performed is to upload two or more SDMX Category schemes: 1 will be used to categorize the cubes, the others to categorize SDMX Data flows: q Click on the [Import SDMX Structure] button q Select the file “BL_DW+IT 1+1. 0. xml” (used for categorizing the cubes) q Click on the [Import SDMX Structure] button q Select the file “DDB_DW+IT 1+1. 0. xml” (used for categorizing the Dataflows) (1) (2) (3) (4) Eurostat 13
Import of the DSD In the «Manager Structure» interface of the «Builder» section, click on the «Import Structure» button. In the pop-up window, click on the «Add File» button, select the STS+IT 1+1. 0, containing the DSD and the referenced artefacts, and click on the «Upload» button. 14 Eurostat
Selection of the categoryscheme for categorizing the cubes When the first cube is created, after having selected the DSD to be used for defining its structure, is shown a popup window containing the list of the available categoryschemes, from which has to be selected that to be used for categorizing the cubes 15 Eurostat
Primary hypercube creation In the «Manager Structure» interface of the «Builder» section: 1. select the STS+IT 1+1. 0 DSD (that has to be final) 2. Click on the «Select Category» button and in the popup window select the «Industry and Construction» category 16 Eurostat
Primary hypercube creation 3. Fill in the «Datacube unique code» textbox the INDTURN code 4. Click on the «Add Name» button, and in the popup windows fill in the italian and english name of the hypercube: • Industrial turnover index 17 Eurostat
Primary hypercube creation Select the conditional Attributes to be included in the hypercube structure (Dimensions and mandatory Attributes are selected as a default) and click on 18 the «Create Datacube» Eurostat
Primary hypercube creation 19 Eurostat
Csv data upload on the primary hypercube through the «SDMX Datamanager» web application 20 Eurostat
Mapping of the csv data structure The columns of the csv file must be mapped against the data structure of the hypercube in order to allow the data import. 21 Eurostat
Mapping of the csv data structure : steps 1. In the «Manager mapping» interface of the «Builder» section, click on the «New Mapping. Set» button. 2. Select the hypercube to be mapped: BL_INDTURN 22 Eurostat
Mapping of the csv data structure : steps 3. Select the csv file IND_TURNOVER. csv to be mapped: 23 Eurostat
Mapping of the csv data structure : steps 4. Perform the mapping: 5. Save mapping 24 Eurostat
Load the csv data file 1. In the «Data import» interface of the «Loader» section, select the mapping previously created: 2. Select the file to be uploaded: IND_TURNOVER. csv 25 Eurostat
Load the csv data file 3. Upload the file by clicking on the button «Upload» 4. Preview the data to be imported 26 Eurostat
Load the csv data file 5. Click on the «Import button» 6. At the end of the «Import» task, the system returns a report indicating the state of the upload (successful or not) and the eventual issues with the indication of the affected records 27 Eurostat
View the imported data 7. View the imported records in the «Manager structure» interface of the Builder section, by clicking on the «Cube Details» button 28 Eurostat
Definition and publication of the Dataflows/views to be disseminated through the « «SDMX Datamanager» web application 29 Eurostat
Hypercubes to be disseminated Starting from the primary hypercube ( «Industrial Turnover index» ), have to be disseminated in SDMX format and via web service the following hypercubes, without performing any duplication of data in the definition of the hypercubes: • Industrial turnover Index, base 2010=100 • Industrial turnover Index, base 2015=100 30 Eurostat
SDMX Dissemination though the ISTAT’s SDMX Datamanager and the SDMX-RI’s Nsi. Web. Service 1. SDMX Datamanager has been developed by ISTAT using standard SDMX libraries provided by Eurostat (Sdmx. Source. NET) 2. In order to provide the data through the SDMX web service are automatically generated by the Datamanager SDMX mappings of the local data structures (DDB hypercubes) against the DSDs referenced by the published dataflows that are derived by the DSD through which as been 31 generated the primary cube Eurostat
Definition of the Dataflows Under the «Industry and Construction» category, have to be created the following dataflows: Dataflow Id Dataflow name (en) INDTURN_2010 Industrial turnover Index, base 2010=100 INDTURN_2015 Industrial turnover Index, base 2015=100 32 Eurostat
Dataflow creation of INDTURN_2010 1. In the «Query builder» section, click on the «New Cube View» button: 33 Eurostat
Dataflow creation of INDTURN_2010 2. In the «Dataflow» tab, insert the following information: 34 Eurostat
Dataflow creation of INDTURN_2010 3. In the «Query» tab, select the BL_INDTURN hypercube 4. Select the following cube columns: 35 Eurostat
Dataflow creation of INDTURN_2010 5. Insert the following filter conditions: 36 Eurostat
Dataflow creation of INDTURN_2010 6. Query preview: 37 Eurostat
Dataflow creation of INDTURN_2010 7. Dataflow categorization: 8. Save Dataflow by clicking on the «Save View» button 38 Eurostat
Dataflow creation of INDTURN_2010 9. View in the list and Set in production: 39 Eurostat
Put in production Dataflow/Views From the Dataflow/Views list, click on Then insert eventually header information about sender and/or receiver for the selected dataflow 40 Eurostat
Dataflow INDTURN_2010 in production 41 Eurostat
Dataflow creation of INDTURN_2015 42 Eurostat
Dataflow creation of INDTURN_2015 Selected cube columns: 43 Eurostat
Dataflow creation of INDTURN_2015 Filter conditions: 44 Eurostat
Dataflow creation of INDTURN_2015 Dataflow categorization: Save Dataflow by clicking on the «Save View» button 45 Eurostat
Dataflow creation List of created dataflows: 46 Eurostat
Browse the SDMX WEB Service content (the created dataflows) through the Eurostat’s NSI Client 47 Eurostat
Eurostat’s NSI Client data browsing Selection of a dataflow (INDTURN_2015) and dimension filtering: 48 Eurostat
Eurostat’s NSI Client data browsing Tabular view of selected data 49 Eurostat
Browse the SDMX Web Service content (the created dataflows) through the ISTAT’s SDMX Data Browser 50 Eurostat
ISTAT’s Data Browser Selection of a dataflow (LOMB_16_FGIUR) and dimension filtering: 51 Eurostat
ISTAT’s Data Browser Tabular view of selected data 52 Eurostat
- Slides: 52