GSIM Mapping to SDMX and DDI Preliminary Findings

GSIM Mapping to SDMX and DDI: Preliminary Findings and Status Arofan Gregory Chris Nelson Metadata Technology Eurostat, June 4/5, 2013 Luxembourg

Outline • • • Working process Goals Mapping Gaps At-a-Glance Business Group Concepts Group Production Group Structures Group Direct mapping between DDI and SDMX Conclusions

Working Process • Group includes representatives from NSIs, international organizations, central banks, and standards bodies – Includes active implementers • Meets bi-weekly via conference calls • Split into two groups – GSIM-DDI and GSIM SDMX • Significant cross-membership to ease coordination

Goals • GSIM 1. 0 had some initial mappings against various standards (incl. DDI and SDMX) – This is the basis of the current work • Will produce a mapping table from GSIM to DDI, GSIM to SDMX • Locate gaps for each standard, provide feedback to standards bodies • Find any gaps in the GSIM model itself • Support the GSIM “plug & play” cases

Design Principles • Support from one standard is enough – Both DDI and SDMX do not need to support everything • No “forced” mappings – If strong semantics don’t exist, identify as a gap

Standards-Mapping Gaps At-A-Glance • Some quick statistics on the current state of the mappings*: – Crimson means a gap – Green means a strong map – Orange means a possible map, either through qualification or further investigation • Note that not all gaps need to be filled – Some are covered by other standards – Some don’t require support from SDMX or DDI * Slides courtesy of Therese Lalor, UN/ECE, presented at METIS, May 2013

SDMX 34 6 93

DDI 42 52 39

DDI 3. 2 29 77 27

Data Collection Specify needs Processes Activity and Design Classifications Unit Data Structures Dimensional Data Structures Dissemination Variables Population and Units

Business Group • Statistical Activity and Design – Not well-supported by DDI, but support should be better in future versions – SDMX has a very generic process model • Data collection area is well-supported by DDI, little or no coverage from SDMX • Specification of needs – DDI is weak, documentary not machine-actionable – Requirements will be passed to the DDI Alliance – Could use SDMX Reference Metadata to support this, but no explicit support

Concepts Group • Major gap: classification management • Otherwise, good support from DDI • Reasonable support from SDMX but no distinction between codes and categories • Populations, statistical units not well supported • We expect strong alignment with future versions of DDI (some changes already to DDI 3. 2 in public review)

Production Group • Not good “actionable” metadata in SDMX or DDI • Gaps here are not critical – Other standards support this better • BPMN • BPEL • We expect better support from DDI in future – DDI 3. 2 offers some better support than 3. 2 • SDMX has a process model, but not as much detail

Structures Group • The best coverage for GSIM is in this part of the model – Data sets – Data structures • This is the key metadata from the standards perspective – Also for many implementations • Dissemination better described by SDMX than by DDI – We do not expect DDI to support this function

Direct Mapping between DDI and SDMX • GSIM provides a framework for understanding how the implementation standards (DDI and SDMX) can be aligned to provide coverage for different aspects of statistical production • This does not remove the need for direct transformations (“mappings”) between the standards in some cases • The typical example is where a microdata set described using DDI is tabulated, producing a DDI Ncube – for reporting and dissemination purposes, this can be directly transformed into an SDMX data set (with corresponding structural metadata)

Conclusions • Fair coverage by standards in some areas • Identified gaps will, where necessary, result in requirements being passed to the standards organizations • Further work is needed to complete the mapping/analysis • GSIM will provide a good model for convergence of standards in future