ESSnet on microdata linking and data warehousing in

ESSnet on microdata linking and data warehousing in statistical production Harry Goossens – Statistics Netherlands Head Data Service Centre / ESSnet Coordinator hct. goossens@cbs. nl ESS-net DWH

General context MEETS action 3. 1: “make better use of data that already exist in the statistical system, including the possibility of estimates” Ultimate aim: “create fully integrated data sets for enterprise and trade statistics at micro level” Ø a 'data warehouse' approach to statistics ESS-net DWH 1

Partnership (FPA) ESS-net coordinator: § Statistics Netherlands (CBS) Co-partners: § Estonia, Italy, Lituania, Portugal, Sweden, UK Starting date: § 4 October 2010 § SGA 1: first year, till 3 October 2011 § SGA 2: last 2 years, till 3 October 2013 ESS-net DWH 2

DWH Definition Broad definition for this ESSnet: A common conceptual model for all available data of interest, enabling the NSI: to (re)use this data to create new data/new outputs, to produce the necessary statistical information and to perform reporting and analysis, regardless of the data’s source. ESS-net DWH 3

Objectives General objective ESS-net: Provide assistance in the development and implementation of a maximum efficient statistical process for business and trade statistics, independent of any (technical) specific architecture Results in daily statistical practice: § § increase the efficiency of dataprocessing in statistical production systems maximize the reuse of already collected data ESS-net DWH 4

Main goals (at start) § Review of current practices in integrated business data systems - realised solutions, ongoing developments - problems, desired solutions, opportunities § Examination of ways in which data can be combined to support new outputs - Role of Business Register, metadata § Provision of recommendations on how the ESS can improve data warehousing § Dissemination of results to all ESS countries (and broader) ESS-net DWH 5

Actions around 3 axes 1. Stocktaking / Inventory of current situation in MS - several MS already introduced data warehousing; - important to make it possible to compile and compare current practices. 2. Analysis and Development - development of a functional model of the S-DWH; - model as basis to assess all issues, raised within this ESSnet, in a generic and standardized way. 3. Dissemination - sharing knowledge and expertise. ESS-net DWH 6

First SGA Activities mainly around axis 1 (stock taking) § What is already been done, experienced by MS ? § DWH is a very broad field and scope: First strictly determine specific detailed subjects/topics to explore and study in depth. and axis 3 (disssemination) § Knowing and sharing what is going on Ø Finding out what the ESS needs / wants ESS-net DWH 7

The Statistical Data Warehouse Rules for generating samples etc Data extracts Dataset Admin data source Business register Working data Selected samples Staging area Dataset Aggregate Statistics Microdata BR snapshots Data extracts Rules for updating BR Input reference frame ESS-net DWH Input / Micro data Storage, combination Outputs 8

First SGA Conclusions § Data Warehousing in statistics is ‘hot’ § Metadata is found important…. . but also often neglected ! § S-DWH is very difficult to compare with common commercial DWH § Visiting NSIs has proven very effective for gathering information AND for sharing knowledge and expertise Ø Great need for knowledge & expertise ESS-net DWH 9

Second SGA § 24 months (till 10/2013 – end of MEETS) § 3 Workshops planned § 4 WP (+ management & coordination) § Interconnecting with other ESSnets § Keep overviews BP up-to-date § Continuation of visiting NSIs to support actively Ø Strong focus on dissemination ESS-net DWH 10

WP 1 - Metadata § Building a framework which defines metadata requirements and roles in the S-DWH context § Study on the use of metadata models and standards: define the various functionalities of a metadata system to facilitate and support the operation of the S-DWH § Provide recommendations and guidelines on the governance of metadata management in the S-DWH ESS-net DWH 11

WP 2 - Methodological topics Building the S-DWH: Methodological topics § Study and recommendations on the position and role of the Business Register (BR) in the context of a S-DWH § Examine selective editing options for the S-DWH and provide guidelines § Study and recommendations on the various ways of ‘linking data’, considering aspects of combining data (from surveys and administrative sources etc. ) § Study on the impact of data confidentiality in micro data linking and data warehousing, considering confidentiality aspects of combining data from various sources etc. ESS-net DWH 12

WP 3 - Technical aspects Building the S-DWH: Technical aspects § Provide a Business Architecture of the S-DWH § Produce a functional architecture of the S-DWH § Produce a modular workflow of the S-DWH § Relate the 'ideal' architectural schemes (frameworks) into an actual implementation and development strategy ESS-net DWH 13

WP 4 - Dissemination § § § Keep the ‘stocktaking of best practices, problems encountered and desired solutions’ up-to-date Create a glossary and keep it up-to-date Create an overall report, containing all deliverables of this ESSnet (guidelines, recommendations, models etc. ) Create an overall handbook to set up the S-DWH Establishing and implementing a ‘knowledge and expertise centre’ to manage the expertise and knowledge acquired in an active/dynamic manner Ø ensure sustainability after this ESSnet is completed ESS-net DWH 14

Project management Specific attention: § overlap, compatibility and interfacing with other ESSnets - sharing and using results § scoping, prioritising - keep the ESSnet manageable ESS-net DWH 15