Modernisation of ESS infrastructure The ESS instruments a

Modernisation of ESS infrastructure: The ESS instruments - a review E. di Meglio – P. Jacques – J. M. Museux 23 -25/5/2011

The Vision n Integration and standardisation of ESS processes : data warehouses n Incorporation of ICT development (SOA, distributed architecture, Web 3. 0) n New methods for producing statistics (reuse, data matching and linking, mixed modes approach, visualisation and analysis of big datasets, …) n New generation of indicators (Stiglitz …) 23 -25 May 2011 MSIS Conference

ESS actions required n Legal framework : from stove pipe legislation to cross cutting framework legislation n Methodological developments /standardisation – Best practices – guidelines – rules – Common tools n Modernisation of IT infrastructure (SOA, web services, distributed and cloud computing, use of standardised IT solutions) n Standardisation and industrialisation – – Common description and semantic Common data exchange standard Common metadata Common tools 23 -25 May 2011 MSIS Conference BACK

Reference ESS Enterprise architecture International standards 23 -25 May 2011 MSIS Conference BACK

Research n Role of Research Framework programme in Official Statistics – FP 5 (35 seminal research projects) – FP 7 (new indicators, data infrastructure, ) and upcoming FP 8 n Preparing and addressing future challenge – New data sources (i. e. Social networking, Internet of Things, Data Streams, mobile info/GI, . . ) – Data analysis / data mining / visualisation of big datasets (Data Deluge) – Web 3. 0, semantics – New modes of data collection and related estimation issues – Understanding non response – Set up e-infrastructures – Critical events forecasting n Collaboration with academics (statistical watch - think tank, adviser for innovation, training of Official statisticians) 23 -25 May 2011 MSIS Conference BACK

ESSnets n A collaborative network of several ESS organisations aimed at providing results beneficial to the whole ESS n 26 ESSnet projects since 2008 n www. essnet-portal. eu n ESSnet review on going – ESSnet coordination - ESSnet planning n ESSnet characterisation according to their contribution to innovation process 23 -25 May 2011 MSIS Conference BACK

ESSnets typology 1. ESSnet research n consolidation and incorporation of research knowledge and testing of prototype tools n Ex. SDC I - ISAD 2. ESSnet pole of expertise n specialisation in process steps – cover wider range of activities (best practices identification, guidelines, tool development, dissemination of knowledge-know how and training n Ex SDC II, DI, SAE, Mixed mode approach 3. ESSnet infrastructure n facilitation to incorporation of international standards, recommendations, cases studies, taylorisation, improvement of standards, setting up of common infrastructure n Ex. EGR, SDMX I & II, MEETS Datawarehouse, Geostat, CORA & CORE, Decentralised access to EU micro data 23 -25 May 2011 MSIS Conference BACK

ESSnets typology 1. ESSnet standardisation n contribute to knowledge and process standardisation (concept, tools, guidelines, …. ) n Ex SDC harmonisation, MEETS Memobust, Preparing Standardisation 2. ESSnet strategy : n support to strategic action like Sponsorship on Standardisation 3. ESSnet piloting n domain dependent, testing in common, practical implementation 23 -25 May 2011 MSIS Conference BACK

ESSnet – contribution to a reference architecture Stand Prep Memobust DI SDC Harm Consistency CORE DW SDMX EGR Decentralised Access ESSnet 23 -25 May 2011 MSIS Conference

Contribution to ESS business process Net SILC Culture Geostat PH Administrative Data Profiling DI ICT Linking SDC 23 -25 May 2011 MSIS Conference SAE BACK

Common tool development n Tool developed in common for implementing ESS guidelines n Need for specific governance and development strategy – Open source – Modular – web service – portability (JAVA) – ESS Steering Group – User group n Best practice – Demetra+ : • Tool for implementing ESS seasonal adjustment guidelines • Demetra+ user group – prototype development • Java toolkit (API, GUI, Data cruncher to be run in batch, library of core functions, structured objects) • Re-used for Ecotrim, NAPS developments – Argus+ : sustainability of T-Argus – proof of concept 23 -25 May 2011 MSIS Conference BACK

Eurostat modernisation - VIPs n Strategic project for Eurostat and ESS modernisation n Joint venture between business area units (relevance) and horizontal units (methodology, IT) n 6 projects (still on going) launched in 2010 -11 1. VIP data validation ESTAT process / infrastructure 2. VIP data matching 3. VIP new dissemination 4. VIP National account production 5. VIP Secure infrastructure for confidential data ESS design 23 -25 May 2011 MSIS Conference 6. VIP on PILLAR approach for household data collection BACK

Eurostat modernisation - VIPs n Common characteristics – Cross domains development – Mainly ESTAT focus - IT or process oriented – Prepare for vertical (MS- Eurostat) integration n Need to maintain global coherence – Planning – yearly work programme – Spin off of ESS(net) project – ESS relevance - coordination 23 -25 May 2011 MSIS Conference

VIP, ESSnet – elements of coherence Stand Prep SDC Harm Memobust DW Data Matching DI Data Validation Pillar Consistency DW CORE Demetra+ Argus+ EGR NAPS SDMX Decentralised Access ESSnet Common tools 23 -25 May 2011 MSIS Conference VIP SICON New Diss

Strategic level – sponsorship n Sponsorship on standardisation n Sponsorship group : 6 general directors or deputies of NSOs, co chaired by Eurostat and a NSO) to build mechanism and steer competence on key challenge for the ESS over a relatively short period (2 years) n Main objectives – to investigate ways to organise and strengthen the sharing of methods and tools and the creation of common infrastructures – to contribute to the establishment of an ESS Enterprise Architecture for standardisation (key principles, …) – to establish a ESS work programme towards a more integrated way of producing European statistics building on existing instrument 23 -25 May 2011 MSIS Conference BACK

VIP, sponsorship and ESSnet – global picture Standardisation Stand Prep SDC Harm Memobust DW Data Matching Communication DI Data Validation Pillar Consistency DW Demetra+ CORE NAPS Argus+ SDMX Quality EGR Decentralised Access ESSnet Sponsorship VIP 23 -25 May 2011 MSIS Conference Common tools SICON New Diss

Conclusions n Many initiative on going – bottom up approach initially n Hard to maintain coherence between different layers, scope n Big hope with the Sponsorship on standardisation n Need for an Enterprise Architecture at ESS level 23 -25 May 2011 MSIS Conference BACK