Reengineering of Administrative Data Acquisition Maura Giacummo Eleonora

Reengineering of Administrative Data Acquisition Maura Giacummo, Eleonora Sibilio, Guido Drovandi, Paolo Giacomi (ISTAT - Italian National Istitute of Statistics) NTTS – Brussels 15 March 2017

ISTAT modernization process • ISTAT designed and implemented a new statistical production and organizational model, reviewing the previous model deeply and pervasively. • This innovation will result in the improvement and full exploitation of the direct statistical surveys results, by integrating them with administrative data • Administrative data : – must have a security policy of acquisition – must be immediately available to subsequent processes of linkage and dissemination

Administrative Data acquisition for statistical purposes has increased from 90 data sets in the 2009 to about 300 in the 2016. It involves about 100 source subsets 40 data providers for more than a terabyte of data, mainly containing personal data. design engineered processes are needed in order to automate and to manage the acquisition and distribution of administrative data.

A. D. acquisition and monitoring system ARCAM Engineered process for Administrative Data acquisition Strengths: security of data transmission - guaranteed through ISTAT standard technologies. different ways of data transmission - meet the suppliers’ needs centralized repository - ensure compliance with the legislation on the data treatment metadata allow the monitoring of the acquisition process

Administrative Data Acquisition INTERNET Provider Temporary Area Administrative Data are saved into a temporary storage area to allow the provider to modify them. INTRANET Permanent Repository Once a data set is moved into the permanent repository, changes are not allowed

Trasmit AD using a simple web application INTERNET Web Acquisition System Provider send files regardless of the size in multiple working sessions. File Data Provider Metadata ARCAM database Metadata File Data Temporary Area transferring files procedure Permanent Repository Files are divided into two different typologies: data and documentation. files. Chunks of fixed dimension (1 MB) sent to the server sequentially. If exception occurs or user closes the session it is possible to continue the transmission from the last stored chunk. INTRANET

Transmit AD using a simple web application XMLHttp Request browser feature - in order to manage huge files SHA-1 - used to compute the hash, to verify data transmission Data confidentiality - only users of same supplier operate on their data sets Different users can collaborate to upload same data but are not allowed to download them

Real-time monitoring and management data transmissions INTERNET Checks data supplies status reports Sends reminders to users not complying with the assigned deadlines Assigns transmission channel (HTTPS, FTP or WS) to data supplies Admin user Enables/disables data supplies for providers Enables/disables requesting users (end points of transmission) Rejects data supplies due to errors or inconsistencies ARCAM Database INTRANET Data download Setup the new annual data supplies timetable using previous timetables as templates. Metadata Delivery system Data Requesting user Permanent Repository

Database Outline Scheme The database managing both descriptive information of dataset and information relating to data acquisition Function User Profile Version File data Data Supply Source subsets Provider

Acquisition, monitoring, dissemination INTERNET Acquisition File data Metadata ARCAM Admin Monitoring Provider ARCAM Database Metadata Temporary Area Transferring files procedure A. D. download Requesting user INTRANET Delivery system Permanent Repository

Conclusions ARCAM was successfully used from 2016 About a thousand files have been acquired through ARCAM - 98 % via HTTPS channel - 2% via SFTP channel - from 150 users - belonging to 40 providers It is planned to extend the ARCAM domain to other type of data like: • Surveys data • Scanner Data