Session 5 Plans and Preparations United Nations Technical

  • Slides: 58
Download presentation
Session 5 Plans and Preparations United Nations Technical Meeting on Use of Technology in

Session 5 Plans and Preparations United Nations Technical Meeting on Use of Technology in Population and Housing Censuses Amman, Jordan 28 t h - 1 st December 2016 Statistics Estonia Diana Beltadze

2007 -2011 Pre-enumeration activities 31. 12. 200931. 03. 2010 Pilot Timetable of Population and

2007 -2011 Pre-enumeration activities 31. 12. 200931. 03. 2010 Pilot Timetable of Population and Housing Census 2011 31. 12. 201131. 03. 2012 Enumeration 2012 - 2013 Post-enumeration activities Testing various aspects of census plan and procedures UNECE meeting in Amman 2016

UNECE meeting in Amman 2016

UNECE meeting in Amman 2016

Training n A 5 -day training for district chiefs and supervisors (15+132); n A

Training n A 5 -day training for district chiefs and supervisors (15+132); n A 5 -day training for census takers (2, 200). UNECE meeting in Amman 2016

Goals of the information system ü ü ü Universal software useable in all surveys

Goals of the information system ü ü ü Universal software useable in all surveys Improve the coverage of studies Improve quality of raw data Improve efficiency of field work management Reduce data collection cost Increase speed of post-collection data processing UNECE meeting in Amman 2016

Description of the applications ü Questionnaire definition tool ü Enumerators application ü E-respondent application

Description of the applications ü Questionnaire definition tool ü Enumerators application ü E-respondent application ü Management application ü E-respondent map application ü Management map application ü Enumerators map application UNECE meeting in Amman 2016

High level view of the involved applications IT-architecture, security, hardware and Census UNECE meeting

High level view of the involved applications IT-architecture, security, hardware and Census UNECE meeting in Amman 2016

E-census data collection process IT-architecture, security, hardware and Census UNECE meeting in Amman 2016

E-census data collection process IT-architecture, security, hardware and Census UNECE meeting in Amman 2016

Live environment ü ü ü ü 6 physical (12 virtual) Oracle Weblogic Java application

Live environment ü ü ü ü 6 physical (12 virtual) Oracle Weblogic Java application servers 2 Oracle physical database servers Apache load balancers ESRI GIS server Check Point firewall and VPN software SAS and VAIS data processing servers Suse Linux Enterprise and MS Windows operating systems used in servers ü MS Windows XP operating system used for enumerators laptops ü Mainly Hewlet Packard hardware used IT-architecture, security, hardware and Census UNECE meeting in Amman 2016

UNECE meeting in Amman 2016

UNECE meeting in Amman 2016

External resources ü ü META database Active Directory Statistical database GIS database UNECE meeting

External resources ü ü META database Active Directory Statistical database GIS database UNECE meeting in Amman 2016

IT security ü IT security analyzes (2008) ü IT security analyzes and complete testing

IT security ü IT security analyzes (2008) ü IT security analyzes and complete testing (2010) UNECE meeting in Amman 2016

IT related risks ü ü ü Data leakage Attacks against IT solutions Load tolerance

IT related risks ü ü ü Data leakage Attacks against IT solutions Load tolerance is not sufficient Failures of software Data loss IT-architecture, security, hardware and Census UNECE meeting in Amman 2016

Data protection ü Operations with sensitive data are logged ü Separate log tables and

Data protection ü Operations with sensitive data are logged ü Separate log tables and special rights in management system and enumerators application ü Hard drive encryption in enumerator’s computer ü Database encryption and high availability IT-architecture, security, hardware and Census UNECE meeting in Amman 2016

Hardware and software purchases ü Cost efficiency analyzes (laptops) ü Load tolerance evaluation (servers)

Hardware and software purchases ü Cost efficiency analyzes (laptops) ü Load tolerance evaluation (servers) ü Later use of hardware and software IT-architecture, security, hardware and Census UNECE meeting in Amman 2016

Preparation of hardware ü Servers and other peripherals were shipped in April 2011, laptops

Preparation of hardware ü Servers and other peripherals were shipped in April 2011, laptops in Sept-Oct 2011 ü External partners were used ü Secure VPN connection between two physical locations created ü 1979 code lines used in personalization script. ü About 26, 4 computers per day was prepared on a assembly line ü Live sample downloaded during the fieldwork UNECE meeting in Amman 2016

IT-architecture, security, hardware and Census UNECE meeting in Amman 2016

IT-architecture, security, hardware and Census UNECE meeting in Amman 2016

Load tolerance testing ü Tests were done between 2009 -2012 ü Test requirements (Live

Load tolerance testing ü Tests were done between 2009 -2012 ü Test requirements (Live environment) ü E-respondents application - 2500 parallel sessions ü Enumerators application – 2000 parallel sessions ü Emulating robots used ü Testing lasted about 9 months (Live environment) UNECE meeting in Amman 2016

Load tolerance testing process ü >2500 sessions - personal computers ü 2500 - 5000

Load tolerance testing process ü >2500 sessions - personal computers ü 2500 - 5000 sessions – Amazon cloud ü 5000 - 7500 sessions – Estonian Information Systems Authority’s servers IT-architecture, security, hardware and Census UNECE meeting in Amman 2016

IT support UNECE meeting in Amman 2016

IT support UNECE meeting in Amman 2016

IT-architecture, security, hardware and Census UNECE meeting in Amman 2016

IT-architecture, security, hardware and Census UNECE meeting in Amman 2016

The average time for fulfilling personal questionnaire IT-architecture, security, hardware and Census UNECE meeting

The average time for fulfilling personal questionnaire IT-architecture, security, hardware and Census UNECE meeting in Amman 2016

Maailmarekordi sünd läbi rahvaja eluruumide loenduse e. UNECE meeting in Amman 2016

Maailmarekordi sünd läbi rahvaja eluruumide loenduse e. UNECE meeting in Amman 2016

Authentication channels UNECE meeting in Amman 2016

Authentication channels UNECE meeting in Amman 2016

Hourly statistics IT-architecture, security, hardware and Census UNECE meeting in Amman 2016

Hourly statistics IT-architecture, security, hardware and Census UNECE meeting in Amman 2016

Fears and problems during E-census ü E-census environment becomes unavailable ü System load is

Fears and problems during E-census ü E-census environment becomes unavailable ü System load is too high, people quit ü Enumerators computers don’t get ready in time ü Security problems stop E-census ü Problems downloading Live sample ü Training fails ü Fieldworks are getting delayed UNECE meeting in Amman 2016

Good decisions ü ü ü Information system for general service Sizable load tolerance tests

Good decisions ü ü ü Information system for general service Sizable load tolerance tests Buying out laptops used by the enumerators Outsourcing information system maintenance Tachometer for indicating system load UNECE meeting in Amman 2016

Subsequent knowledge ü ü It’s all about testing Pilots as real as possible Hire

Subsequent knowledge ü ü It’s all about testing Pilots as real as possible Hire more people for longer period Buy experience, because there’s no time to experiment and learn UNECE meeting in Amman 2016

Critical competences ü System architect ü A moderator between software development and maintenance team

Critical competences ü System architect ü A moderator between software development and maintenance team IT-architecture, security, hardware and Census UNECE meeting in Amman 2016

Hardware Infrastructure n Oracle Weblogic J 2 EE server LIVE cluster n 2 physical

Hardware Infrastructure n Oracle Weblogic J 2 EE server LIVE cluster n 2 physical server nodes (Linux) n Load balancer (Apache) n Oracle Database server n 1 physical server (Linux) n Cold standby (Linux) n Interviewers computers (Acer One netbooks) n Mobile internet connection modems n Portable GPS receivers UNECE meeting in Amman 2016

Preparation of hardware Interviewers computers n Windows XP operating system with last updates n

Preparation of hardware Interviewers computers n Windows XP operating system with last updates n Forbidden hardware for non working purposes n Webcam n USB/MMC storage n Wi-Fi/LAN UNECE meeting in Amman 2016

Preparation of hardware (continues) Interviewers computers n Forbidden functionality for non working purposes n

Preparation of hardware (continues) Interviewers computers n Forbidden functionality for non working purposes n Local computer and registry n BIOS password protection, denial of bootable removable storage n Web traffic limitations n Forbidden access to control panel functionality n Forbidden access to hidden files n Hibernate in case of closing the lid UNECE meeting in Amman 2016

Preparation of hardware (continues) Interviewers computers n Forbidden functionality for non working purposes n

Preparation of hardware (continues) Interviewers computers n Forbidden functionality for non working purposes n Domain Policy n Screen saver with password prompt in case of an idle n Interviewer only specific VPN rules UNECE meeting in Amman 2016

Preparation of hardware (continues) Interviewers computers n Forbidden functionality for non working purposes n

Preparation of hardware (continues) Interviewers computers n Forbidden functionality for non working purposes n Domain Policy n Screen saver with password prompt in case of an idle n Interviewer only specific VPN rules UNECE meeting in Amman 2016

Preparation of hardware (continues) Interviewers computers n Personalization n True Crypts default password change

Preparation of hardware (continues) Interviewers computers n Personalization n True Crypts default password change n Delicate data folders access rights n Hiding the delicate data folders n Verification n True Crypt and Active Directory passwords n VPN connection n Interviewers application load, update and synchronization UNECE meeting in Amman 2016

Security of data n Passwords n One password for everything n Predefined and unchangeable

Security of data n Passwords n One password for everything n Predefined and unchangeable passwords n True Crypt password changed and stored in the process of personalization n Active Directory authentication for Windows logon and web applications UNECE meeting in Amman 2016

Security of computers n Fully encrypted hard drive n HTTP/SFTP connection inside encrypted VPN

Security of computers n Fully encrypted hard drive n HTTP/SFTP connection inside encrypted VPN tunnel UNECE meeting in Amman 2016

Physical security n Servers n Access protected server rooms n Interviewers computer n Based

Physical security n Servers n Access protected server rooms n Interviewers computer n Based on interviewers training n n Handle personal passwords Protect computer in bad weather conditions Act in case of robbery or computer loss Handle unauthorized use of computer UNECE meeting in Amman 2016

Our partners IT-architecture, security, hardware and Census UNECE meeting in Amman 2016

Our partners IT-architecture, security, hardware and Census UNECE meeting in Amman 2016

During census spatial- and attribute data were renewed through enumerator’s map application Enumerator’s tasks:

During census spatial- and attribute data were renewed through enumerator’s map application Enumerator’s tasks: Renewing of existing buildings attributes - the type and condition of the building Add new houses to map and add their type and condition UNECE meeting in Amman 2016

The map data on Enumerator’s map application n Raster data (1: 1 - 1:

The map data on Enumerator’s map application n Raster data (1: 1 - 1: 25 000): n Ortophoto – from Estonian Land Board OR n Base Map – topographic data from Estonian Land Board, cartographic work from e. GEOStat project – Statistics Estonia UNECE meeting in Amman 2016

The map data on Enumerator’s map application n Vector data (1: 1 –. .

The map data on Enumerator’s map application n Vector data (1: 1 –. . . ) n District borders n Building numbers from ADS n Cadastral parcels (for agricultural census) n Administrative borders n Bus stops n Topographic data (roads, rivers, forests, etc. ) UNECE meeting in Amman 2016

Enumerator’s map application building data update n Data syncronisation n The XY coordinates of

Enumerator’s map application building data update n Data syncronisation n The XY coordinates of new buildings and the type and condition of buildings is sent through secure VPN with GPRS connection to the Statistics Estonia server. n This takes place together with the census data syncronisation UNECE meeting in Amman 2016

E-census map application n Every citizen can come to our web page and ecount

E-census map application n Every citizen can come to our web page and ecount themselves. n They search and choose their address from our database (ADS) and connect all their answers to ADS address point and XY coordinates n If we don’t have data on the citizen’s address, they can add their location on a web map and write their own address n The map is similar to enumerator’s map UNECE meeting in Amman 2016

Supervisory application’s map n The supervisors can see how their enumerators are working n

Supervisory application’s map n The supervisors can see how their enumerators are working n It is also displayed on a map n The map looks just like enumerator’s map UNECE meeting in Amman 2016

What are districts? n Districts are bordered areas on the Census enumerator’s map where

What are districts? n Districts are bordered areas on the Census enumerator’s map where he/she has to collect data. n Districts are NOT the areas by which the Census data are later being presented. n District borders must be easily detectable on the landscape to make the Census enumerator’s work faster and better. UNECE meeting Kristjan in Amman Roosild 2016

GIS technology used n Arc GIS Server for the Java Platform (Advanced/Standard/EDN) 10. 0.

GIS technology used n Arc GIS Server for the Java Platform (Advanced/Standard/EDN) 10. 0. 3 + REST API n Arc GIS Desktop (Info/Editor) 9. 3/10. 0 n Arc SDE n Arc GIS Viewer for Flex n Districting for Arc GIS n X tools Pro UNECE meeting in Amman 2016

e. GEOStat n e. GEOStat is the Estonian geostatistical database n Enables to collect,

e. GEOStat n e. GEOStat is the Estonian geostatistical database n Enables to collect, manage, process, analyse, visualise and maintain spatial data n We get data from Estonian Land Board: n n n Topographic data Address data Administrative and settlement units data Cadastrial information Ortophotos n Is related to data collection system VVIS: n Maps n Location info of buildings n Hierarchy of district division UNECE meeting in Amman 2016

Data for data collection system n Vector data and basemaps for enumerators desktop and

Data for data collection system n Vector data and basemaps for enumerators desktop and e. Census application n Address data n Address locators n Arc. GIS services UNECE meeting in Amman 2016

Recruiters map application UNECE meeting in Amman 2016

Recruiters map application UNECE meeting in Amman 2016

e. Census web map application UNECE meeting in Amman 2016

e. Census web map application UNECE meeting in Amman 2016

e. Census web map application UNECE meeting in Amman 2016

e. Census web map application UNECE meeting in Amman 2016

e. Census web map application UNECE meeting in Amman 2016

e. Census web map application UNECE meeting in Amman 2016

Enumerators desktop application UNECE meeting in Amman 2016

Enumerators desktop application UNECE meeting in Amman 2016

Management Map Application UNECE meeting in Amman 2016

Management Map Application UNECE meeting in Amman 2016

Address operator map application during data processing UNECE meeting in Amman 2016

Address operator map application during data processing UNECE meeting in Amman 2016

Conclusion n Statistics Estonia has built an electronic data collection environment n The system

Conclusion n Statistics Estonia has built an electronic data collection environment n The system supports the whole data collection process, including n Study form creation n Task division between collectors n CAPI module (filling in of study forms in collectors’ laptops) n CAWI module (public e-survey environment) n Preliminary data quality assurance module n Final testing 2 weeks in 2009 , LIVE launch in 3 months in 2010 during agricultural census UNECE meeting in Amman 2016

Thank You! UNECE meeting in Amman 2016

Thank You! UNECE meeting in Amman 2016