Data Capture Overview United Nations Statistics Division Regional

Data Capture Overview United Nations Statistics Division Regional Workshop on the 2010 World Programme on Population and Housing Censuses: International standards, contemporary technologies for census mapping and data processing Minsk, Belarus, 8 -12 December 2008

Overview of Presentation q Definition of data capture q Methods of data capture: - Different Methods - Advantages and disadvantages q Issues to consider Regional Workshop on the 2010 World Programme on Population and Housing Censuses: International standards, contemporary technologies for census mapping and data processing Minsk, Belarus, 8 -12 December 2008

What’s Data Capture? “Data capture is the system used to convert the information obtained in the census to a format that can be interpreted by a computer. ” Source: United Nations Principles and Recommendations for Population and Housing Censuses, Rev. 2, p. 68. Regional Workshop on the 2010 World Programme on Population and Housing Censuses: International standards, contemporary technologies for census mapping and data processing Minsk, Belarus, 8 -12 December 2008

Data Capture Methods 1) Keyboard data entry 2) Optical mark recognition/reading (OMR) 3) Optical character recognition/intelligent character recognition (OCR/ICR) 4) Personal digital assistant (PDA) 5) Internet q q Advantages/disadvantages/costs/impacts at both data capture and later stages Combination of more than one of the above methods Regional Workshop on the 2010 World Programme on Population and Housing Censuses: International standards, contemporary technologies for census mapping and data processing Minsk, Belarus, 8 -12 December 2008

Keyboard Data Entry q Response codes from census form are manually entered into computers q Sophisticated version involves computer assisted key entry where operator selects a response from options displayed on the screen q Use of method based on time and cost considerations, and feasibility to implement more sophisticated technology q Method also used to process textual responses into classification categories Regional Workshop on the 2010 World Programme on Population and Housing Censuses: International standards, contemporary technologies for census mapping and data processing Minsk, Belarus, 8 -12 December 2008

Advantages and Disadvantages of Keyboard Data Entry Advantages q q q Method requires simple software systems and low -end computing hardware Less costly (depending on the costs of manpower) There will be a large number of PCs available for other uses after census Disadvantages q q Requires more staff Task takes much longer time to complete than with automated data entry Potential for errors during data entry Standardization of operations is difficult as performance may be individually dependant Regional Workshop on the 2010 World Programme on Population and Housing Censuses: International standards, contemporary technologies for census mapping and data processing Minsk, Belarus, 8 -12 December 2008

Data Capture Technologies q Imaging and intelligent character recognition offer great potential and benefits for data capture q Use of technology for data capture should be to enhance effective and efficient data capture and not for technology’s sake q Awareness of long lead times and technology infrastructure required for successful implementation of intelligent character recognition Regional Workshop on the 2010 World Programme on Population and Housing Censuses: International standards, contemporary technologies for census mapping and data processing Minsk, Belarus, 8 -12 December 2008

Optical Mark Recognition/Reading (OMR) q OMR is a form-scanning method whereby responses are read into a computer without a keyboard q OMR technology reads responses to “tick-box” type questions on specially designed paper q Only presence or absence of a mark is detected by the machine q The scanned responses are transformed into codes q Handwritten responses must be manually entered or coded using computer-assisted methods Regional Workshop on the 2010 World Programme on Population and Housing Censuses: International standards, contemporary technologies for census mapping and data processing Minsk, Belarus, 8 -12 December 2008

Advantages and Disadvantages of OMR Advantages q Improved data accuracy q Data capture faster than keyboard data entry q Equipment is relatively inexpensive q Relatively simple to install and run q A well-established technology that’s been used in many countries Disadvantages q Restrictions as to form design q Restrictions on type of paper and ink q Precision required in printing process/cutting of sheets q Response boxes should be correctly marked with appropriate pen or pencil q Won’t capture textual responses Regional Workshop on the 2010 World Programme on Population and Housing Censuses: International standards, contemporary technologies for census mapping and data processing Minsk, Belarus, 8 -12 December 2008

Optical Character Recognition (OCR)/ Intelligent Character Recognition (ICR) q OCR and ICR combine scanning and character recognition technology to scan the whole form and interpret the responses q OCR technology recognizes machine-printed characters only q ICR technology reads both machine-printed and handwritten responses in specific locations of the page and transforms the responses into codes q For OCR, handwritten responses must be manually entered or coded using computer-assisted methods Regional Workshop on the 2010 World Programme on Population and Housing Censuses: International standards, contemporary technologies for census mapping and data processing Minsk, Belarus, 8 -12 December 2008

Advantages of OCR/ICR q Form design is not as stringent as for OMR q Processing time can be reduced due to automated nature of the process q Allow for digital filing of questionnaires resulting in efficiency of storage and retrieval of questionnaires for future use q Some handwritten responses can be automatically coded thereby improving data quality Regional Workshop on the 2010 World Programme on Population and Housing Censuses: International standards, contemporary technologies for census mapping and data processing Minsk, Belarus, 8 -12 December 2008

Disadvantages of OCR/ICR q Higher costs of equipment (sophisticated hardware/software required) q High calibre IT staff required to support the system q Handwriting on census forms be as close as possible to the model handwriting to avoid recognition error q Possibility for error during character substitution which would affect data quality q Tuning of recognition engine to accurately recognize characters is critical with trade-off between quality and cost Regional Workshop on the 2010 World Programme on Population and Housing Censuses: International standards, contemporary technologies for census mapping and data processing Minsk, Belarus, 8 -12 December 2008

Personal Digital Assistant (PDA) q Contents of the census form are stored onto the PDA so that the questions appear sequentially on the screen q Data are entered into a hand-held computer instead of onto a paper census form q Data are then electronically transmitted to an NSO database for further processing Regional Workshop on the 2010 World Programme on Population and Housing Censuses: International standards, contemporary technologies for census mapping and data processing Minsk, Belarus, 8 -12 December 2008

Advantages and Disadvantages of use of the PDA Advantages q Instant data capturing at the point of collection, reducing manual input errors q q q Immediate data validation, reducing re-verifications at later stage Time effective with real time logical validation rules, reducing logical errors Faster processing of census information leading to timely availability of results Disadvantages q Setting up of process may take a long time as it requires extensive testing q Requires that enumerators have ability to use the device which may require administering a test q Requires intensive training of enumerators on use of device (training is more complicated) q Need to recharge the battery which could run out during enumeration q Possibility of equipment failure Regional Workshop on the 2010 World Programme on Population and Housing Censuses: International standards, contemporary technologies for census mapping and data processing Minsk, Belarus, 8 -12 December 2008

Internet-based Data Collection Use of the Internet for census data collection is growing q - However, the method is always complementary to other more established methods q Like with PDAs, the on-line form is not a downloadable version of the paper form q Use of this method requires a password in order to access and fill in the form q Development of the internet system for data collection is generally outsourced for lack of in-house expertise Regional Workshop on the 2010 World Programme on Population and Housing Censuses: International standards, contemporary technologies for census mapping and data processing Minsk, Belarus, 8 -12 December 2008

Advantages/Disadvantages of use of the Internet Advantages q Reduced resources necessary form handling and data capture q Better opportunity to enumerate difficult to reach and to enumerate geographic area and population groups q Automatic filtering of irrelevant questions q Better quality data due to inbuilt interactive verification mechanism q Faster availability of census results through simplified data entry and editing Disadvantages q q q Requires that respondents have a computer with Internet access Management of responses can be problematic, e. g. , that households have responded once and only once Requires high security system to ensure safe transfer of data Need to build parallel processing system as not everyone will use the Internet Requires mechanism to check for omitted and duplicate submissions Is costly and requires a lot of resources for setting up and adequately test the system Regional Workshop on the 2010 World Programme on Population and Housing Censuses: International standards, contemporary technologies for census mapping and data processing Minsk, Belarus, 8 -12 December 2008

Issues to Consider in Choosing a Method q Method to use is dependant on national circumstances q Choice of method should be part of the overall strategic objective of the census in terms of timeliness, accuracy and cost q Choice of processing system and technology to use need to be established early in census cycle q Enough time is required to test and implement the system q When imaging technology is used for data capture, extensive testing is required well in advance of the census q Possibility to outsource when the required expertise is not available in-house Regional Workshop on the 2010 World Programme on Population and Housing Censuses: International standards, contemporary technologies for census mapping and data processing Minsk, Belarus, 8 -12 December 2008

Issues to consider (cont. ) q Extensive testing of the system is also critical when data collection is either by PDA or via the Internet q Design and paper quality of census form should be linked to method of data capture q When imaging technology is to be used, adequate training of enumerators on how to properly fill in the forms is crucial Regional Workshop on the 2010 World Programme on Population and Housing Censuses: International standards, contemporary technologies for census mapping and data processing Minsk, Belarus, 8 -12 December 2008

Thank you Regional Workshop on the 2010 World Programme on Population and Housing Censuses: International standards, contemporary technologies for census mapping and data processing Minsk, Belarus, 8 -12 December 2008