Scanning Technology and Its Application in Ethiopia Yakob

Scanning Technology and Its Application in Ethiopia Yakob Mudesir Deputy Director General Central Statistical Agency of Ethiopia E-mail: yakobm@ethionet. et Web: www. csa. gov. et October 2009 Kampala, Uganda

Outline: • Background • Scanning Technology • Requirements for Scanning • The Ethiopian Experience

Background § Population and Housing Census process is the largest data capturing exercise that a country can undertake. § It involves capturing of millions of forms § The Central Statistics Agency (CSA) started using old techniques like Punched Card Reader as early 1960’s. § Two Population and Housing Censuses have so far been conducted in Ethiopia using the traditional method of data capturing. § The first Population and Housing Census was carried out in 1984.

Background § During the 1984 Census: Ø Data capture was done on manual keyboard based entry using mainframe computer Ø FORMSPEC data entry system was used Ø It took more than 2 years to capture the data for about 42 million people. § In the case of the 1994 Census: Ø Data capture was again done on manual keyboard entry basis using PC’s Ø CENTRY data entry system (IMPS) was used

Background § It took about 18 months to capture the data for the population of about 53 million. § The entry work was done on 2 -shift basis § About 180 data entry clerks were involved § Around 90 Pc’s were used

Some Limitations of the Keyboard Manual Entry Method § Time consuming v Does not allow the availability of timely data v The data will be weaker in representing the current or existing situation § Subject to additional non-sampling errors v Human error due to manual keying v Due to the volume of the data, a 100% verification as in the case of sample surveys, is difficult.

Some Limitations of the Keyboard Manual Entry Method § Involves a great deal of human resource management. v Large number of data entry operators and equipment required

The need for alternative solution The need to have the census result right on time and the limitations discussed above forced the statistical offices to look for other alternatives. . specially related to large volume of data - The Emerging of Scanning Technology

The Scanning Technology § The Scanning Technology in general implements two basic techniques v Mark recognition, like the Optical Mark Reader (OMR) v Character recognition, like the Optical Character Recognition (OCR) and the Intelligent Character Recognition (ICR)

The Scanning Technology § OMR is the recognition of shaded marks (blobs) on the forms v The positioning of these blobs on a form determines the alphanumeric characters they represent § The character recognition is the recognition of alphanumeric characters on forms and they are of 2 types: v OCR which is the recognition of machine printed characters and. . v ICR which refers to the capture of handprinted characters from a form

Major Benefits of the Scanning Technology § Significant decrease in time required to capture the data § This helps to get timely data v Users’ need satisfied (policy makers, planners, researchers, etc. ) § Reduces the non sampling error § No need to worry to store millions of forms for possible future references v Scanning captures the whole content of a questionnaire in an electronic image format

The Process involved in Scanning Recognition & Extraction Scanning 1 1 35 33 1 2 07 1 01 2 E X P O R T

Requirements for Effective Scanning - Proper planning - On the DESIGN of the questionnaire - On PRINTING the questionnaire - On RECORDING answer - On Questionnaire HANDLING - On securing ADEQUATE SPACE for questionnaire movement Why? In order to minimize the rejection rate and increase the Recognition Rate

Requirements for Effective Scanning § Proper training v Both on Hardware and Software v This helps to “ own” the technology Ø Being able to use the technology after the departure of the trainers / technical advisors § A well organized space forms and data flow is required

Requirements for Effective Scanning - A Reliable Network System

Requirements for Effective Scanning - A WELL STRUCTURED SPACE FOR FILE FLOW Data Processing Center Receiving Registering the EA for Scanning Waiting Room Warehouse Registering the EA Received from the Field 1 4 3 5 Scanning Room Receiving the Questionnaires 2 6 Store Validation Room 7 8 Processing Room

Requirements for Effective Scanning Proper File Management BO X File recorder: Records the ID of the outgoing questionnaires Machine room box handler Checks the number, orientation and damage to questionnaires Filing Box handler Checks the village code on the box is the same as on the questionnaire Filing Box handler Returns the questioanires to the shelves Principal Supervisor File recorder: Records the ID No of the incomming questionnaires. Machine room box handler returns box to the File recorder Senior Supervisor Guillotine Machine operator Cuts the binding edge off all questionnaires in the village in one operation and places them on the scanner Stitching machine operator lifts paper off scanner and stitches in the bottom left corner Loos e Pape r Scanner Paper Handler Responsible for checking paper throughput through the scanner Scanner technician Ensures smooth running of the machine and assists with the paper handling Scanner Supervisor Loos e Pape r Computer operators (x 2) Responsible detecting errors in the scanning process

Requirements for Effective Scanning § Proper file management and care v Checking batch (EA) IDs and orientation of forms v Proper recording of the in-coming and outgoing questionnaires v Ensuring the EA code on each box is the same as the on the questionnaires v Close attention in detecting errors in the scanning process is required

The Ethiopian Experience

Experience Sharing § Study tour made in two African countries § Tanzania v To learn from their successes v Data capture of the 2002 Census of Tanzania was done in about 26 days v General report tables were produced within 3 months from the start of the scanning

Experience Sharing § Ghana v To learn from their difficulties v Data capture of the 2000 Census took about 6 months - ( forms from 29, 000 EAs) v 3 Scanners were used (Kodak, Fujitsu) > The larger scanner was Kodak 500 D > Speed: About 500 forms/min v Power failure was one of the major problems > Loss of some data occurred as a result > A large generator was installed to minimize the effect of the frequent power cut

Identification of the Technology § For scanning of the 2007 Census the Optical Mark Reader (OMR) technique was used § Scanning Technology to be used Photo. Scribe Series PS 900 Scanners DRS Scanning Technology product

Identification of the Technology DRS Photo Scribe Series PS 900 § High speed Imaging Mark Reader § Windows XP professional § CD R/WR drive § Network connectivity § A TFT monitor, Keyboard, mouse § Speed: up to 8, 500 forms / hour

Design and Printing of Forms § Types of the 2007 Census forms v Short questionnaires v Long questionnaires v Household Listing Forms v Summary Forms v Community Level Forms v Batch Header Form – Scanned to create EA database

Design and Printing of Forms Long Questionnaire

Design and Printing of Forms Control Database Form Summary Form

Pilot § Data from the Pilot Census successfully scanned (OMR), key-corrected, exported to text format, tabulated and tested. § One scanner (PS 900 Photo Scribe) was used to capture the pilot data § Technical experts from the DRS company assisted in capturing, validating and exporting the pilot data

Training v Hardware and Software training v 16 professionals trained v The training in general took about 7 working days v SOSKITW for Windows: - a DRS software package for scanning was used v Components of the SOSKITW Software : Ø SOSGen : - used to generate scanning decodes for completed OMR forms (How marks on forms are interpreted and stored) Ø SOSInp : - used to scan, validate and export scanned data.

Preparatory Activities § Equipment purchased and installed v 10 additional PS 900 i. M 2 DRS Scanners v 16 high capacity PC’s for key-correction § Census data processing work plan prepared v Recruitment of temporary staff v Staff training (scanning technology, CSPro) v Retrieval and organization of completed forms v Scanning and validation v Computer editing and tabulation (For each activity: duration and responsible body are indicated)

Preparatory Activities v. About 33 teams for registering and organizing forms are organized 3 persons assigned per team §Census data processing teams organized v Batch header database group v Scanning and validation team v. Shift supervisors v Two senior programmers responsible for the overall scanning process v Other sub-professional staff assigned v 4 batch header scanning technicians v 16 data validation workers

Preparatory Activities – The scanning room organized – An air conditioner for the scanning room installed – A high capacity automatic generator installed to ensure uninterrupted power supply

The Scanning Process § Organized forms taken from store to the waiting room § Batch header information printed and associated with its respective EA box § The existence of each EA verified § Checked EAs sent to the scanning room § Scanned forms are finally sent back to the stores § Scanned data are validated / key-corrected

The Scanning Process The actual scanning started mid July and the scanning work has been completed in November 2007

Data Cleaning / Computer Editing § Scanned, key-corrected and exported data § Batch Edit program based on edit specs provided by subject matter specialists developed and run on the data. § The software to be used in editing the data will be the Census and Survey Processing System (CSPro)

Future Plan Owning the Technology üTwo Professionals have been trained in England for two weeks on scanned document processing using Doc. XP in close collaboration with DRS üPrinting of the Questionnaire locally üQuestionnaire design üOur Professionals are working hard to process the upcoming welfare Monitoring Survey using scanning technology

THANK YOU