Overview of Census Evaluation Methods United Nations Statistics

Overview of Census Evaluation Methods United Nations Statistics Division United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Content q Objectives of evaluation of the quality of census data q Scope and organization of evaluation programme q Sources of census errors q Types of census errors q Methods for evaluation of errors United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Why do we need to evaluate the census? q The census is a huge operation comprised of many stages q It is not perfect and errors can and do occur at all stages of the census operation q Many countries have recognized the need to evaluate the overall quality of their census results and have employed various methods for evaluating census coverage as well as certain types of content error United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Aims of evaluation of data q To identify errors and find a solution to correct before releasing the final results q For unavoidable errors: n To provide users with a measure of the quality of census data to help them interpret the results q To serve as a basis for constructing the best estimate of census aggregates, such as total population q To provide suggestions and assist the plans for future censuses United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Planning a Census Evaluation Program q A census evaluation program should be developed as part of the overall census program and integrated with other census activities q Census errors can happen at all phases of the census operation, including questionnaire design, mapping, enumeration, data capture, coding, editing and imputation q Evaluation of data quality may have two parts: § § Preliminary evaluation will enable the identification of any problem areas that have not been previously detected More extensive evaluation should be undertaken on data quality to inform users about unavoidable problems and establishing best estimates United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Scope of evaluation q Census evaluation should include at least the followings: n n Analyze consistency in data and between variables Analyze evidence of age misreporting Analyze the quality of data collected in the census with appropriate methodology such as fertility, mortality, migration, educational and economic chacateristics Compare census data with independent data sources (surveys, registers) or previous censuses United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Institutional organization q Establishing the census evaluation team n Team should be trained in the evaluation techniques n Team should consist of members who have experience in census operations and analysis of census topicsdemography , education, housing, labor force, etc. n Team should have background knowledge of historical events and changes in population structure in the country n Team should collaborate with related research institutions United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Information on census processes q It is necessary that the evaluation team have a good understanding of the census process n n Which population groups were included/excluded Whether and how the data should be weighted Any known problems with the enumeration and/or data entry and editing processes If and how missing values have been edited o If there are no missing values on age and sex, the data has almost certainly been imputed o Imputed values should ideally be flagged o Editing rules for logical imputation, hot-decking or any other method that was used should be well understood and their effects carefully considered United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Main census phases Census Questionnaire Mapping Pilot census Enumeration Data processing Evaluation Improve the quality of data through quality assurance programme during each process Dissemination United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Quality assurance programme for controlling errors Measure quality Implement corrective action Identify problems Without such a programme, the census data may contain many errors which can severely diminish the usefulness of the results Identify causes of problems United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Overview of sources of errors Data processing Respondents Enumerators Questionnaire Errors can be many kinds from different sources Census maps United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Sources of errors q Errors in mapping and listing living quarters § Incomplete or inaccurate maps and/or listing § Inaccurate demarcation of enumeration areas o Overlapping or missing some areas o Unclear boundaries of enumeration areas United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Sources of errors q Errors in questionnaire design § Poorly designed questions or instructions § Poor sequencing of the questions § Poor communication between respondent and enumerator § Skip pattern- not clear or not placed appropriately United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Design 1: Separate form for every individual in the household United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Design 2: In the form of household list United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Sources of errors n Enumerator errors o Not fully explaining the meaning of the questions to the respondents or changing the wording of the questions o Making errors in recording the responses o Not asking some questions and creating unknown data n Respondent errors o Misunderstanding or deliberate misreporting o Proxy responses –when someone other than the person to whom the information pertains- provides the responses to the questions United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Sources of errors n Data entry errors : Invalid entries or mistakes in scanning and capturing data o Data capture system can ensure that the value of each field is within the permissible range of values for that item n Coding errors: giving wrong code to the information n Errors in editing/imputation : o The editing process changes or corrects invalid and inconsistent data by imputing non-responses or inconsistent information with plausible data o Any of these editing operations can introduce new errors United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Types of census errors q Coverage errors: n Errors in the count of persons or housing units resulting from cases having been “missed” or “counted erroneously” or “double counting” q Content errors: n Errors in the recorded characteristics of persons, households or housing units United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Coverage error q Omissions : Missing housing units, households, and/or persons during census enumeration q Erroneous inclusions : Housing units, households and persons enumerated when they should have not been enumerated in specific EA q Duplications: Occur when persons, households or housing units are counted more than once/ or captured more than ones United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Coverage error q Sources of coverage error: n Incomplete or inaccurate maps or address lists of enumeration areas, n Failure by enumerators to canvas all the units in their assignment areas or all the individuals in the units n Duplicate counting of some units or individuals, n Erroneous enumeration of certain categories of persons such as visitors or non-residents United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Coverage errors Gross error q Sum of duplications, erroneous inclusions and omissions Net error q Difference between over-counts and under-counts n Under-count if the number of omissions (“missing” people) exceeds the number of duplicates and erroneous enumerations n Over-count if total of the number of duplicates and erroneous enumerations exceeds the number of omissions United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Content errors q Content errors arise from the incorrect reporting or recording of the characteristics of persons, households and housing units q Every phase of census data collection and processing has the potential for introducing content errors into the census results n Enumerators, respondents, scanning, data capture, coding, editing/imputation United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Methods for the evaluation of census errors Single Source of Data (rely on the census being evaluated) q Demographic analysis n Consistency checks n Analysis of distribution or ratios of particular census topics Multiple Sources of Data q Non-matching studies n Demographic analysis using multiple census rounds n Comparison with administrative sources or existing surveys q Matching studies – not covered in this workshop n Post Enumeration Surveys n Record checks Source: U. S. Census Bureau, 1985. Evaluating Censuses of Population and Housing United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Multiple Sources of Data – Matching studies – Record checks q Census records are matched with a sample of records from official registration systems such as the vital registration system q The relevant respondents to the census questionnaire are traced to the time of the census q Sources include: n n n n Previous censuses Birth registration School enrollment National identification cards/registers Immigration registers Voter registration lists Health or social security records United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Multiple Sources of Data – Matching studies – Record checks q Both coverage and content errors can be measured through the above comparisons To evaluate coverage efficiently the following preconditions are essential: q A large and clearly-defined segment of census population (if not the entire population) should be covered by the registration system q The census and registration systems should be independent of one another q There should be sufficient information in the records to be able to match them with census respondents accurately United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Multiple Sources of Data – Matching studies – Record checks To evaluate content efficiently the following preconditions are essential: q The register system should contain relevant items covered in the census such as age, sex, education, relationship, marital status etc. q Definitions of variables should be identical between the census and the register United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Record checks – strengths and weaknesses q Can provide separate estimates of coverage and content error, net and gross error q With the right data, more characteristics can be evaluated compared to what can be done with non-matching studies q Calls for a high level of technical skill and registration system q Matching is expensive q In many countries, registration systems are not sufficiently complete for this method to be feasible United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Multiple Sources of Data – Matching studies – Post. Enumeration Surveys (PES) q A PES entails the complete re-enumeration of a representative sample of the population, which is then matched to the corresponding records from the census enumeration q PES can fulfill multiple objectives: n Assess the degree of coverage of the main enumeration n Assess implications of coverage error for usefulness of the data n Examine characteristics of those who have been missed by the main enumeration n Develop recommendations for design of future censuses and surveys United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Multiple Sources of Data – Matching studies – Post. Enumeration Surveys (PES) The PES should be independent of the census n A survey is conducted using a sampling frame independent of the census. Persons from this survey are then matched to the census to estimate the number of persons missed or erroneously enumerated in the census Advantages: n The results of a PES can be used to separately evaluate coverage vs. content error and net vs. gross error n Incorporates matching of individuals or units between the census and PES – this allows for a direct comparison of results n Its results are generally more reliable than those of the census q United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Multiple Sources of Data – Matching studies – Post. Enumeration Surveys (PES) Challenges: q Requires highly skilled field and professional staff q Matching is complex and costly q To be valid, the PES has to be conducted in a short time after the census to limit the complicating effects of population change, recall bias etc. United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Conclusion Ø Ø A number of methods exist for carrying out census evaluation In practice, many countries use a combination of such methods in order to improve the quality of evaluation programme United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

• , a number of methods exist for carrying out census evaluation. In practice, many countries use a combination of such methods in order to fully serve these obj United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

DATA VALIDATION-I Evaluation of editing and imputation United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Census processing overview q Steps of data processing depend on the technology used in general, the process covers the following steps: Preparat ion Data capture Editing/ Coding Imputati on Validation Processing control United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar Master file

Validation n It is a process of checking consistency in data after editing/imputation which may cause errors due to incorrect application of editing and imputation programme n To identify consistency errors, it is necessary to review some key aggregate tables for checking consistency among variables and with expected values/distribution to identify the unusual values United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Basic definitions q Editing: List of rules to determine invalid and inconsistent data q Imputation : The process of resolving problems concerning invalid or inconsistent data – and missing values- identified during editing Ø All records must respect a set of validity rules formulated to correct errors in order to be declared the records correct United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Some examples for invalid data-Myanmar pilot census questionnaire q Age n Equal to 99 o Instruction – if it is greater or equal to 98, write 98 n The age is written in one digit, such as 1 5 q Place of birth, place of usual residence and place of previous residence n n The code is not consistent with the code list The code is written in one or two digits o Instruction, the code is in three digits United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Some examples for inconsistent data-Myanmar pilot census questionnaire q Age and marital status n The age is below the minimum age of marriage and marital status is not single q Sex and children ever born alive n The sex of the enumerated person is male and children ever born alive or/and number of living children or/and number of dead children has a value q Last live birth and household deaths n There is an infant birth who is not alive, but no infant death registered in the household deaths United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Some examples for inconsistent data-Myanmar pilot census questionnaire q Sex, age and relationship to the head of household n n The sex of the head of household and spouse are same The age difference between the head of household and son/daughter is less than 15 or 14 q Age, the highest completed level of education and occupation n The age is 30, completed level is primary school and the person is secondary school teacher United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Assessing the performance of the imputation q Compare the distribution of the observed values with the distribution of the imputed values n if non-response and inconsistent data are distributed randomly, o no difference is expected between the distribution of the observed and the imputed values n If there are differences between the people who responded and those who did not or not giving accurate data o The imputed data should not follow the same distribution as the observed data United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Assessing the performance of the imputation q Compare the distribution of the observed values with the distribution of the all values including the imputed values n In general, imputed values should have a minimal effect on the distribution of the complete data o Unless the non-response rate is particularly high or the bias for certain characteristics is very strong United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Assessing the performance of the imputation q Classification of data for analysis of imputation n Consistent data: the values which meet with all editing rules Non-response : no value Inconsistent data: the values which failed at least one editing rule United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Assessing the performance of imputation Table 1: Non-response, edit failure and imputation rates Variables Non- Edit All response failure N N N Type of accommodation 22, 877 583 Number of rooms 22, 877 710 Number of bedrooms 22, 877 600 Central heating 22, 877 821 Total Non- Edit Imputed response N % Total Failure Imputed % % 583 2. 55 12 722 3. 10 <0. 1 3. 16 45 645 2. 62 0. 2 2. 82 821 3. 59 2. 55 3. 59 Source: England Wales, Office for National Statistics, 2011 Census: Item Edit and Imputation: Evaluation Report, June 2012 United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Assessing the performance of imputation Table 2: Distribution of bedrooms Number of bedrooms N 0 1 2 3 4 5 6 7 8 or more % (1) 62 2, 378 6, 097 9, 375 3, 279 809 166 39 27 (2) 0. 3 10. 7 27. 4 42. 2 14. 7 3. 6 0. 7 0. 2 0. 1 22, 23 Total 2 100 Difference Total Change (Imputed. Including (total Observed) imputed observed) N % % (6)=(1)+ (3) (4) (5)=(4)-(2) (3) (7) (8)=(7)-(2) 5 0. 8 0. 5 124 19. 2 8. 5 192 29. 8 2. 3 Source: England 228 35. 3 -6. 8 and Wales, 70 10. 9 -3. 9 Office for 19 2. 9 -0. 7 National 5 0. 8 0. 0 Statistics, 2011 1 0. 2 0. 0 Census: Item 1 0. 2 0. 0 Edit and Imputation: Evaluation 645 100 0. 0 Report, June 2012 United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Assessing the performance of the imputation Table 2: Distribution of bedrooms Difference Total Change Observed Imputed (total. Observed) Including imputed observed) responses Number of N % % bedrooms (1) (2) (3) (4) (5)=(4)-(2) (6)=(1)+(3) (7) (8)=(7)-(2) 0 62 0. 3 5 0. 8 0. 5 67 0. 3 0. 014 1 2, 378 10. 7 124 19. 2 8. 5 2, 502 10. 9 0. 240 2 6, 097 27. 4 192 29. 8 2. 3 6, 289 27. 5 0. 066 3 9, 375 42. 2 228 35. 3 -6. 8 9, 603 42. 0 -0. 192 4 3, 279 14. 7 70 10. 9 -3. 9 3, 349 14. 6 -0. 110 5 809 3. 6 19 2. 9 -0. 7 828 3. 6 -0. 020 6 166 0. 7 5 0. 8 0. 0 171 0. 7 0. 001 7 39 0. 2 1 0. 2 0. 0 40 0. 2 -0. 001 8 or more 27 0. 1 1 0. 2 0. 0 28 0. 1 0. 001 Total 22, 232 100 645 100 0. 0 22, 877 100 0. 000 Source: England Wales, Office for National Statistics, 2011 Census: Item Edit and Imputation: Evaluation Report, June 2012 United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Assessing the performance of imputation Maximum change United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Assessing the performance of imputation Maximum absolute percent change between the observed and final (imputed) distributions across all categories within each of the questions Source: England Wales, Office for National Statistics, 2011 Census: Item Edit and Imputation: Evaluation Report, June 2012 United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Assessing the performance of imputation Maximum absolute percent change between the observed and final (imputed) distributions across all categories within each of the questions Source: England Wales, Office for National Statistics, 2011 Census: Item Edit and Imputation: Evaluation Report, June 2012 United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Understanding data editing and potential errors Boundary of school age Boundary of working age Source: England Wales, Office for National Statistics, 2011 Census: Item Edit and Imputation: Evaluation Report, June 2012 United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Index of dissimilarity q To assess the degree of change induced by imputation on the initial distribution of variables Where; k : the categories of the variable f : the percentage distribution of the variable before imputation f * : the percentage distribution of the variable after imputation United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Index of dissimilarity § It assumes a 0 value when the two distributions before and after imputation are equal § It is greater than 0 when they are different; and reaches its maximum value of one when there is maximum dissimilarity between the two distributions when both are concentrated in one category which is different from each other United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Understanding data editing and potential errors Ø Analysis identified that an error in the hotdeck procedure reduced likelihood that women would be allocated a birth month of November or December Source: Estimation of fertility from the 2001 South Africa census data, Tom Moultrie & Rob Dorrington, Centre for Actuarial Research, University of Cape Town United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

Understanding data editing and potential errors ØData on deaths in the household – cases where age of deceased was hot-decked show different age pattern of mortality than cases that were not subject to imputation Source: Estimation of mortality using the 2001 South Africa census data, Rob Dorrington, Tom Moultrie and Ian Timaeus, Centre for Actuarial Research, University of Cape Town United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar

United Nations Workshop on Evaluation and Analysis of Census Data, 1 -12 December 2014, Nay Pyi Taw , Myanmar