New and Emerging Methods Maria Garcia and Ton

New and Emerging Methods Maria Garcia and Ton de Waal UN/ECE Work Session on Statistical Data Editing, 16 -18 May 2005, Ottawa

Introduction New methods of data editing and imputation ¡ Subdivided into 5 different themes: ¡ l l l Automatic editing Imputation E & I for demographic variables Selective editing Software

Invited Papers l l l WP 30: Methods and software for editing and imputation: recent advancements at ISTAT (ISTAT, Italy) WP 32: Using a quadratic programming approach to solve simultaneous ratio and balance edit problems (USCB, US) WP 31: Smoothing Imputations for categorical data in the linear regression paradigm (USCB, US)

Automatic editing: papers (1/2) Six papers: l l l WP 30: Methods and software for editing and imputation: recent advancements at ISTAT (ISTAT, Italy) WP 32: Using a quadratic programming approach to solve simultaneous ratio and balance edit problems (USCB, US) WP 33: Data editing and logic (Australia)

Automatic editing: papers (2/2) l l l WP 43: Automatic editing system for the case of two short-term business surveys (Republic of Slovenia) WP 44: A variable neighbourhood local search approach for the continuous data editing problem (Spain) WP 46: Implicit linear inequality edits and error localization in the SPEER edit system (USCB, US)

Automatic Editing: main developments Methods based on Fellegi-Holt model ¡ Developments at SORS l l ¡ General system combines error localization with outlier detection Plans for automation of implied edit generation Further improvements of SPEER l l Preprocessing program for generation of implied edits Improve error localization

Automatic Editing: main developments ¡ Framework of Fellegi-Holt theory in propositional logic l l Generation of implied edits framed as logical deduction Automatic tools that can potentially be used for finding minimal deletion set

Automatic Editing: main developments Methods based on some other approach ¡ Erroneous unit measures l ¡ Ratio and balance constraints l l ¡ Model as cluster analysis problem Hybrid ratio editing and quadratic programming Controlled rounding Error localization as a combinatorial optimization problem l l Continuous data Successful on very large data sets

Imputation: papers (1/2) Six papers: l l l WP 30: Methods and software for editing and imputation: recent advancements at ISTAT (ISTAT, Italy) WP 31: Smoothing imputations for categorical data in the linear regression paradigm (USCB, US) WP 36: Integrated modeling approach to imputation and discussion on imputation variance (Statistics Finland)

Imputation: papers (2/2) l l l WP 40: Imputation of data subject to balance and inequality restrictions using the truncated normal distribution (Statistics Netherlands) WP 41: On the imputation of categorical data subject to edit restrictions using loglinear models (Statistics Netherlands) WP 48: Improving imputation: the plan to examine count, status, vacancy and item imputation in the decennial census (USCB, US)

Imputation: main developments Model based methods ¡ Discrete Data l l ¡ Constrained loglinear model Linear regression model Continuous Data l Truncated normal distribution followed by MCEM

Imputation: main developments Implementation of imputation methods ¡ Use Bayesian networks for imputation of discrete data ¡ Development of QUIS for imputation of continuous data l l written in SAS uses EM algorithm, nearest neighbor, and MI

Imputation: main developments Implementation of imputation methods ¡ Integrated Modeling Approach (IMAI) l l ¡ Summary and analysis of principles of IMAI Estimation of imputation variance U. S. Decennial Census l l l Research on alternative imputation options Administrative records, model based imputation, CANCEIS, hot deck Development of a truth deck for evaluation

E & I for demographic variables: papers Three papers: l l l WP 30: Methods and software for editing and imputation: recent advancements at ISTAT (ISTAT, Italy) WP 35: Edit and imputation for the 2006 Canadian Census (Statistics Canada) WP 38: New procedures for editing and imputation of demographic variables (ISTAT, Italy)

E & I for demographic variables: main developments ¡ Further improvement of CANCEIS l l l ¡ capability of processing all census variables improved editing and imputation of alphanumeric, discrete, continuous and coded variables improved user interface Development of DIESIS l combined use of “data driven” approach (NIM) and “minimum change” approach (Fellegi-Holt)

E & I for demographic variables: main developments ¡ Development of DIESIS l l l Use of graph theory to improve quality of sequential imputation Optimization procedure to locate the household reference person New approach for selection of donors based on partitioning passed records into smaller subsets of similar characteristics ¡ search for donor records within the smaller clusters ¡

Selective editing: papers Two papers: l l WP 42: Evaluation of score functions for selective editing of annual structural business statistics (Statistics Netherlands) WP 45: An editing procedure for low pay data in the annual survey of hours and earning (Office for National Statistics, UK)

Selective editing: main developments Continued use and development of selective editing ¡ Evaluation of selective editing approaches ¡ l ¡ experiments with different sets of score functions Development of “hybrid editing” l l validate a sample of failed records use associated data to impute remaining records

Software: papers Four papers: l l WP 34: The transition from GEIS to BANFF (Statistics Canada) WP 37: Concepts, materials and IT modules for data editing of German statistics (Destatis, Germany) WP 39: SLICE 1. 5: a software framework for automatic edit and imputation (Statistics Netherlands) WP 47: Improving an edit and imputation system for the US Census of agriculture (NASS, US)

Software: main developments ¡ Flexibility l l ¡ ¡ Testing and implementation of the software Quality control measures l ¡ modules rather than large systems are developed standard statistical packages are used (SAS in BANFF and US Census of Agriculture) e. g. for (donor) imputation Integration of the edit and imputation software in entire production process l process chain: planning, data collection, edit and imputation

General points for discussion ¡ Are there any really new approaches? l l ¡ Develop new approaches or consolidate old approaches? l l ¡ new approaches extensions of existing ideas? new approaches combinations of old ones? development versus evaluation studies and testing prototype software versus implementation of production software Is our focus shifting? l l l from editing towards imputation? from development towards implementation? from computational aspects towards quality issues?

Automatic editing: points for discussion ¡ ¡ ¡ Can operations research techniques be combined with techniques from mathematical logic? What are the (dis)advantages of using SAT solvers when compare to direct integer programming methods? What is the quality of the imputations when editing data using the quadratic programming approach?

Automatic editing: points for discussion ¡ ¡ ¡ What is the quality of the solutions found by using the combinatorial optimization approach on real survey data? How fast is this approach on realistic data? Can finite mixture models be used for detection of other types of systematic errors? Should we invest on developing generic tools or software tools tailored to a particular application?

Automatic editing: points for discussion Are there any other types of surveys that are worth the effort of generating implied edits prior to error localization? ¡ What are the most cost-effective methods for edit/imputation in terms of resources, time, clerical intervention, quality of results? ¡

Imputation: points for discussion ¡ ¡ ¡ What are the (dis)advantages of using complex mathematical models for missing data imputation? Are these models too complex for survey practitioners? What are the expected computational difficulties of applying complex models to real survey data? What are the largest (most complex) surveys that can be imputed using these models?

Imputation: points for discussion What is the quality of the imputations carried out using model based methods for filling-in missing data? ¡ Can we compare the different imputation models? ¡

Imputation: points for discussion Can more guidelines for the IMAI process be developed? ¡ To what extent can we develop a systematic way of applying IMAI? ¡ Is imputation variance an important issue at the moment, or should we (still) focus on imputation bias? ¡

E & I for demographic variables: points for discussion Can CANCEIS/DIESIS be used for other data besides demographic census data? ¡ Can CANCEIS/DIESIS be further developed? ¡ Should we use a combination of edit and imputation methods or a single method for demographic variables? ¡

Selective editing: points for discussion Can selective editing be successfully applied to large/complex surveys? ¡ Can current methods for selective editing be further developed? ¡ Can a general theory for selective editing be developed? ¡ How promising is hybrid editing? ¡

Software: points for discussion Should we develop generic software or software tools for particular applications? ¡ How can we ensure the flexibility of software? ¡ Are the software tools fast enough for large/complex data sets? ¡ To what extent should we aim to automate the editing process? ¡