Improving efficiency by introducing macro editing in Statistics

Improving efficiency by introducing macro editing in Statistics New Zealand business performance surveys Paper by Tim Hawkes, presented by Emma Bentley May 2011

Background Suite of business performance (BP) surveys, many started around 2005 Quality review recommendations: • Reduce manual micro edits, cut processing costs • More emphasis needed on macro editing • ‘Big picture’ perspective needed for analysts New approach to editing developed for 2010 survey round 2

Previous editing approach Micro editing – time consuming (2 FTE!) • Validity edits • Consistency edits • Statistical edits Very few automatic edits, lots of manual intervention Macro editing – limited • Pressure of publication deadlines • Not standard across the suite of surveys 3

Recommended new approach BP analysts reviewed macro practices in-house and editing strategies internationally Decided to improve: • Automatic micro editing – Consistency edits – Validity edits • Macro editing – Create more time for this – Wider range of strategies available to BP team 4

Automatic micro editing Sum of components not equal to given totals Routing and validity edits A routing question with: Both Yes and No then the section is blank Auto edit to No Both Yes and No then the section is answered Auto edit to Yes Yes then section is blank Auto edit to No or leave for imputation 5

Macro editing Calculating estimates during the data collection process • Produce initial estimates when 50 -60% of target response rate achieved • Early detection of influential observations for validation • Problems with processing or estimation system can be identified and resolved early in production process • Issue if insufficient responses for estimation purposes • Analysts gain better understanding of estimation process 6

Macro editing • Top-down editing • E. g. drill down to industry and stratum level estimates • Identify unusual components, drill down more • May provide evidence for estimate, or identify error Use of sampling errors to identify suspicious estimates • Compare with expectations and previous results • Unusual responses that have effect on sample errors, may also effect estimates 7

Macro editing • Top contributor method • Ranked lists of top contributors e. g. for level estimates or movements • Compare lists to previous years • Graphical analysis • Not widely used by BP surveys • Processing checks • Monitoring template developed containing these checks 8

Implementation • Interactive training provided by Statistical Methods division • Analysts asked to identify potential questions suitable for automatic edits • Up to the lead analyst to implement the new approach and determine their macro editing techniques • BP team able to take more ownership of editing strategy now that the surveys are well established and topic knowledge developed 9

Review: Quality indicators for business performance surveys Quality Indicator Business Operations Survey 2009 2010 Average number of edit failures per unit R&D Survey 2008 2010 ICT Supply 2008 2010 Energy Use 2009 2010 5. 72 6. 66 5. 00* 4. 23* 1. 34 1. 00 2619 787 4000* 900* 1206 740 Clerical edits 4130 2497 Resource usage (staff hours) – Micro 337. 5 187. 5 375* 150* 375* 100* Resource usage (staff hours) - Macro 15 40. 5 112. 5* 150* 188** 375** * Estimate ** Estimate which also includes resource usage on imputation. No decline in quality detected 10

Review: Qualitative feedback • Generally positive • Reduction in amount of manual micro editing – popular with analysts! • Timing for incorporating changes a challenge • Set up of automatic editing, will be reusable in future • Saved time from micro editing taken up by implementation of other methodology changes 11

Review: Qualitative feedback • More time allocated to macro editing • Better macro strategies helped analysts better understand the processes and data, this in turn helps subsequent analysis stage • Conservative approach for first cycle of new editing strategy, room for more efficiencies • Improved ability to calculate quality indicators would help assess efficiencies in future • Reviews are useful! 12