Quality Assessments of Statistical Production Processes in Eurostat

Quality Assessments of Statistical Production Processes in Eurostat Pierre Ecochard and Małgorzata Szczęsna pierre. ecochard@ec. europa. eu, malgorzata. szczesna@ec. europa. eu

Quality assessment in the ESS: 2005 -2007 “Over the last three years (2005 -2007), how many of your statistical processes have been monitored using: ”

Quality assessment in the ESS: 2008 -2010 “Over the next three years (2008 -2010), how many of your statistical processes do you plan to monitor using: ”

Audits and self-assessments in the ESS Number of NSIs conducting: Audits 5 NSIs SAs 9 NSIs 7 NSIs 10 NSIs

Background The European Statistics Code of Practice The Eurostat Quality Assurance Framework Documentation Measurement • Quality reports • Quality indics • Process descriptions • Process variables etc Evaluation Conformity Quality Assessments Labelling

What is Eurostat Quality Assessment? A systematic review and evaluation of all stages of the statistical production process with the use of the DESAPbased Checklist Follow-up User needs Data collection Validation Confidentiality Documentation Dissemination IT conditions; Management, planning and legislation; Staff, work conditions and competence

Assessment Outputs (1) Assessment Report

Assessment Outputs (2) Accessibility and clarity Assessment Diagram Overall quality of metadata 5 Comparability across countries Relevance 4 User satisfaction 3 Comparability 2 1 0 Comparability over time Accuracy Overall accuracy Coherence General coherence Timeliness of final publication

Assessment Outputs (3) Highlight of good practice

Categories of Eurostat assessments Process 2 Process 1 Self. Assessment Process 5 Process 4 Process 3 Supported Self-Assessment Peer Review Characteristics: - Periodicity - Legal Basis - Output - ESTAT intervention Rolling Review Similiarity: DESAP-based Checklist, outputs Difference: extent of external interventation in a review

Office-wide implementation plan n Approach piloted in two domains in 2007 n Most of the statistical process will benefit from an assessment within a three-year period 2008 -2010 n 33 reviews planned for 2008: – – 14 Supported Self-Assessments 14 Self-Assessments 4 Rolling Reviews 1 Peer Review n Follow-up report to Eurostat management by the end of 2008

Benefits of quality assessment For production teams: n An opportunity for a systematic analysis of the production process n Identify and prioritize improvement actions n Spread and benefit from the Good Practice For Eurostat: n Identify horizontal problematic issues n Foster standardisation of statistical processes n Support resource allocation, planning and programming n Show quality commitment

Feedback n The general workflow works well, the Checklist is flexible and the assessments are considered useful by domain managers n Heads of Unit should be involved earlier n The diagram can be a red herring: it should be used with caution n The ownership of the results should be made very clear n More assessment should involve an external expert

Keys for success in implementing quality assessments n Top management commitment n Middle management acceptance n Sound communication n Long term perspectives n Implementation and fine-tuning in pilot projects n Standardised use of methods n Clear responsibilities and ownership n Sufficient resources allocated for the assessments