Towards a Research Agenda for Official Statistics Lars

Towards a Research Agenda for Official Statistics Lars Lyberg Statistics Sweden ETTS 2009 Brussels February 18 -20, 2009 Email: lars. lyberg@scb. se

The current scenario • An overwhelming number of initiatives • 150+ ESS R&D groups (advisory, coordination, executive boards, expert, working, organizing committees, round table, steering, technical, task forces) • Problems with priorities, finishing off, and implementation

Examples of initiatives • A mixture of traditional methodological areas and more visionary ideas • Traditional areas: confidentiality, nonresponse, small area estimation, harmonization, response burden • More visionary ideas: model-based statistics, combining data from different sources, more efficient collaboration models such as ESSnet, standardization and evaluation efforts such as rolling reviews, Code of Practice, LEG on Quality, the new Regulation

So what is lacking? • A concerted effort • A project prioritization process based on strategic goals for the ESS • Recognizing risks • Involving the stakeholders even more • Taking care of all links in the statistics production process (i. e. more on conceptualization, response process, measurement errors, quality control) • Systematic implementation and evaluation of best practices

Anything more that is lacking? • The systematic division of labour • • NSIs have similar challenges NIPS, CENEX, ESSnet, sharing results • Capacity building (ESS competence development) • Cooperation with universities and agencies world-wide • Standards

A dog’s breakfast

Having said that…. . • Admittedly the general conditions are difficult and vary a lot • The total openness about ESS problems is admirable • Considerable persistence and success in some work areas • Lots of work laid down on – – – Subject-matter Methodology Strengthening the role of official statistics Quality Stakeholder dialogue • Lots of recommendations provided

Let’s go back to the ESS basics • ESS shall provide reliable, comparable statistics for EU decision making, monitoring and evaluation • Eurostat shall ensure that methods, criteria and practices within ESS are as harmonized as possible and statistics as comparable as possible • NSIs and MSs shall provide national data and technical expertise and advice • Implicit is that production should be efficient

How central are these dimensions in ESS R&D work? • “Reliable” should probably read “accurate” – Small errors – Stable processes that deliver what they are supposed to – Important error sources left behind • Comparative – Hidden under the harmonization label – Generic to ESS but comes out as an “add on” • Efficient production – Central in many R&D projects

Accurate data • Do not leave any key processes behind • Develop and/or apply standard operating procedures (SOP) (quality assurance) • Validate process performance (quality control) • Allocate assurance and control resources with a risk management perspective • Quantify total survey error (TSE) • How should TSE be interpreted in a comparative setting? • Resources outside ESS: TSE workshop, other specialized networks and conferences, SOPs and guidelines

Comparative issues • What could and should be standardized (harmonized)? • How to achieve procedural equivalence? • How to adjust concepts so that we get a uniform meaning across countries? • How should differences in funding and methodology resources be handled? • How should national pride and conflicts of interest be handled? • Resources outside ESS: CSDI network, guidelines for comparative research published by U o Michigan and CSDI

Controlling Quality Level Main stakeholders Control instrument Measures Indicators Product User, client Product specs, SLA Frameworks, compliance, MSE, user surveys Process Survey designer Process variables, CBM, SOP, standards, checklists, verification Variation via control charts, other paradata analysis Organization NSI, owner, society Excellence models, ISO, Co. P, reviews, audits, selfassessments Scores, strong and weak points, user surveys, staff surveys

Hypothetical project priority matrix Selection criteria Significance ratings Project scoring Links to ESS’s strategic goals 10 x 1 Relates to key process 10 x 2 Risk of no useful result 10 x 3 Customer satisfaction impact 10 x 4 Data availability 9 x 5 Measurability 9 x 6 Cost vs revenue 8 x 7 Time until useable results are achieved 7 x 8 Resource availability 5 x 9

Matrix use • Project ideas submitted • Each project suggestion scored by 10 x 1+10 x 2+10 x 3+9 x 4+……. . 5 x 9 • Scoring is done by project management group • Rolling R&D plan • Projects executed in ranking order

Endnote • More search for and use of what is already available • Continue the work on simplified coordination • Collaborate globally • The new Regulation should be operationalized such as Article 10: European statistics shall be developed, produced and disseminated on the basis of uniform standards and of harmonized methods • Emphasize the comparative aspect of ESS • Implementing best practices