Metadata and Quality Indicators Reuse for Quality reporting

Metadata and Quality Indicators Reuse for Quality reporting Work Session on Statistical Metadata (METIS) Giovanna Brancato, Marina Signore Istat Geneva, 10 -12 March 2010

Work Session on Statistical Metadata (Metis) Outline 1. Istat information System on Survey Documentation (SIDI/SIQual) 2. Istat Quality Reporting for external users 3. Metadata and Quality documentation reuse for Istat quality reporting 4. Metadata and quality reporting reuse for ESMS Standards Geneva, 10 -12 March 2010

Work Session on Statistical Metadata (Metis) Quality documentation and dissemination at Istat: SIDI and SIQuality report navigation system (intranet/internet) Process report Quality pilots Quality indicators Programming Management Systems Geneva, 10 -12 March 2010 SIQual S I Quality metadata D I Process metadata Statistical Yearbook

Work Session on Statistical Metadata (Metis) SIDI metadata Key concept Statistical production process: producing statistical information process aimed at primary survey: data collection from reporting units (households, businesses, institutions) including administrative sources surveys 225 documented primary surveys 156 active, 53 ceased, 16 suspended secondary study: processing of statistical data already available from previous primary surveys or secondary studies 101 documented secondary studies 81 active, 14 ceased, 6 suspended Geneva, 10 -12 March 2010

Work Session on Statistical Metadata (Metis) SIQual SIDI navigation environment Intranet: metadata and standard quality indicators functionalities for queries on organisational units (direction/service/unit) Internet: metadata SIQual Geneva, 10 -12 March 2010

SIDI Standard Quality Indicators Work Session on Statistical Metadata (Metis) Completeness status Group Coverage and Unit nonresponse Coding Editing and Imputation Revision policy * Timeliness and punctuality * Comparability * Coherence btw provisional and final * Coherence with other sources % of active processes with indicators 67. 20 23. 10 28. 60 7. 50 92. 24 60. 99 17. 78 12. 30 Resources and Costs * Including Primary Surveys and Secondary Studies Geneva, 10 -12 March 2010 65. 98

Work Session on Statistical Metadata (Metis) Istat quality reporting for external users Tools Ø Short methodological notes (SMNs) Ø Process reports (PR) Ø Quality reports (QR) Geneva, 10 -12 March 2010

Work Session on Statistical Metadata (Metis) Istat short methodological note (SMN) Use Compact reports included in the Statistical Yearbook Structure General conceptual metadata: observed issues, analysis unit Methodological process: reporting unit, periodicity; EU regulation, sampling design, data collection modes Quality metadata: synthesis of the main activities carried out to prevent, monitor and evaluate nonsampling errors (unit nonresponse and measurement error), validation techniques Dissemination of provisional and final results: Timeliness, geographical and sector level of aggregation Languages Italian (English) Geneva, 10 -12 March 2010

Work Session on Statistical Metadata (Metis) SIDI metadata reuse in the SMN Example SIDI documentation on observed issues: Attendance at museums and art exhibitions Chronic diseases Cinema attendance Friendship relations Games and other leisure activities Health conditions Housing conditions … Geneva, 10 -12 March 2010

Work Session on Statistical Metadata (Metis) SIDI metadata reuse in the SMN Example SIDI documentation on the activities to prevent unit non response: - Survey presentation letter signed by Istat President - Guarantees on statistical confidentiality - Interviewer identification badge - Description of survey objectives by interviewers - Telephone contacts to make an appointment for the interview - Establishing a toll free line or telephone number for further explanations Compacted description in the short methodological note: “Activities for preventing unit nonresponse” Geneva, 10 -12 March 2010

Work Session on Statistical Metadata (Metis) SIDI quality indicators reuse in SMN Examples SIDI documentation on Timeliness: single value average over a period Geneva, 10 -12 March 2010

Work Session on Statistical Metadata (Metis) Istat process report (PR) Use Standard process documentation to be included in publications on survey results as methodological annex. General structure - Introduction - Objectives - Survey design - Data capturing - Data pre-treatment - Processing - Documentation Geneva, 10 -12 March 2010

Work Session on Statistical Metadata (Metis) SIDI metadata reuse in PR The report is pre-filled in with the information drawn from SIDI An editor window allows to adjust and modify the text to turn it in a fluent language Geneva, 10 -12 March 2010

Work Session on Statistical Metadata (Metis) Istat quality report (QR) Use It is a short report, including a subset of quality indicators, to be disseminated together with press releases on Istat web site Structure It is organised into the following sections: - Relevance - Accuracy - Timeliness and punctuality - Clarity and accessibility - Coherence and comparability Geneva, 10 -12 March 2010

Work Session on Statistical Metadata (Metis) Indicators included in the quality report (QR) Qualtiy indicators - Sampling error (confidence interval or coefficient of variation) - Weighted/Unweighted Response rate [respondent units / (eligible units and unresolved units)] - Net imputation Rate [imputations from blank to nonblank values/imputable values] - Modification Rate [changes from nonblank to nonblank values/imputable values] - Mean Revision (size and direction of the revisions) - Absolute mean revision (average size of revisions) - Timeliness (of provisional/final results, of data sources) - Comparability (length of the series) - Coherence between provisional and final estimates (difference or relative difference btw provisional and final estimates) Geneva, 10 -12 March 2010

Work Session on Statistical Metadata (Metis) SIDI metadata and quality indicators reuse in QR • Implementation phase • Development of the product quality report integrated to SIDI management environment • Limited reuse of metadata • Reuse of standard quality indicators following the short methodological approach • Need of developing new quality indicators and revising some existing ones Geneva, 10 -12 March 2010

Work Session on Statistical Metadata (Metis) ESMS requirements Definition Statistical metadata to be used by Member States for documenting statistical data to Eurostat metadata quality indicators (ongoing work) harmonisation within Eurostat, OECD, IMF Uses Comparisons and evaluations Metadata and quality information for the users Geneva, 10 -12 March 2010

Work Session on Statistical Metadata (Metis) Nature of ESMS metadata Organization: Institutional mandate, confidentiality legislation, release policy, quality management Data set and variables: classification system, sector coverage, statistical concepts and definitions, statistical unit and population, reference area, time coverage, base period, unit of measure, reference period Dissemination frequency and formats Eurostat quality components: relevance, accuracy, … Statistical processing Standard quality indicators [Simulation: compilation of ESMS template with metadata and quality indicators for a survey ~ 60%] Geneva, 10 -12 March 2010

Work Session on Statistical Metadata (Metis) Issues on ESMS Type of data and frequency of data collection Organisational data: univocally defined throughout the institution Metadata: stable over time Quality indicators: survey edition Standard quality indicators Harmonisation: among different Member States, among different EU regulations Eurostat Handbook/Standards for Quality Reports (2009) Priority setting: identification of a minimum set Use: additional indicators upon request for expert users Aggregation: survey edition, annual averages (for monthly surveys) Dissemination: summary measures, ranges of variations, graphical representation Geneva, 10 -12 March 2010

Metadata and quality indicators reuse for quality reporting Thank you for your attention ! Geneva, 10 -12 March 2010

Work Session on Statistical Metadata (Metis) SIDI main features Metadata documentation standards Thesaura: lists of standard items to be used to document process activities and quality control actions Content: Topics of the survey, analysis units, questionnaire Process: Reporting unit (sources of the secondary study), survey design, data collection, data transformation, data processing Quality: Activities carried out to prevent, monitor and evaluate survey errors Metadata qualitative descriptions: free notes supporting standard metadata items Geneva, 10 -12 March 2010

Work Session on Statistical Metadata (Metis) Istat process report (PR) Use Standard process documentation to be included in publications on survey results as methodological annex. General structure - Introduction (name, period, description) - Objectives (information needs and periodicity, issues) - Survey design (population, sampling, reporting unit and frame, questionnaire design and testing, pilot survey) - Data capturing (data collection mode, field operations, sensitive questions, unit nonresponse monitoring, mandatory participation, response error) - Data pre-treatment (coding, manual revision, data entry, editing and imputation) - Processing (estimation, validation, use of generalised software) - Documentation (list of relevant documents) Geneva, 10 -12 March 2010

Work Session on Statistical Metadata (Metis) Istat quality report (QR) Use It is a short report, including a subset of quality indicators, to be disseminated together with press releases on Istat web site Structure It is organised into the following sections: - Relevance (regulations and use of the data) - Accuracy (methodological context, sampling error, nonsampling error, revision policy) - Timeliness and punctuality (production of preliminary results, timeliness) - Clarity and accessibility (links to data and documentation) - Coherence and comparability (national or international regulations and manuals, over time data comparability) Geneva, 10 -12 March 2010