Data quality and surveys for SEEAW Training Course

Data quality and surveys for SEEAW Training Course on Water Accounting Amman, Jordan 10 -13 March 2008 Michael Vardon United Nations Statistics Division 1

Outline • • Data quality Data sources Surveys options Surveys • Examples of surveys from Australia

Data quality • • • Relevance Accuracy Timeliness Accessibility Interpretability Coherence Statistics Canada – Quality Guidelines 4 th Edition 2003 http: //www. statcan. ca/english/freepub/12 -539 -XIE/12 -539 XIE 03001. pdf

Relevance “The relevance of statistical information reflects the degree to which it meets the real needs of clients. It is concerned with whether the available information sheds light on the issues that are important to users. Assessing relevance is subjective and depends upon the varying needs of users. The Agency’s challenge is to weigh and balance the conflicting needs of current and potential users to produce a program that goes as far as possible in satisfying the most important needs within given resource constraints. ” Statistics Canada – Quality Guidelines 4 th Edition 2003 p. 6

Accuracy “The accuracy of statistical information is the degree to which the information correctly describes the phenomena it was designed to measure. It is usually characterized in terms of error in statistical estimates and is traditionally decomposed into bias (systematic error) and variance (random error) components. It may also be described in terms of the major sources of error that potentially cause inaccuracy (e. g. , coverage, sampling, non response, response). ” Statistics Canada – Quality Guidelines 4 th Edition 2003 p. 6 -7

Timeliness “The timeliness of statistical information refers to the delay between the reference point (or the end of the reference period) to which the information pertains, and the date on which the information becomes available. It is typically involved in a trade-off against accuracy. The timeliness of information will influence its relevance. ” Statistics Canada – Quality Guidelines 4 th Edition 2003 p. 7

Accessibility “The accessibility of statistical information refers to the ease with which it can be obtained from the Agency. This includes the ease with which the existence of information can be ascertained, as well as the suitability of the form or medium through which the information can be accessed. The cost of the information may also be an aspect of accessibility for some users. ” Statistics Canada – Quality Guidelines 4 th Edition 2003 p. 7

Interpretability “The interpretability of statistical information reflects the availability of the supplementary information and metadata necessary to interpret and utilize it appropriately. This information normally includes the underlying concepts, variables and classifications used, the methodology of data collection and processing, and indications or measures of the accuracy of the statistical information. ” Statistics Canada – Quality Guidelines 4 th Edition 2003 p. 7

Coherence “The coherence of statistical information reflects the degree to which it can be successfully brought together with other statistical information within a broad analytic framework and over time. The use of standard concepts, classifications and target populations promotes coherence, as does the use of common methodology across surveys. Coherence does not necessarily imply full numerical consistency. ” Statistics Canada – Quality Guidelines 4 th Edition 2003 p. 7

Data sources • Administrative • e. g. Licensing data bases • Company reports • Many companies have reports that include information on the environment (e. g. water and energy use, pollution control measures) • Academic sources • Can be government or non-government • Surveys

Surveys Data • Physical data • Monetary data Survey options • Adding questions to existing surveys • Running specialized surveys Survey methodology • Mail-out/mail-back, telephone interview, personal interview • Mail-out/mail-back the cheapest and hence most often used • Regardless of methods it is critical to test forms in the field prior to the survey. • Sample design

Survey testing regime Good questionnaire design will help minimize response biases. • Forms should attractive and easy to understand fill in. • Develop a draft of form based on data requirements and knowledge of survey respondents, using terms and concepts familiar to survey designers and survey respondents. Test on a small number via personal interviews (10) • Develop 2 nd draft of form and test on a larger number of respondents (30 -60) via personal interviews and mail-out/mail-back (if this is the method to be used)

Specialized Surveys Advantage • Can collect a range of data need for the accounts as well as additional data to aid interpretation and understanding • Can select sample to represent total water use (not total economic activity) Disadvantage • Costly to develop and run

Adding questions to existing surveys Advantage • Direct link to the other data collected in the survey • Cheaper than running specialized surveys Disadvantage • Financial officers or business accountants who typically fill in business survey forms may not be familiar with environmental data • Sample selection may not be ideal

Sample selection may not be ideal For example, in business surveys sample may be selected to estimate total economic activity, not total water use or supply As such large water using industries (food and beverage manufacturing, metal manufacturing, paper manufacturing, electricity generation) may be under-represented in sample design. This problem can be overcome by adding sample in these industries

Example questionnaires from Australia Specialized surveys • Water Supply and Sewerage Survey • Water Survey (supplement to Agricultural Survey) Adding questions to existing surveys • Mining Survey • Manufacturing Survey • Agricultural Census All surveys mail-out /mail-back

ABS Surveys – Water Supply and Sewerage Industries Survey

ABS Surveys – Water Survey, Agricultural

ABS Surveys – Agricultural Census

ABS Surveys – Economic Activity Survey (Mining)

ABS Surveys – Economic Activity Survey (Manufacturing)

Intensive follow-up, 1 • Survey forms will often be filled in by a financial officer or business accountant who may not have easy access to the data required • Intensive follow-up required and should be targeted at large water using industries. For example: • Pulp and paper manufacturing, • Food and beverage manufacturing • Metals manufacturing • Electricity generation

Intensive follow-up, 2 Steps for intensive follow-up in Australia • 1 st Reminder letter • 2 nd Reminder letter • Phone call - important to carefully train staff. Also it may be possible to collect key data items over the phone • Final reminder • Notice of Direction (fill in the form or be summoned to court)

Data checking / input editing • Identification of outliers or improbable (that is very large or very small numbers) are identified for each industry (or region). • Double check very large numbers with the data supplier (often units are wrongly reported (e. g. m 3 reported not 1, 000 m 3 or acres not hectares) • Check blank cells – they may be zero

Contact details Michael Vardon Adviser on Environmental-Economic Accounting United Nations Statistics Division New York 10017 USA Room DC 2 1532 Phone: +1 917 367 5391 Fax: +1 917 363 1374 Email: vardon@un. org