Developing a Survey Program Evaluation Process By Brenda

Developing a Survey Program Evaluation Process By Brenda G. Cox Battelle Memorial Institute and Nancy J. Kirkendall Energy Information Administration

n This research is being conducted under a subcontract Battelle has with Z, INC. from the Energy Information Administration (EIA). The views expressed are the authors’ and do not necessarily reflect the official position of EIA.

Performance Assessment Rating Tool (PART) n n n Encourages rigorous assessment of government programs Intended to boost the quality of government programs Provided the impetus for this investigation to design templates to evaluate government programs that involve a family of related surveys

Overview of Study Activities n n Development of a Survey Evaluation Template Development of a Survey Program Evaluation Template Test of Concept of the Survey Program Evaluation Template

Components of the Survey Evaluation Template n n n Survey Name Brief Description of the Survey Objectives Target Population Sampling Frame Sample Design

Components of the Survey Evaluation Template (cont. ) n n n Sample Allocation Data Collection Editing and Imputation Weighting Data Analysis Summary of Findings

Program Evaluation for a Family of Surveys: Step 1 n n Includes a summary evaluation of each component survey Survey Evaluation Template used to profile each survey

Program Evaluation for a Family of Surveys: Step 2 n n n Document which respondents report to which survey and data requests made of them Identify unnecessary burden on survey respondents across survey components and whether specific subpopulations are over burdened Determine if the survey program needs restructuring

Program Evaluation for a Family of Surveys: Step 2 (cont. ) n n n Determine how the data across surveys are used in analysis and presented to public Delineate the relationship across designs between data reporting and sample designs Determine if one or more surveys should be modified to build synergy across surveys

Program Evaluation for a Family of Surveys: Step 2 (cont. ) n n n Determine if some data items are being collected repetitively across surveys Examine the relationship between question wording and consistency from EIA’s and the respondents’ points of view Decide if there is a better design forms and instructions

Program Evaluation for a Family of Surveys: Step 2 (cont. ) n n n Ascertain how well confidentiality protection is applied—is confidentiality assured and response encouraged? Examine response rates across surveys—are particular subgroups prone to greater nonresponse? Determine if data are reasonably timely and understandable to users

Program Evaluation for a Family of Surveys: Step 3 n n n Document strengths of the program and desirable design elements already in place Specify areas for improvement and specific recommendations for improvement Suggest enhancements that would increase the utility of the data across surveys

Questions for the Committee n n Are there additional items that should be included in the Survey Evaluation Template? Do some items need to be modified? Is this overkill for evaluation of an individual survey? Any suggestions for important components that should be included in the Program Evaluation Template?