Evaluating Research Efficiency in EPA Phillip Juengst Accountability

Evaluating Research Efficiency in EPA Phillip Juengst, Accountability Team Leader Photo image area measures 2” H x 6. 93” W and can be masked by a collage strip of one, two or three images. The photo image area is located 3. 19” from left and 3. 81” from top of page. Each image used in collage should be reduced or cropped to a maximum of 2” high, stroked with a 1. 5 pt white frame and positioned edge-to-edge with accompanying images. Office of Research and Development June 6, 2008

Efficiency Measures, the PART, and Research • • • Where we were: EPA struggled to identify acceptable and meaningful efficiency measures for the PART Ultimate outcomes and cost effectiveness ($ per outcome) not easily measured for research Efficiency is important, but research focus should be: 1. Are we doing the right science? 2. Are we doing it well? 3. Are we making an impact? Office of Research and Development Office of Resources Management and Administration 1

Identifying Best Practices EPA identified significant variation across agencies Agency Examples of research PART efficiency measures approved by OMB NIH • Percent reallocation of laboratory resources based on external review • Continuing to develop an electronic administration system NSF • Percent of award decisions made available within 5/6 months • Percent of operational facilities with lost time of <10% DOE • Maintain direction to funding ratio <12% • Administrative costs as a percent of program costs • Average achieved operation time of facilities NOAA • Percent of grants awarded on time • Volume of data ingested annually • Cost per site characterization NASA • Cumulative percentage baseline cost overrun • Percentage of funding subject to peer review/competition DOD • Portion of funding chosen on the basis of merit review DOT • Ensure that at least 90% of projects are on-time/budget DOI • Avg cost per analytical result (adj. for inflation) is stable or declining over 5 -yr period USDA • Cost per R&D tool developed, delivered, and used *Bold indicates similar to EPA-proposed measure(s) not accepted by OMB Office of Research and Development Office of Resources Management and Administration 2

EPA Research Efficiency Measures • Measures vary by program, making it difficult to assess overall efficiency of research in a consistent and meaningful way – – – Average time (in days) to process research grant proposals Average time (in days) to respond to requests for technical support Number of peer reviewed publications per FTE Average cost to produce Integrated Science Assessments Percent variance from planned cost and schedule Office of Research and Development Office of Resources Management and Administration 3

Steps Toward Performance Improvement • In 2006, EPA began collaborating with OMB and our Board of Scientific Counselors on measures of outcomes and efficiency • Short-term (2007) results: – PART scores up to “Moderately Effective” – EPA green on PMA (research efficiency measures in place) • Broader vision: EPA engaged the National Academy of Sciences, OMB, other agencies, and private sector in a study of research efficiency Office of Research and Development Office of Resources Management and Administration 4

New Directions for Managing Research Performance • 2008 NAS study provides broad recommendations for all research agencies – Agencies should use expert panels to evaluate “investment efficiency, ” beginning with an evaluation of the relevance, quality, and performance of research – To improve program management, agencies should use quantitative “process efficiency” measures based on inputs, outputs, and some intermediate outcomes – OMB should have more consistent and equitable implementation of PART requirements Office of Research and Development Office of Resources Management and Administration 5

Implementing the NAS Recommendations • Agency dialog about how best to implement NAS recommendations – EPA/OMB led government-wide panel discussion on April 10 th to discuss how best to implement NAS recommendations – Approx. 100 representatives attended from more than 25 departments/ agencies, including OMB (Robert Shea and examiners from most branches), NAS, NIH, NSF, DOE, NASA, NIOSH, etc. – Summary and video being posted on EPA’s website, and shared with the research and evaluation communities http: //www. epa. gov/ord Office of Research and Development Office of Resources Management and Administration 6

Next Steps for EPA • Developing the right efficiency measures for research – Emerging interagency consensus that agencies should be able to “experiment” without risk to PART scores/PMA – EPA implementing NAS recommendations: 1. Expanding use of BOSC for assessing “investment efficiency” – How efficiently has the program invested and managed resources to achieve its long-term goals? 2. Exploring “% of budget allocated to direct science activities” – Endorsed by NAS when balanced by other measures – Drives administrative efficiencies at the lab level – Linked to long-term goals/EPA mission and balanced by existing quality/outcome measures Office of Research and Development Office of Resources Management and Administration 7

Next Steps for EPA and other Agencies – Using research efficiency measures to manage • • EPA undergoing major reorganization to achieve administrative efficiencies Long-range goal: efficiency improvements help maintain, if not increase, investments in direct science and research outcomes—better EPA decisions that improve human health and the environment – EPA will continue to lead and participate in Federal dialog to develop better measures that drive performance improvement • • Follow-up sessions (WREN? ) to discuss implementation of NAS recommendations OSTP’s Science of Science Policy Roadmap Office of Research and Development Office of Resources Management and Administration 8