NSF Business Analysis Overview of Merit Review Activities

NSF Business Analysis Overview of Merit Review Activities Presentation for Business and Operations Advisory Committee Joseph F. Burt, Director Division of Human Resource Management May 6, 2005 NSF BUSINESS ANALYSIS

Business Analysis Approach: Concurrent consideration of human capital and next-generation technology-enabled systems in an an analysis framed around the Agency’s core business processes 4 Core Business Processes: – Resource Allocation – setting the right priorities… – Merit Review – identifying people, ideas and tools with the greatest potential for impact… – Award Management and Oversight – the award cycle, beginning to end… – Knowledge Management – the right information, in the right places, at the right time… – Performance Assessment and Accountability – the highest standards of excellence and integrity… 4 Major Tasks: – Document Current Business Process Environment – Identify Business Process Opportunities for Improvement – Develop Business Cases and Implementation Plans for Each Opportunity NSF BUSINESS ANALYSIS 1

NSF management and staff have been integrally involved in all aspects of the Business Analysis 4 NSF Project Team - NSF project manager and leads for each of the technical areas of the project: business processes, human capital management, and technologies and tools 4 Booz Allen Hamilton (BAH) Team - Contractor staff working in partnership with NSF to conduct the Business Analysis 4 Integrated Process Teams - NSF-wide working groups focused on each of the core business processes – worked in concert with the NSF and BAH teams to document current environment and identify and develop opportunities for process improvement 4 NSF Management and Staff - Involved as resources for information, ideas, and reality checks – over 600 NSF staff have participated in the Business Analysis to date, through interviews, focus groups, or participation in working groups 4 Business Analysis Steering Committee - Composed of managers and staff from across NSF – serves as sounding board for the NSF/BAH project team, validates project plans, and vets all Business Analysis deliverables and products 4 Director, Deputy Director, and Assistant Directors – Provide high-level direction and oversight; involved at key decision points throughout the project NSF BUSINESS ANALYSIS 2

The Business Analysis has focused on NSF’s Merit Review process since the inception of the project in FY 02 Business Analysis Review of NSF’s Merit Review Process FY 2003 FY 2004 FY 2005 FY 2002 Start: 6/26/02 FY 2006 Baseline Analysis Applicant Survey Effective Practices Review Process Designs To-be Process Designs Implementation Planning Implementation (Timeframe TBD) Baseline Analysis and Applicant Survey 4 Identify key (mission-critical) processes to analyze 4 Map as-is process flows 4 Document accompanying metrics on cycle times, roles and responsibilities, etc. 4 Identify process bottlenecks and issues Effective Practices Research 4 Identify key organizations to research 4 Document comparative organizational information 4 Identify effective practices that NSF could incorporate in process designs 4 Develop an understanding of why specific strategies are successful Preliminary Design Options and Implementation Plans 4 Clarify and justify the costs, benefits, and impacts of adopting process designs 4 Identify risks and estimated costs of process designs 4 Identify high-level implementation issues 4 Survey applicant community (2003 NSF Applicant Survey) NSF BUSINESS ANALYSIS 3

Of the staff that support MR and AM&O, the percentage of time that staff spent in each of these activities varies, with Program Directors spending 55 percent on MR and 12 percent on AM&O Directorates Division Directors Program Directors Self-Reported % of Annual Working time DGA Program Assistants/ Lead Program Assistants Science Assistants Grants & Agreement Specialists MR Activities 29% 55% 66% 55% 10% AM&O Activities 8% 12% 23% 8% 84% 63% 33% 11% 37% 6% Other activities, including: • Develop scientific program • Research, compile, and produce reports • Strategic planning • Outreach and community relations • Personnel management, incl. recruitment • Oversee and provide advice regarding MR, AM&O • Participate in working groups/task forces • Participate in cross. Directorate and interagency planning and/or special initiative work • Attend meetings/ workshops, make invited presentations, or visit sites to convey program information • Research, compile, and produce reports • General planning • Normal "make the wheels turn" activities (daily email, phone calls, administrative staff oversight) • Research, compile, and produce statistics or reports • Arrange non-panel travel • Manage award funding obligations and budget spreadsheets • Coordinate project director quarterly meetings • Orient new staff (IPAs) • General administrative management (letters, correspondence not related to proposals) • Research, compile, and produce statistics or reports • Manage program website • Prepare external presentations • Manage and coordinate “nuggets” • Compliance checking and correcting compliance Issues • General administrative (misc. email, phone calls, queries, etc. ) • Attend meetings • Conduct special initiatives Source: MR/AM&O Process Workload Analysis, March 14, 2003, Booz Allen Hamilton Analysis NSF BUSINESS ANALYSIS 4

During 2003, the Business Analysis team surveyed over 6, 000 FY 02 NSF applicants regarding NSF’s merit review and award monitoring and oversight processes 4 Proposal Dwell Time – NSF currently processes >70% of proposals within 6 months; 74% of survey respondents would like to see their proposals processed in five months or less 4 Support during Proposal Preparation and Submission – Survey respondents believe that NSF does a better job of proposal preparation and submission support than other research funding agencies – 88% satisfied with ease of Fast. Lane use – 84% satisfied with quality of process information – 78% satisfied with help regarding proposal content – 82% satisfied with help regarding proposal submission 4 Fairness and Quality of Review Process – Survey respondents indicated that NSF could improve in terms of fairness and quality of merit review process – 35% somewhat or very dissatisfied with fairness of process – 35% somewhat or very dissatisfied with quality of process NSF BUSINESS ANALYSIS 5

In the analysis of the Merit Review process, three strategic areas of opportunity emerged that are being addressed in the current phase of the Business Analysis project 4 Workload of Program Directors and Grants Staff – Increase in the volume and complexity of proposals impacts the effectiveness of Program Directors in performing their merit review responsibilities – 48% increase in proposal workload from 2000 to 2004; workload survey data indicate that Program Directors spend 55% of their time on merit review activities 4 Increase in Programs that Require Cross-Directorate Coordination – The workload issue is compounded by the need for more formalized processes for programs that require cross-Directorate coordination – A sample of cross-Directorate programs indicates that the volume of proposals in such programs has increased by more than 50% over a three year period 4 Increasing Demand on Reviewer Community – The increase in volume of proposals is making recruiting and attracting reviewers more difficult NSF BUSINESS ANALYSIS 6

The Business Analysis has identified key opportunities for improvement that address the major workflow issues facing NSF’s merit review process 4 Workload of Program Directors and Grants Staff 4 Increase in Programs that Require Cross-Directorate Coordination 4 Approved: – Improve the overall utility of the Project Reporting System – Pilot an enhanced “customer service center” to address administrative inquiries – Coordinate program due dates on a two-year out basis with windows for cross-NSF and unsolicited proposals 4 Under Consideration: – Introduce a “fast track” solicitation process for existing programs – Provide PD flexibility to streamline review analysis NSF BUSINESS ANALYSIS 7

Key opportunities for workflow improvement in NSF’s merit review process (continued) 4 Increasing Demand on Reviewer Community 4 Under Consideration: – Develop a centralized, searchable database that incorporates PI and reviewer information – Establish standard training that would be available to reviewers prior to completing a review or serving on an NSF panel – Establish a centralized process for online acceptance and internal assessment of reviewer applications – Consider targeted outreach efforts for recruiting new reviewers NSF BUSINESS ANALYSIS 8