NIH Peer Review of Small Business Applications Dana

NIH Peer Review of Small Business Applications Dana Jeffrey Plude, Ph. D. Acting Director Division of AIDS, Behavioral and Population Sciences 11 th NIH SBIR/STTR Conference July 2009 National Institutes of Health U. S. Department of Health and Human Services

NIH DUAL PEER REVIEW • Scientific Review Group Evaluate scientific and technical merit Recommend level of support, duration Does not make funding decisions • Advisory Council Reviews the review Funding recommendations to Institute/Center Evaluate priorities/relevance Advise on Policy • Funding decision made by Institute/Center 2

CSR Review Divisions AIDS, Behavioral Neuroscience, Development and Population Sciences and Aging Brain Disorders & Clinical Neuroscience Molecular, Cellular & Developmental Neuroscience Integrative, Functional & Cognitive Neuroscience Emerging Technologies & Training in Neuroscience Biology of Development & Aging Biobehavioral & Behavioral Processes Risk, Prevention& Health Behaviors Population Sciences & Epidemiology Healthcare Delivery & Methodologies AIDS & Related Research Physiological and Basic and Integrative Biological Sciences Pathological Sciences Biological Chemistry & Macromolecular Biophysics Bioengineering Sciences & Technologies Cell Biology Genes, Genomes & Genetics Oncology 1 – Basic Translational Interdisciplinary Molecular & Training Translational and Clinical Sciences Endocrinology, Metabolism, Nutrition & Reproductive Sciences Cardiovascular and Respiratory Sciences Immunology Surgical Sciences, Biomedical Imaging and Bioengineering Infectious Diseases & Microbiology Musculoskeletal, Oral And Skin Sciences Digestive, Kidney & Urological Systems Oncology 2 – Translational Clinical Vascular and Hematology

PEER REVIEW IN CSR • Scientific Review Groups/Study Sections/Review Committees/Special Emphasis Panels - Small Business applications are reviewed in Special Emphasis Panels. • Scientific Review Officer is responsible for the management of the meeting. • Chair and 10 - 24 members who are from academia and small businesses • 40 - 100 applications are reviewed at each study section meeting (face to face meeting). • CSR is experimenting with review platforms – video conferences, asynchronous extended discussion, and others. 4

What It Looks Like: Video Enhanced Discussions

What It Looks Like: Asynchronous Electronic Discussions

SCIENTIFIC REVIEW OFFICER • Performs administrative and technical review of applications to ensure completeness • Selects reviewers based on broad input • Assigns reviewers (generally at least two reviewers and one reader per application) • Manages review meeting/Designated Federal Official • Prepares Summary Statement • Provides requested information about review recommendations to Institutes/Centers and advisory councils 7

CRITERIA FOR SELECTION OF PEER REVIEWERS • • Demonstrated scientific expertise Doctoral degree or equivalent Mature judgment Work effectively in a groups Breadth of perspective Impartiality Interest in serving Adequate representation of women and minority Scientists • Geographic distribution • Small Business representation 8

REVIEW MEETINGS • Closed to the public • Introductions • SRO instructions: Confidentiality, conflict of interest, new NIH policies, procedures • Approximately 60% of the applications are Discussed, the ‘less competitive’ 40% of the applications are Not Discussed • Applications are discussed in ‘score order’ based on Preliminary “Overall Impact Score” given by the three assigned reviewers Note - any member can nominate an application for discussion 9

REVIEW MEETINGS (cont). Individual review of top 60% applications Conflicts excused Assigned reviewers/readers: preliminary scores, evaluations General discussion, including human subjects, vertebrate animals, biohazard issues Private assignment of priority score Discussion of budget, data sharing, resource sharing issues “Mock” Study Section video http: //cms. csr. nih. gov/Resourcesfor. Applicants/Inside the. NIHGrant. Review. Process. Video. htm 10

REVIEW LOGISTICS • Reviewers receive applications and assignments one to two months in advance of meeting. • Reviewers post preliminary scores and critiques on a secure web site in advance. • Reviewers may see other critiques only after their own are posted. • Are not aiming for consensus but outliers must explain their views • Critiques are modified in light of discussion. • SRO prepares Resume and Summary of Discussion. 11

2008: The Year of Peer Review Enhancing Peer Review “Fund the best science, by the best scientists, with the least administrative burden…” Elias Zerhouni, MD, Former Director, NIH

Recommendations Priority 1: Engage the Best Reviewers Increase Flexibility to Better Accommodate Reviewers Recruit Reviewers Acknowledge Reviewers more formally Compensate Reviewers Time and Effort Improve Review Quality with Training Priority 2: Improve the Quality & Transparency of Review Modify Rating System to Focus on Specific Review Criteria Align Summary Statement with Review Criteria Shorten and Align Application with Review Criteria Priority 4: Continuous Review of Peer Review Priority 3: Ensure Balanced & Fair Reviews Across Scientific Fields and Career Stages Support for Early Stage Investigators Review of Established Investigators Enhanced Review of Clinical Research Expand Awards Encouraging “Transformative Research” Reduce Need for Resubumissions

Amended Applications: To speed the funding of meritorious science and minimize reviewer burden: • As of January 25, 2009, all original new applications (i. e. , never submitted) and competing renewal applications will be permitted only a single amendment (A 1)

Balanced and Fair Reviews Across Career Stages and Scientific Fields • New Investigator: Not previously competed New Investigator: successfully as PD/PI for a significant NIH independent research award. • Early Stage Investigator: Within 10 years of completing Early Stage Investigator: terminal research degree or is within 10 years of completing medical residency (or the equivalent). • These Designations are IRRELEVANT for Small Business applications – only R 01 research grant applications are coded for New PI/ESI

Scoring To improve the transparency of the scoring process: • Score applications on five review criteria using a scale of 1 -9 • Preliminary overall impact score using 1 -9 scale • Discussed applications will receive an overall score from each eligible (i. e. , without conflicts of interest) panel member and these scores will be averaged to one decimal place, and multiplied by 10. The 81 possible priority scores will thus range from 10 -90. • Percentiles will be reported in whole numbers. • All applications will receive scores: Not Discussed applications will receive initial Discussed criterion scores from the three assigned reviewers

Scoring on a 9 -Point Scale • Goal: Improve the transparency of the scoring process: Score applications on five review criteria using a scale of 1 -9. • Preliminary overall impact score using 1 -9 scale. o Should not be the average of the criterion scores. o Not Discussed applications will receive initial criterion scores from the three assigned reviewers.

Scoring Descriptions Impact High Impact Moderate Impact Low Impact Score Descriptor 1 Exceptional 2 Outstanding 3 Excellent 4 Very Good 5 Good 6 Satisfactory 7 Fair 8 Marginal 9 Poor Strengths/Weaknesses

Scoring Expectations: • Reviewers participate in entire meeting . • To rank the applications in order of review based on preliminary score, all applications must be scored; applications with initial scores of 7 -9 are those unlikely to be discussed. • Once at the meeting, the full range of scores is available to assign an overall impact score for all discussed applications.

Scoring (cont. ) • Criterion scores are considered part of the critique and will not be discussed at the review meeting. They may be changed in the EDIT phase in IAR.

Clustering • Phase I, Phase II and Fast Track Applications will be clustered together (provided sufficient numbers). Phase I: FEASIBILITY Phase II: EFFICACY Fast Track: Both Components

Clustering of Clinical Applications • The goal of clustering clinical applications in review is to increase fairness in the process of reviewing clinical applications. A clinical application will be defined as human subjects research minus Exemption 4.

Order of Review • Goal: To discuss applications in order of average preliminary score. • Why? Concern: variation of scores during different times of the meeting. One recommendation was to recalibrate scores at the end of the meeting. • The Solution Recalibrate “dynamically” throughout meeting.

Order of Review • For calibration purposes… Begin meeting by discussing the best scored application (any activity code). o All Activity codes are clustered if f feasible (if at least 10 discussed, may include R 41/43, R 42/44 and Fast Track as groups that can be clustered).

Order of Review Summary • Discussion order is based on the average of the impact scores from assigned reviewers. • Final scores of discussed applications may differ from preliminary scores as re-calibration happens dynamically.

Not Discussed • Discuss ~ 40% of small business applications. • SRO will then ask if there any other applications that panel wishes to discuss. • The remaining applications will not be discussed (applications receive criterion scores only).

Final Scores • Discussed applications will receive an overall score from each eligible (i. e. , without conflicts of interest) panel member and these scores will be averaged to one decimal place, and multiplied by 10. The 81 possible priority scores will thus range from 10 -90. • Percentiles will be reported in whole numbers.

Summary Statements • Summary statement will be shorter and more focused. • Discussed applications will also have a summary of the panel’s discussion at the meeting. • ALL applications will be scored. Not discussed applications will receive criterion scores only.

Critiques To improve the quality of the critiques and to focus reviewer attention on the review criteria: • Electronic template for critiques that will prompt for strengths and weaknesses for each criterion. • Summary statement that will be shorter and more shorter focused. • Discussed applications only also will have a summary of the panel’s discussion at the meeting. discussion • In 2009, ALL applications will be scored • Not discussed applications will receive criterion scores only in addition to the reviewers’ critiques scores

Template-Based Critiques • Critique template contains a total of 18 boxes. Reviewers should provide text for only those criteria that are applicable. 1. Overall Impact 7. Resubmission 13. Biohazards 2. Significance 8. Renewal 14. Budget and Period of Support 3. Investigator(s) 9. Revision 15. Select Agents 4. Innovation 10. Protection of Human Subjects 16. Applications from Foreign Organization 5. Approach 11. Inclusion of Women, Minorities, and Children 17. Resource Sharing Plan 6. Environment 12. Vertebrate Animals 18. Additional Comments to Applicant

Template-Based Critiques • Goal: Write evaluative statements and to discourage summarizing the application. Comments should be in the form of bullet points or, if necessary, short narratives. Do not record scores on the critique template. The entire template is uploaded to IAR to become part of the summary statement. 1. Significance Strengths Weaknesses Please limit text to ¼ page

REVIEW CRITERIA Specifically tailored for small business applications: Significance Investigator(s) Innovation Approach Environment Overall Impact – weigh as appropriate, consider impact on scientific field 32

SIGNIFICANCE • Does the project have commercial potential to lead to a marketable product, process or service? • If the aims of the project are achieved, how will scientific knowledge or clinical practice be advanced? • Does this study address an important problem and what commercial and societal benefits might be derived from the proposed research? • What will be the effect of these studies on the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field? • Does the project lead to enabling technologies (e. g. , instrumentation, software) for further discoveries? • Will the technology have a competitive advantage over existing/alternate technologies that can meet market needs? 33

INVESTIGATORS • Is the PD/PI appropriately trained and capable of coordinating and managing the proposed project? • Are the investigators well suited to carry out this work and does the investigative team bring complementary and integrated expertise to the project (if applicable)? • Is the work proposed appropriate to the experience level of the PD/PI and other researchers, including consultants and subcontractors (if any)? • Are the relationships of the key personnel to the small business and to other institutions appropriate for the work proposed? 34

INNOVATION • Are the aims original and innovative? • Does the project challenge existing paradigms or clinical practice; address an innovative hypothesis or critical barrier to progress in the field? • Does the project develop or employ novel concepts, approaches, methodologies, tools, or technologies for this area? 35

APPROACH • Are the conceptual or clinical framework, design, methods, and analyses adequately developed, wellintegrated, and appropriate to the aims of the project? • Is the proposed plan a sound approach for establishing technical and commercial feasibility? • Does the applicant acknowledge potential problem areas and consider alternative strategies? • Are the milestones and evaluation procedures clearly described and appropriate? • For applications designating multiple PD/PIs, is the leadership approach, including the designated roles and responsibilities, governance and organizational structure consistent with and justifies by the aims of the project and the expertise of each of the PD/PIs? 36

ENVIRONMENT • Is there sufficient access to resources (e. g. , equipment, facilities)? • Does the scientific and technological environment in which the work will be done contribute to the probability of success? • Do the proposed studies benefit from unique features of the scientific environment, or subject populations, or employ useful collaborative arrangements? • Is there evidence of institutional support? 37

SPECIAL REVIEW CRITERIA • Resubmissions Responses to comments from the previous review Improvements in application • Phase II Applications Progress in Phase I – objectives met, feasibility demonstrated Commercialization Plan Commercial potential • Fast Track Applications Measurable goals/milestones Commercialization Plan Commercial potential Funding commitments/resources 38

ADDITIONAL REVIEW CRITERIA • Protection of human subjects from research risk • Inclusion of women, minorities and children • Care and use of vertebrate animals • Biohazards Problems in these areas will have a deleterious impact on the review outcome/score 39

OTHER REVIEW CONSIDERATIONS • • Budget – amount, duration Plans for Data Sharing if required Plans for Model Organism sharing Genome Wide Association Studies – plans for sharing data generated These do not contribute to the score 40

REVIEW OUTCOME • Not recommended for further consideration – rare, usually due to serious ethical or safety concerns • Deferral – also rare, study section does not have sufficient information to make a determination • Not Discussed 40% (approximately) • Scored 60% (approximately) 41

SUMMARY STATEMENT Not Discussed/Priority Score Overall Resume and Summary of Discussion if Discussed Essentially unedited critiques Budget recommendations Administrative Notes Available in Commons only to Principal Investigator 42

Example 1 – Phase II project PROTECTION OF HUMAN SUBJECTS UNACCEPTABLE RESUME AND SUMMARY OF DISCUSSION: The proposed project is intended to provide web-based education to and collaboration between parents and teachers of children with ADHD. The proposed program addresses a timely and important problem and if effective, this product would be a welcomed tool. The investigative team is exceptionally well qualified, Phase I successfully demonstrated the program’s feasibility and Phase II is well poised to refine the prototype. During discussion, however, the Committee noted several issues that limit the potential of the Phase II project. For example, the proposed assessment of the program’s effectiveness does not rely on parents and teachers of students with ADHD, and it does not assess the actual collaborative experiences of parent-teacher dyads that use it. Another concern centers on the plan to provide parents with clinical tools without proper guidance from mental health professionals. …There is also some question about the feasibility of creating a product that is relevant and useful for the full range of experience with ADHD that characterizes the target audience. 43

Example – Fast Track project RESUME AND SUMMARY OF DISCUSSION: The proposed project develops a brief screen for depression in antenatal and postpartum women. Although it is unclear how feasible it is to obtain a reliable screening tool that is shorter than the 10 -item Edinburgh Postpartum Depression Scale, if the project is successful the product would provide an important clinical tool. The project is especially significant because there is not an adequate scale to assess depressive symptoms among pregnant women who have physical symptoms. Using item response theory to develop the screener is innovative and will potentially result in a highly marketable product. The application outlines a strong commercialization plan. There are, however, some aspects of the proposed methodology that are questionable and there is concern that all of the members of the expert panel are directly involved in the project. Also Phase II lacks a gold standard assessment of depression for comparison with the final product. … These issues notwithstanding, the proposed product is innovative and, if successful, would be a significant contribution to the field. 44

ADVISORY COUNCIL/BOARD CONSIDERATION • Second required step in dual peer review process • Most applications considered in en bloc action; a few are specifically discussed (budget change, deferral) 45

QUESTIONS