NSERC DG Advices YannGal Guhneuc yanngael gueheneucpolytmtl ca

NSERC DG Advices Yann-Gaël Guéhéneuc yann-gael. gueheneuc@polytmtl. ca Version 0. 6. 2 2016/05/21 Département de génie informatique et de génie logiciel This work is licensed under a Creative Commons Attribution-Non. Commercial. Share. Alike 3. 0 Unported License

Questions: they are all welcome at yann-gael. gueheneuc@polymtl. ca 2/82

Disclaimer: I cannot be held responsible for the failure or success of your applications, were you to follow or not these advices 3/82

Disclaimer: NSERC does not endorse in all or in part the content or the form of these slides in any ways 4/82

NSERC DG Advices n Each NSERC DG application is evaluated according to 4 criteria and 6 merit indicators – Excellence of the researcher – Merit of the proposal – Contribution to the training HQP – Cost of research 5/82

NSERC DG Advices n Each NSERC DG application is evaluated according to 4 criteria and 6 merit indicators – Exceptional – Outstanding – Very strong – Strong – Moderate – Insufficient 6/82

NSERC DG Advices n How are these criteria rated by the reviewers using the indicators? n How to ease the reviewers’ jobs? And, how to be possibly more successful? 7/82

Outline n Process in a nutshell n – Candidate’s point of view – Internal reviewer’s point of view – – – • Off-line work • Competition week n Funding decisions – In a nutshell – Bins – ER vs. ECR n Criteria and indicators – – Advices Introduction Excellence of the researcher Merit of the proposal Contribution to the training HQP CCV (Form 100) Application (Form 101) Conclusion n Further readings n “Values” of criteria “Meanings” of indicators NSERC rating form My own rating form 8/82

Outline n Process in a nutshell n – Candidate’s point of view – Internal reviewer’s point of view – – – • Off-line work • Competition week n Funding decisions – In a nutshell – Bins – ER vs. ECR n Criteria and indicators – – Advices Introduction Excellence of the researcher Merit of the proposal Contribution to the training HQP CCV (Form 100) Application (Form 101) Conclusion n Further readings n “Values” of criteria “Meanings” of indicators NSERC rating form My own rating form 9/82

Process in a Nutshell n From the candidate’s point of view – August 1 st: submission of Notice of Intent • No. I, formerly Form 180 – November 1 st: final submission of • CCV (formerly Form 100) • Application (formerly Form 101) • Publication samples – March/April: announcement of the results 10/82

Process in a Nutshell n Each application is reviewed by – 5 colleagues: first and second internal reviewers and 3 readers – 0 to 5 external reviewers n From the internal reviewer’s point of view – Two main parts • Off-line work, e-mails/readings • Competition week in Ottawa 11/82

Process in a Nutshell n Off-line work – August 27 th: reception of all the submissions • In 2012, 322 submissions • In 2014, 324 submissions • In 2015, 327 submissions – September 7 th: ratings (expertise levels and conflicts) of all the submissions • High, Medium, Low, Very Low, Conflict, X (language) 12/82

Process in a Nutshell n Off-line work – September 24 th: final choice of the 1 st internal reviewers for each applications • In 2012, 14 applications as 1 st internal reviewer, as 2 nd internal reviewer, 17 as reader = 46 15 • In 2014, 9 applications as 1 st internal reviewer, as 2 nd internal reviewer, 28 as reader = 39 2 – But was session chair… • In 2015, 13 applications as 1 st internal reviewer, as 2 nd internal reviewer, 29 as reader = 49 7 13/82

Process in a Nutshell n Off-line work – October 5 th: choice by the 1 st internal reviewer of 5 external referees • In 2012, 14 applications = 70 • In 2014, 9 applications = 45 • In 2015, 13 applications = 65 • May include referees suggested by the candidate but may also replace all of them 14/82

Process in a Nutshell n Off-line work – October 22 nd: ratings of applications from other evaluation groups • In 2012, 1 application • In 2014, 0 applications – 39 applications to evaluate for joint reviews or not • In 2015, 0 applications 15/82

Process in a Nutshell n Off-line work – Early December: final list of readings • In 2012, 47 applications • In 2014, 39 applications • In 2015, 49 applications 16/82

Process in a Nutshell n Off-line work – January/February: reception of the reports from the external referees • In 2012, 123 reports • In 2014, 128 reports • In 2015, 128 reports – February 18 th to 22 nd: competition week in Ottawa during which each application is discussed and rated 17/82

Make it easier Process in a Nutshell for the reviewers n Off-line work – In 2012 (and every year…), a lot of work! • • 322 submissions 47 applications (including joint publications) 70 referees 123 referee reports 18/82

Process in a Nutshell n Competition week – February 18 th to 22 nd: competition week in Ottawa during which each application is discussed and rated – 5 days • Very intense, demanding, and tiring 19/82

Process in a Nutshell n Competition day – Starts at 8: 30 am – Divides into • 31 15 -minute slots • 2 15 -minute breaks • 1 45 -minute lunch – Ends at 5: 15 pm 20/82

Process in a Nutshell n Competition slot – In a 15 -minute slot, the ratings of an application are chosen by the reviewers – Or the application is “deferred”, to be rediscussed at the end of the day • In 2012, 1 application • In 2014, 4 applications • In 2015, 3 applications 21/82

Process in a Nutshell n Competition slot – 1 st internal reviewer gives ratings with justifications, which must be facts in the forms • Summarises external reviews – 2 nd internal reviewers contrasts, supports, adds missing facts from the forms – The readers complement or challenge ratings given by 1 st and 2 nd internal reviewers, must be supported by facts from the forms 22/82

Not exactly the Process in a Nutshell NSERC criteria n Competition slot – 1 st internal reviewer gives ratings with justifications, which must be facts in the forms • A typical presentation follow this pattern – Candidate: career, funding, visibility, publications, HPQ record, planned training – Proposal: context, lacks, characteristics (Incremental? Applicable? Feasible? ) – External: summary of the referees' reviews, summary of the provided contributions then, the reviewer would give his ratings 23/82

Process in a Nutshell n Competition slot – Session chair keeps the time strictly – Session chairs makes sure that any discussion sticks to the facts 24/82

Process in a Nutshell n Competition slot – Ratings are anonymous • Secret electronic vote • Session chair announce results – Ratings are consensual • If reviewers/readers strongly disagree, the application will be re-discussed at the end of the day – I did not see any strong debates: mostly 1 st and 2 nd internal reviewers agreed, backed-up by the readers – Some facts were sometimes highlighted and ratings were changed accordingly 25/82

Process in a Nutshell n Competition slot – Any criteria rated as moderate or insufficient receives comments from the committee, reflecting the reviewers’ consensus • NSERC provided typical comments, for example: “The Evaluation Group was of the opinion that the applicant did not take advantage of the available space to provide compelling information on how the proposed problems could be solved. The applicant is encouraged to situate his/her work and contributions more carefully within the context of the [subject]. ” 26/82

Outline n Process in a Nutshell n – Candidate’s point of view – Internal reviewer’s point of view – – – • Off-line work • Competition week n Funding decisions – In a nutshell – Bins – ER vs. ECR n Criteria and indicators – – Advices Introduction Excellence of the researcher Merit of the proposal Contribution to the training HQP CCV (Form 100) Application (Form 101) Conclusion n Further readings n “Values” of criteria “Meanings” of indicators NSERC rating form My own rating form 27/82

Funding Decisions n In a nutshell – Each proposal is rated by the reviewers secretly after the discussions – The medians of the ratings are used for criteria – For example • • Excellence of researcher: {S, S, M, M, M}, rating is M Merit of the proposal: {V, V, S, S, M}, rating is S Impact of HQP: {V, S, S, S, M}, rating is S The application rating is therefore {M, S, S} 28/82

Funding Decisions n Bins – The numeric “values” of the ratings are “added” • For example, {M, S, S} 2+3+3 = 8 – The application is placed into one of 16 bins – The bins are labelled A through to P and correspond numerically to 18 down to 3 29/82

Funding Decisions n Bins – Bins A and P are uniquely mapped to {E, E, E} and {I, I, I} while other bins contain a mix of numerically equivalent ratings, e. g. , {V, S, M} is in the same bin as {S, S, S} and {M, S, V} • For example, the application rated {M, S, S} is in K – Not all applications in a bin are funded: {S, S, S} may be funded while {M, S, V} is not • Because of the moderate indicator for the first criteria – Cut-off point depends on year 30/82

Funding Decisions n ER vs. ECR – Candidates are divided into • ER: established researchers, who already applied (funded? ) to NSERC DG • ECR: early-career researchers, who are “within two years of the start date of an NSERC eligible position” – ECR are funded one bin “lower” (better) than ER 31/82

Outline n Process in a Nutshell n – Candidate’s point of view – Internal reviewer’s point of view – – – • Off-line work • Competition week n Funding decisions – In a nutshell – Bins – ER vs. ECR n Criteria and indicators – – Advices Introduction Excellence of the researcher Merit of the proposal Contribution to the training HQP CCV (Form 100) Application (Form 101) Conclusion n Further readings n “Values” of criteria “Meanings” of indicators NSERC rating form My own rating form 32/82

Criteria and Indicators n “Values” of criteria – Excellence of the researcher – Merit of the proposal – Contribution to the training HQP – Cost of research 33/82

Criteria and Indicators n “Values” of criteria – Excellence of the researcher • Knowledge, expertise and experience • Quality of contributions to, and impact on, research areas in the NSE • Importance of contributions 34/82

Not really important Criteria and Indicators Amounts of previous grants (in particular n “Values” of criteria – Merit of the proposal. NSERC DG) should • Originality and innovation be ignored • Significance and expected contributions to research; • • • potential for technological impact Clarity and scope of objectives Methodology and feasibility Extent to which the scope of the proposal addresses all relevant issues Appropriateness of, and justification for, the budget Relationship to other sources of funds 35/82

Criteria and Indicators n “Values” of criteria – Contribution to the training HQP • Quality and impact of contributions • Appropriateness of the proposal for the training of HQP in the NSE • Enhancement of training arising from a collaborative or interdisciplinary environment, where applicable 36/82

Criteria and Indicators n “Values” of criteria – Cost of research • Rationale Not really important but you cannot have more than what you ask, no matter the merit 37/82

Criteria and Indicators n “Meanings” of indicators – Exceptional – Outstanding – Very strong – Strong – Moderate – Insufficient 38/82

Criteria and Indicators n “Meanings” of indicators – Exceptional • I did not see many exceptional ratings, obviously – Outstanding – Very strong – Strong – Moderate – Insufficient 39/82

Criteria and Indicators 40/82

Criteria and Indicators n NSERC rating form – NSERC provides a 2 -page rating form • I found that this rating form does not follow the presentation pattern during the competition slot because it spreads information • However, each application was obviously rated according to the 4 criteria and 6 indicators 41/82

Criteria and Indicators n NSERC rating form (1/2) Applicant: University: Application I. D. : Applicant Status: Title of Application: Evaluation criteria (See Section 6 of Peer Review Manual for complete details) Exceptional Very Strong Outstanding Excellence of researcher(s) oderate Insufficient Exceptional Outstanding Very Strong Moderate Insufficient Strong Knowledge, expertise and experience Quality of contributions to, and impact on, research areas in the NSE Importance of contributions For Team applications: complementarity of expertise between members and synergy Rationale for rating: Merit of the proposal Originality and innovation Significance and expected contributions to research; potential for technological impact Clarity and scope of objectives Methodology and feasibility Extent to which the scope of the proposal addresses all relevant issues Appropriateness of, and justification for, the budget Relationship to other sources of funds Rationale for rating: Contributions to training of highly qualified personnel Quality and impact of contributions during the last six years Appropriateness of the proposal for the training of HQP in the NSE Enhancement of training arising from a collaborative or interdisciplinary environment, Rationale for rating: where applicable Cost of research (relative cost of the proposed research program as compared to the norms for the field) Low Normal High Rationale for Cost of Research: 42/82

Criteria and Indicators n NSERC rating form (2/2) Other comments (e. g. , duration should exceptionally be less than norm, special circumstances, quality of samples of contributions provided, environmental impact, ethical concerns. Your Program Officer should be notified accordingly): Summary of assessment by external referees (please highlight any comments that would be deemed inappropriate for the Evaluation Group to consider in their discussions): Points for message to applicant (if rating of “Moderate” or “Insufficient” on any criterion or duration shorter than norm): Discovery Accelerator Supplement (DAS): Regular DAS: DAS in Targeted Areas : Rationale for DAS Recommendation: Yes No No 43/82

Criteria and Indicators n My Researcher Proposal HPQ own rating form Career 1: <Indicator> Funding Visibility Publications HPQ Record 3: <Indicator> Planned training Context 2: <Indicator> Lacks Incremental? Applicable? Feasible? Summary of the referees' reviews Provided contributions 44/82

Outline n Process in a Nutshell n – Candidate’s point of view – Internal reviewer’s point of view – – – • Off-line work • Competition week n Funding decisions – In a nutshell – Bins – ER vs. ECR n Criteria and indicators – – Advices Introduction Excellence of the researcher Merit of the proposal Contribution to the training HQP CCV (Form 100) Application (Form 101) Conclusion n Further readings n “Values” of criteria “Meanings” of indicators NSERC rating form My own rating form 45/82

Advices n Introduction – Reviewers receive 2 -3 dozens of applications – Overall, upon first review, the quality is impressive, thus generating a positive reaction – The objective is to discriminate, however, initiating a vigorous search for flaws 46/82

Advices n Introduction – Reviewers may perceive aspects of applications as confusing, ambiguous, incomplete, or just not compelling – They will not give the benefits of the doubt • I witnessed some excellent researchers (IMHO) receiving low ratings because of sloppiness in their applications 47/82

Advices n Introduction – Reviewers will most likely “mine” the CCV, application, and publications to make up their minds regarding the 4 criteria Make it easy for them to mine your applications! 48/82

Advices n Introduction – CCV (Form 100) • Is used for two of the three important criteria – Application (Form 101) • Is necessary for the merit of the proposal 49/82

Excellence of the Researcher n CCV (Form 100) – Career: pages 1 -2 – Funding: pages 3 -… – Visibility • “Other Evidence of Impact and Contributions” • Awards, chairing, editorship, organisation, seminars: anything showing external acknowledgments – Publications • “Other Research Contributions” • Quantity vs. quality 50/82

Excellence of the Researcher n Application (Form 101) – “Most Significant Contributions to Research” must be carefully used – “Additional Information on Contributions” should be used as well n Provided sample publications – Should be explained for the non-experts n External reviewers – Confirm/contrast findings 51/82

Merit of the Proposal n Application (Form 101) – Context • Is the application well positioned? – Lacks • Any problems not discussed? – Incremental? • How innovative? – Applicable? • Usefulness, even remote? – Feasible? • Methodology 52/82

Merit of the Proposal n Application (Form 101) – Reviewers may also look for • • Knowledge of the key issues (background) Originality and innovation (background limits) Clarity of scope and objectives Methodology – Trust/confidence that you can do work • Significance 53/82

Merit of the Proposal n CCV (Form 100) – Essentially nothing n Provided sample publications – Essentially context n External reviewers – Confirm/contrast findings 54/82

Contribution to the Training of HQP n CCV (Form 100) – HPQ Record • Pages 1, 5 -… • Make it consistent, report what the students do now n Provided sample publications – Essentially nothing n External reviewers – Confirm/contrast findings 55/82

Contribution to the Training of HQP n Application (Form 101) – “HQP Training Plan” is essential and must be well thought out • • Strategies for recruitment and graduation Collaboration with academia and industry Processes, meetings, hands-on Distinguish B. Sc. , M. Sc. , and Ph. D. – “Past Contribution to HQP Training” is also very important, their success is our successes 56/82

Forms n In general, my reading/rating was made easier when the application carefully followed the NSERC suggested guidelines/templates – Any missing/misplace/different category was disruptive and sent a bad signal: “I want to do differently from others” 57/82

CCV n Here is what I particularly looked at “ACADEMIC, RESEARCH AND INDUSTRIAL EXPERIENCE” – Current and past positions – Any industrial experience – Experiences that could explain the absence of publications, of HQP 58/82

CCV n Here is what I particularly looked at “RESEARCH SUPPORT” – The candidate, given experience, having appropriate funds? • NSERC and industry are most important • Others can help but should be explained, possibly in the contributions (F 100) or in the budget (F 101) • Amounts are not so important but give a signal • They shows the candidate’s willingness, relevance 59/82

CCV n Here is what I particularly looked at “HIGHLY QUALIFIED PERSONNEL (HQP)” – The candidate’s contribution to the training of past/current HQP • Titles of the projects should be meaningful and focused; unrelated titles send a bad signal • “Present positions” are important and show that the candidate follows his/her students • The degree obtained/pursued by the HQP (Ph. D. ? M. Sc. ? B. Sc. ? Others? ) 60/82

CCV n Here is what I particularly looked at “ARTICLES IN REFEREED PUBLICATIONS” – Evidence of scientific contributions • Quality was first and foremost: I looked at the venues and assessed their quality – Acronyms must be explained, publisher names, years… must be given, systematically – Bibliographic metric values may be given; Google Scholar citation counts is accepted; others metrics are discounted • Quantity was a plus but, without quality, it sent a bad signal: “I publish without focus” 61/82

Application n Here is what I particularly looked at “TRAINING OF HIGHLY QUALIFIED PERSONNEL” – Total numbers of students to know if the candidate is actively supervising students – Any increase/decrease – Any inflation of the numbers just before the submission, it sends a bad message 62/82

Application n Here is what I particularly looked at “MOST SIGNIFICANT CONTRIBUTIONS TO RESEARCH” – Evidence of scientific contributions • I ask myself “do I know this candidate? ” If unknown, I searched the places of publications to assess the quality of the contributions • References must better include where the papers were published to ease the reviewer’s task • The contributions should be related to the topic of the current NSERC DG to show continuation 63/82

Application n Here is what I particularly looked at “OTHER EVIDENCE OF IMPACT AND CONTRIBUTIONS” – Evidence of external acknowledgments • Anything that could help me confirm my impression on the merit of the candidate: awards, chairing, editorship, organisation, seminars – Unknown venues just to “fill in” that part must be avoided • Lack thereof was sending a bad signal: “I am not involved in the community” or “The community does not want me” 64/82

Application n Here is what I particularly looked at “TITLE OF THE PROPOSAL” – The title, which must be relevant and accurate, for a quick understanding (or not!) of what the proposal is all about 65/82

Application n Here is what I particularly looked at “PROVIDE A MAXIMUM OF 10 KEY WORDS…” – The keywords, which must be relevant and accurate, to get an deeper idea of what the proposal is all about 66/82

Application n Here is what I particularly looked at “TOTAL AMOUNT REQUESTED FROM NSERC” – The total amount requested, which could raise my curiosity if “too” high (say, more than 70 K) or “too” low (say, less than 30 K) • I would then read the budget to understand the detailed numbers 67/82

Application n Here is what I particularly looked at “SUMMARY OF PROPOSAL FOR PUBLIC RELEASE” – This very important part • In my domain, would help me understand the application and form an idea of what to expect • Outside my domain, would help me understand the application as a whole • I would look for clear objectives, expected results and their significance, and contributions to HQP • Any typo or grammar error raised warning! 68/82

Application n Here is what I particularly looked at “TOTAL PROPOSED EXPENDITURES” – The use of the money towards HQP • “Most” of the money should go to fund HQP; I mean at least 80%-90% in the CS Evaluation Group • But some money should be used for students’ travels (not only for the candidate’s) • I would also look for any “surprising” amounts in any “surprising” category 69/82

Application n Here is what I particularly looked at “BUDGET JUSTIFICATION” – The use of the money towards HQP • It should include a table showing, per year, the use of the money on students and possibly pictures of uncommon equipment • Budget is not so important but errors or lack of justification sent a bad signal • Also, it is important to be consistent: 5 K are not enough to equipped a whole lab. with computers… • It should also explain any specific institutional rules or complementary source of funds • When naming conferences for travel, only name conferences 70/82 relevant to the proposal and possibly justify the choice

Application n Here is what I particularly looked at “RELATIONSHIP TO OTHER RESEARCH SUPPORT” – Possible duplication of funds • If there is a clear relation between two or more funds, in that case, it should be clear how the money of the NSERC DG will be used, e. g. , students 71/82

Application n Here is what I particularly looked at “PROPOSAL” – The context, objectives, expected results in the first few lines or maybe an example to help me understand what the proposal is all about – A story – Any missing/misplace/different category, which was disruptive and sent a bad signal: “I want to do differently from others” 72/82

Application n Here is what I particularly looked at “PROPOSAL” – An up-to-date background • Few recent references are interesting • Some discussions of the references and of their contributions and limitations is definitely needed • If possible some clear, simple, and convincing running examples illustrating the references limits 73/82

Application n Here is what I particularly looked at “PROPOSAL” – The contrast with the background • Then, the proposal should explain how it will improve on the limits of the state-of-the-art or state-of-thepractice and demonstrate its feasibility • Then, it should also explain how it will go beyond clear feasibility to show innovation • It must be limited to 3 main objectives, with milestones and students 74/82

Application n Here is what I particularly looked at “PROPOSAL” – Clearly stated goal(s), challenge(s), expected results and their significance, for each objective • The proposal should not hide weaknesses and challenges but explain them and the methodology to lessen their impacts • It should also explain how to evaluate the success/ failure of a goal and what happens “if…” 75/82

Application n Here is what I particularly looked at “PROPOSAL” – The association of students with the objectives • Possibly, the proposal should explain the recruiting of students in the next section, “Contribution to HQP” – Its possible significance • On society, not on yourself or your small community • Use of other, existing funds is “proof” of interest from external, broader society 76/82

Application n Here is what I particularly looked at “CONTRIBUTION TO HQP” – The context of training and its methodology • It is a chance to explain how this application will concretely be used to train students • It is better if there already some student names • If the merit of the candidate is low and the candidate boast too many students, it can be negative 77/82

Application n Here is what I particularly looked at “REFERENCES” – Consistency • References should be consistent and well-written, again, any typo or grammar errors raised a warning! 78/82

Outline n Process in a Nutshell n – Candidate’s point of view – Internal reviewer’s point of view – – – • Off-line work • Competition week n Funding decisions – In a nutshell – Bins – ER vs. ECR n Criteria and indicators – – Advices Introduction Excellence of the researcher Merit of the proposal Contribution to the training HQP Form 100 Form 101 Conclusion n Further readings n “Values” of criteria “Meanings” of indicators NSERC rating form My own rating form 79/82

Conclusion n Follow the guidelines / templates carefully n Be simple, clear, straightforward – Even with the weaknesses n Do not forget anything but do not show off either, explain, explain n Avoid at all costs typo / grammar errors 80/82

Outline n Process in a Nutshell n – Candidate’s point of view – Internal reviewer’s point of view – – – • Off-line work • Competition week n Funding decisions – In a nutshell – Bins – ER vs. ECR n Criteria and indicators – – Advices Introduction Excellence of the researcher Merit of the proposal Contribution to the training HQP Form 100 Form 101 Conclusion n Further readings n “Values” of criteria “Meanings” of indicators NSERC rating form My own rating form 81/82

Further Readings NSERC Discovery Grants – An Insiders View by Larry W. Kostiuk (Co-Chair, Fluids GSC 1512) n How to Succeed in the New NSERC Discovery Grant Competition Model by Evangelos Milios, Nur Zincir-Heywood, and Stavros Konstantinidis n NSERC Discovery Grant Applications: Hints and Insights by Jason P. Leboe-Mc. Gowan n Advice on NSERC Discovery and RTI Applications by Robert Bridson n 82/82