The NRF rating system and its appeals some

The NRF rating system and its appeals: some facts and some fallacies Duncan Mitchell

● I am a theoretical physicist and the panel consists only of experimental physicists, so they can’t assess me ● My h index is better that of five B-rated researchers I know, but the committee gave me a C ● I was elected Vice-Chairman of the International Society of Orchid Botanists so I must have an international reputation ● My research has improved but my rating has gone down

Reviewers ● The only assessors of research output ● Your choices and the specialist committee’s choices ● Excluding potential reviewers Pick a likely reviewer and write your application for her/him

The reviewers’ focus ● Your best outputs of the review period ● Peer-reviewed journal articles ● Other legitimate research outputs ● Postgraduate students Be careful out about conference proceedings Teaching textbooks are not research outputs

Scoring the reviewers’ reports ● Familiarity with the research field ● Evidence-based assessment of the applicant’s research in the review period ● Bias and hostility ● Halo effect ● Congruence with data in the application Good reports count more

Arriving at the rating ● Enough usable reports? ● Search for the language: …… coherent body of work? …… promise, leadership? …… significant international impact? ● Agreement between committee, assessor and chair Count up the A, B, C, P, Y, RU reports (weighting the good reports)

The Appeals Committee ● Mandate ● Membership ● Evidence: ……. appeal letter. . …. . . RRAP sheets ……. documents that served before the specialist committee/ Executive Evaluation Committee ……. and it has the right to call for more reports

The Appeals Committee’s deliberations ● Is the appeal legitimate? ● ● What issues in the letter need a response? Did the rating process follow the rules? Were there enough usable reviewers’ reports? Was the set of reviewers (with usable reports) balanced? Does the Appeals Committee agree with the rating decision?

The difficult issues ● ● Applicants on a cusp Established early-career researchers Members of large research teams Researchers competent in more than one field of research ● Researchers still active but past their peaks

● I am a theoretical physicist and the panel consists only of experimental physicists, so they can’t assess me ● My h index is better that of five B-rated researchers I know, but the committee gave me a C ● I was elected Vice-Chairman of the International Society of Orchid Botanists so I must have an international reputation ● My research has improved but my rating has gone down