RAE Review Interim feedback from facilitated workshops Jonathan

RAE Review: Interim feedback from facilitated workshops Jonathan Grant and Steven Wooding

Plan n Introduction n n Results from 6/9 workshops n n n Who we are and what we do Where we went and who we spoke to What we did and why we did it Task 1: 2: 3: 4: What is quality research? How should it be assessed? Assessing model systems Building a better system Implementation and implications Some emerging observations n n Preference for expert review, but needs refining Increased transparency and clarity of process Tension between comparability and appropriateness Need structures to support change

Who we are and what do we do n RAND Europe n Independent not-for-profit public policy think-tank n ‘Cousin’ of RAND; US based independent think-tank employing 1600 researchers n RAND Europe. Established in Leiden (NL) in 1992; Cambridge in 2001 n In UK programmes include Transport, Information Society, Health and R&D Policy n Current projects include VFM study on government department research for NAO and on scientific mobility for WHO

Where we went and who we spoke to n Based on 6/9 Workshops n 93 people Positions n n n 26 38 13 8 8 Administrators Senior academics Academics Research Fellows Unclassified Fields n 32 Medicine, science & engineering n 16 Social Science n 15 Arts and Humanities n 29 Not research active n 1 Unclassified *Excludes Cambridge, Reading & Belfast (n=c 50)

What we did and why we did it n n Facilitated workshops n Provide framework for structured thinking n Captured outputs in a standard and comparable form n Allows comparison between mixed and like groups of people (e. g. , administrators only vs. mix of Ho. Ds, fellows, and research officers) Purpose is to listen, not evaluate

Agenda n Task 1: What is quality research? How should it be assessed? n Task 2: Assessing model systems n Task 3: Building a better system n Task 4: Implementation and implications

Task 1: What is high quality research? How should it be assessed? n Purpose n n Stimulate wide ranging thinking on the most important aspects marking out high quality research and research assessment systems Task 1 n Introductions n Identify 5 characteristics of high quality research n Identify 5 characteristics of research assessment system n Vote (5 votes for each; allocated as seen fit)

‘Top 10’ characteristics of high quality research

‘Top 10’ characteristics of research assessment systems

Task 2: Assessing models systems n Purpose n n Evaluate the strengths and weaknesses of 4 ‘model systems’: Expert review; Algorithms/metrics; Selfassessment; and, Historical Ratings Task 2 n Split into 4 groups of c 4 -5 n 2 groups look at ‘good’ aspects of 2 systems each n 2 groups look at ‘bad’ aspects of 2 systems each n Also identify questions that need answering

Algorithms / metrics ‘Good features’ n n Transparent (4) Objective (3) Cheap (3) Simple (2) Selected questions n n n ‘Bad features’ n n Not suitable for all subjects (5) Spurious objectivity (3) Open to ‘game playing’ (2) Metrics are proxy measures (2) How to recognise novelty? What do you count? How to ensure comparability?

Expert review ‘Good features’ n n Acceptable to community (5) Based on specialised knowledge (3) ‘Bad features’ n n n Questions n n n Not comprehensive (3) Not transparent (3) Perceived bias (3) In consistent (2) Expensive (2) Who are the experts and how are they selected? How do you build in an appeals mechanism How do you recognise innovation?

Historical ratings ‘Good features’ n n n Light touch (3) Cheap (2) Ability to plan (2) Questions n n n ‘Bad features’ n n n Inhibits change (3) Low credibility (3) Perpetuates silos (2) How do you take account of changing performance? Who makes the judgement? (HEI or individual) How far back?

Self-assessment ‘Good features’ n n n Sensitive to discipline (2) Formative – considers self (2) Ownership & Trust (2) ‘Bad features’ n n Questions n n n No cross discipline comparability (3) Open to game playing (3) No confidence in system (2) Effort could be large (2) How would you police it? Who sets the goals? How do you penalise inflated results?

Task 3: Building a better system n Purpose n n Design ideal research assessment system Task 3 n Split in to 3 or 4 different groups of c 4 -7 n Select ‘seed’ system from Task 2 n Build on this using aspects of other system n Present back to plenary

Identifying a base system n Starting point n Expert review (16/18 breakout groups) n Self-assessment (1/18 breakout groups) n 1 ‘failed’ (to reach a decision) n (remember based on 6/9 workshops)

Refining the expert system n Transparency and clarity of process n n Establish rules at outset n Early (i. e. , up to 5 years before assessment) n Don’t change rules during process Provide feedback n n Legal contract between FCs and HEIs (rules won’t change, in return for guarantee of no challenge) Clarity of funding outcome

Refining the expert system n Longer time period between assessments n Review every 8 -10 years n ‘Triggering’ mechanisms for interim review at 4 -5 years. n n Self declaration for new or emerging areas n Metrics for decline areas Sampling (selective or random) with other departments

Refining the expert system n Broader panels n Mirror Research Councils/AHRB n Broad or supra-panels have broad freedom to establish discipline specific rules n Sub-panel operate to those rules, reporting to supra -panel n Aims to solve tension between comparability and appropriateness

Refining the expert system n n Continuous rating scale n More grades n Summation of individuals ‘scores’ n Possibly based on ranking Continuous funding scale n No (or reduced) step changes in funding

Refining the expert system n Conceptual (non subject) Ao. As n n Based on research outputs n Formal sciences Theorems n Explanatory sciences Laws n Design sciences Technological rules n Human sciences Artifacts and knowledge ‘Users’ from relevant Ao. A

Refining the expert system n ‘Lay‘ Panel members n n Rolling review n n One discipline every year Transfer fees n n Possible experience lay members (such as judges) as panel chair Reward departments that nurture future high flyers Return all staff

Task 4: Implementation and implications n Purpose n n Examine system would be put into practice and evaluate repercussions Task 4 n Working in same groups with ‘devil’s advocate’ n Identify 5 steps for implementation n Identify 5 things that could go wrong with the system n Identify 5 changes to the UK research as a result of the new research assessment system

Implication of change n Some changes to UK research: n Ensure research capacity is activated wherever it is found n Better support and funding for younger researchers n Good support for emerging areas of research n Equal opportunities are improved n Better recognition of interdisciplinary research n Funding directly following excellence n Less obsession with the RAE

Emerging observations n Preference for expert review, but needs refining n Increased transparency and clarity of process n Tension between comparability and appropriateness n Need structures to support change But …. n Excludes Cambridge, Reading and Belfast n Not analysed by discipline and or profession