Peer Reviewed The International Career of a Qualitycontrol

Peer Reviewed: The International Career of a Quality-control Instrument and New Challenges 24– 25 April 2008, Social Science Research Centre Berlin (WZB), Berlin Peer review practices in the research field: Light and shade of the Italian experience (1997 -2007) Eliana Minelli – University Carlo Cattaneo, LIUC Gianfranco Rebora - University Carlo Cattaneo, LIUC Matteo Turri – University of Milan Università Carlo Cattaneo 1 Eliana Minelli - Gianfranco Rebora – Matteo Turri

The focus of the paper • This paper aims to review the ten year evaluation experience in Italian universities, focusing on the light and shade of peer review in the field of scientific research, at the national and individual university level Our analysis considers: • an important peer review experience in research departments applied autonomously by the University of Trento after the year 1999; • the national evaluation exercise performed by the CIVR (Italian Committee for the Evaluation of Research) regarding the years between 2001 and 2003. Università Carlo Cattaneo 2 Eliana Minelli - Gianfranco Rebora – Matteo Turri

Peer review in universities • Evaluation is not a technique but a set of many different elements that takes shape through evaluation reports • Evaluation by peer review can play an important role in facilitating and legitimising decision-making because it profits from the professional nature of academic activity • Peer review has been the main methodological framework adopted in universities, particularly in the field of research evaluation Università Carlo Cattaneo 3 Eliana Minelli - Gianfranco Rebora – Matteo Turri

Peer review in universities • Professional bureaucracy (Mintzberg 1983) • Democratic organisation with bottom-up decision-making and controls that are limited to selecting professional staff because the output of the typical processes cannot easily be quantified and measured. • “loosely coupled organization” ( Weick 1976) refers to the features of organisations involved in educational tasks • Weak points of this kind of organizations: – the weak coordination – great discretionary power of operating processes – slight inclination to introduce innovations Università Carlo Cattaneo 4 Eliana Minelli - Gianfranco Rebora – Matteo Turri

Peer review in universities • • When accepted by the organisation, immediately evaluation becomes a reference point for decision-making due to the previously mentioned loosely coupled system. When evaluation output is available it enters the most critical gaps in professional bureaucracy and helps to set a large number of important organisational effects in motion: – evaluation stimulates government bodies to make decisions and provides them with answers that enable them to justify and give reasons for the choices they make; – in a professional environment that has considerable discretionary power and very few rules, evaluation establishes reference points that are also time-related and which the organisation adopts as a stimulus; – the “fence-sitting” attitude of academic activities is disrupted by evaluation which introduces stimuli for modifying behaviour and professional practices. Università Carlo Cattaneo 5 Eliana Minelli - Gianfranco Rebora – Matteo Turri

Peer review in universities • • • These effects are only visible if evaluation is accepted by organisational actors and basic values are shared by evaluated people: this explains the importance of peer review in the field of evaluation. Peer review is not so much interested in the use of particular techniques or indications as in a call for professional values shared by the persons being evaluated (Henkel, 1998 – Turri, 2005 – Reichert, 2006). Two possible advantages in adopting peer review: – makes it possible for the fundamental values of the discipline to be formally recognised by the evaluation exercise; – legitimise the self-regulation of the scientific community which sees the peers’ evaluating role as a shield against accusations of self-referencing and thus a defence against attempts to harm academic autonomy through the introduction of evaluation based on external criteria. Università Carlo Cattaneo 6 Eliana Minelli - Gianfranco Rebora – Matteo Turri

Peer review in universities • • • In the field of research, peer review makes it possible to take into consideration the enormous variety of aspects that are necessary to express a rating on scientific activities. Researchers are used to peer review because they accept the fact that others judge their work when they decide to publish the results of their piece of research Peer review seems to be well adapted to the need for autonomy and accountability to coexist and integrate because it gives impartial external elements to university decision-making on the quality of research and its lines of development. Università Carlo Cattaneo 7 Eliana Minelli - Gianfranco Rebora – Matteo Turri

Peer review in universities • Weak points and degenerations: – peer review is a conservative practice that is only slightly disposed towards innovation or multidisciplinary experiences because it is tied to disciplinary values – visits by peers are irregular and if no self-evaluating infrastructure exists this may foster an outward behaviour that is not linked to every day practices. – inappropriate behaviour may be shown by peers such as lack of impartiality when making their assessment – time and costs involved in peer review are certainly greater than in mechanical evaluation procedures such as journal ranking and bibliometric analysis Università Carlo Cattaneo 8 Eliana Minelli - Gianfranco Rebora – Matteo Turri

Two experience of peer review in Italy • Two specific cases are analysed : – In university: peer review of departments in the University of Trento, one of the most significant complete experiences of this kind in Italy (Turri, 2005), – In the university system: VTR (Three year evaluation of research) promoted by the CIVR was carried out and took into consideration publications between 2001 and 2003. Università Carlo Cattaneo 9 Eliana Minelli - Gianfranco Rebora – Matteo Turri

University of Trento Between 1999 and 2003 the University of Trento evaluated the scientific research of all its departments with an evaluation exercise based on the following principles: • to support research policy and not evaluate or penalise the work of individual researchers; • to establish the international positioning of each department in relation to the disciplinary scientific sector in which it was working; • to entrust review to members of the disciplinary scientific community who were chosen according to their prestige and authoritativeness; • to allow the peers to use all the methodologies they deemed necessary for ascertaining the disciplinary excellence of the department. Università Carlo Cattaneo 10 Eliana Minelli - Gianfranco Rebora – Matteo Turri

University of Trento The University of Trento adopted the following peer review procedure: • each department set up an evaluation committee • the peers were given access to all the necessary information and documentation for assessing the activity of the department; • the peers made a two day visit to the department during which time they had confidential talks with all the staff; • after their visit the peers prepared a written report for the department’s and university’s governing bodies which included the international positioning of the department according to the RAE scale (from 1 to 5). Università Carlo Cattaneo 11 Eliana Minelli - Gianfranco Rebora – Matteo Turri

Results from peer review in University of Trento • • • The exercise has met with little resistance and organisational inertia and university bodies made good use of its output Evaluation in the university was considered to be coherent with the values of the university community and was therefore accepted The opinions of the Heads of Departments and the university hierarchy in Trento University pinpointed : – – – Improved awareness of research Stimulus for managing research Guide for the allocation of funds Tonic for decision-making bodies Internationalisation Università Carlo Cattaneo 12 Eliana Minelli - Gianfranco Rebora – Matteo Turri

Improved awareness of research • As one interviewee pointed out “its use and the way in which we are using it has consisted in pinpointing some weak points that have to be put right, but it also made us more confident about certain strategies that we had put into action: so, it was useful for defining the positioning of the department. ” Università Carlo Cattaneo 13 Eliana Minelli - Gianfranco Rebora – Matteo Turri

Guide for the allocation of funds • One Head of Department affirmed: “it has been useful on two fronts – that of the university in order to have a more precise idea of the amount and the quality of the activities that are carried out and thus allocate funds accordingly. The second front, inside the department, has served to implement and justify the distribution of funds in the various research areas…” • “when I prepared the budget proposal, which is the most important proposal made to the scientific research commission by the department every year, I greatly relied on the evaluation report made during peer review”. Università Carlo Cattaneo 14 Eliana Minelli - Gianfranco Rebora – Matteo Turri

The three year CIVR research evaluation exercise (VTR, 2001 -2003) • The CIVR was set up in 1998 to promote research evaluation by supporting quality and the enhanced use of national technological and scientific research • The VTR examined research activities in the three year period from 2001 to 2003 and had its roots in the peer review experience in Europe • The evaluation exercise aimed to evaluate research activities in state and legally-recognized Italian universities through peer review based on assessment of the merits of particular research products indicated by the universities. • Great expectations on the increased use of performance measurement in the allocation of funds from the state to the universities and from the universities to the various departments but no significant effects Università Carlo Cattaneo 15 Eliana Minelli - Gianfranco Rebora – Matteo Turri

Characteristic traits of the evaluation procedure (3) • Attention to output: The ratings obtained by the research structures in each specific area are a direct result of the assessment given to the individual research products • Change in the relationship between universities, government and evaluation bodies: the VTR required the universities to indicate the input on which they had to be assessed. The evaluation exercise has also led many universities to question their ability to carry out quality research and to make a comparison with other universities. Università Carlo Cattaneo 16 Eliana Minelli - Gianfranco Rebora – Matteo Turri

Characteristic traits of the evaluation procedure (3) • Enhanced decision-making in universities: Selecting research products for evaluation also enhances decision-making in university departments and shifty behaviour is overcome • Direct involvement of academics in evaluation: The widespread participation in the VTR has brought results such as the initiatives of various disciplines to reflect on and formalise their criteria of excellence. Università Carlo Cattaneo 17 Eliana Minelli - Gianfranco Rebora – Matteo Turri

Characteristic traits of the evaluation procedure (3) • Automatic evaluation: Many disciplines indiscriminately adopt international criteria that use automatic mechanisms to assess the quality of excellence. In practice the criteria for assessing the quality of research products are left to the discretion of the area panels. • Inadequate (not used) for the allocation of funds: the reports show the peaks of excellence in the national system but serve no purpose in other matters such as the productivity of university staff; the VTR results are little used at a national level. Università Carlo Cattaneo 18 Eliana Minelli - Gianfranco Rebora – Matteo Turri

Lights: PR is a useful tool in a change management perspective • There is less resistance to peer review than to other evaluation tools • The national evaluation exercise has led many universities to question their ability to carry out quality research and to make a comparison with other universities • Evaluation stimulates and strengthens coordination inside universities, creating the conditions for adapting academic professional autonomy to the department’s and university’s needs for coordination • The use of peer review based on assessment of research products by high-level experts has put research structures under pressure and strengthened the desire to comply with international standards • International character of the exercise performed by the CIVR and the universities’ voluntary evaluation exercises are fully aligned with the requisites of the process for integrating education and university research in Europe and satisfy a widely-felt viewpoint Università Carlo Cattaneo 19 Eliana Minelli - Gianfranco Rebora – Matteo Turri

Shadows: consequences are weak The weak side of Italian evaluation activities consists in the uses of evaluation results : • Substantial lack of any kind of competitive system for rewarding universities that adequately meet the requirements of families and enterprises • These procedures have only been used sporadically for modifying research funds and even less for intervening in university governance or influencing academic power structures Università Carlo Cattaneo 20 Eliana Minelli - Gianfranco Rebora – Matteo Turri

Doubts and controversies • Controversies exist in some large disciplinary areas on the adoption of guidelines for the evaluation of scientific outputs – disciplinary areas where the aims and criteria of peer review in research are basically accepted – areas where peer review is remote and needs additional elaboration – areas that are affected by controversies pertaining to schools of thought and the balance between different scientific approaches. • The Italian experience is still not sufficiently developed to verify the limits of peer review Università Carlo Cattaneo 21 Eliana Minelli - Gianfranco Rebora – Matteo Turri