Quality Assurance in Higher Education and Vocational Education

Quality Assurance in Higher Education and Vocational Education and Training WS 8/Panel 1: Reflection on the demands on quality in HE Gudrun Biffl/AUT/WIFO

Globalisation and Creation of a Common HE-Area by 2010 in Europe (Bologna Process) • increased the demand from students, employers, and academics for indicators of the international academic standing of universities. Global companies recruit globally; students intending to study abroad choose from an array of universities; international agencies, who are the buyers of research, seek expertise on the basis of international rankings/indicators. Potentially conflicting targets: • national objectives, e. g. democratisation of access versus concerns for quality in HE, • internal functioning of universities versus demands emananting from the external environment (globalisation, Bologna Process) • Standardisation versus specialisation

What does quality mean? How is it measured? • Different aspects of quality for different interest groups – students, staff, scientific community, nation states, supranational (EU) and international institutions • Output of what: – Research or Teaching? • • How to Measure Performance: teaching/research Ranking of HE-institutions as an entity and/or in the various disciplines in a country, in Europe, world wide? • One numerical value? On the basis of what indicators and their respective weights • Many indicators – based on what criteria?

What are the concerns of students • • • scores required for entry, class size, ratios of academic staff to students, degrees held by academic staff, attrition/graduation rates, and spending on student services, especially libraries and IT. • Research performance of academic staff gets a low weight

What are the concerns of funding institutions • overall standing of an institution is evaluated to a large extent on the basis of research outputs and citations/patenting/affiliations of Nobel Prize winners at different stages in their careers. Research performance is measured over a period of years (up to two decades for highly-cited researchers). • Governments are increasingly using performance measures to allocate funds to universities - as governments want their country to leave a footprint in science (the scientific impact of nations) funding for research excellence increases • Rankings of a given discipline across universities may differ from the ranking of institutions as single entities. • The variability in the quality of departments within an institution is falling as a result of the growth in interdisciplinary research and quality controls on departments and universities. • “Bad” departments in good universities are becoming rarer.

Definition and determinants of quality in HE: according to a survey of academic leaders worldwide the determinants may be grouped into six broad categories, : 1. standing of academic staff : Highest weighting: 40% of a compounded rank indicator, greatest range in scores by universities 2. quality of undergraduate intake (weight 11%), 3. quality of undergraduate programs (14%): this is the area where standardisation takes place 4. quality of graduate programs (16%), 5. Resources (11%), and 6. peer opinion (8%).

How to measure academic staff research quality • the quantity and quality of research publications; • citations of the research output; • research income, especially from competitive grants; • recognition of academic standing through election to academies, • receipt of prestigious awards or invitations to conferences; • status of degrees held by academic staff; • research output/degrees held - best measure of current performance

How to measure quality of students and teaching excellence • • • Degree of selectivity of student intake (tertiary entrance score (TES, baccalaureate-Matura grades) are generally the result of State-wide examinations at the end of schooling: selection may vary by degree course, high correlation between TES values for students and their graduation rates and scores number of completions (through-put), placement on graduation (employability), continuation into higher studies, particularly progression rates into Ph. D. courses, awards won or honours grades achieved, diversity of the student body and class size, student evaluation (of Ph. D. programmes in terms of research quality) The ratio of students to academic staff (with the increased role of teaching through the internet input from technical staff should also be included) teaching and supervision excellence of academic staff are reflected in the measures of the quality of undergraduate and graduate programs

How to measure resources • Resource levels can be measured in physical units or monetary amounts. • The use of physical units facilitates international comparisons (support staff, library volumes and IT facilities). • Monetary measures include total revenue per student (indicator of the resources available for teaching and research training) or staff member (indicator of the resources available for administrative and technical assistance for research), and expenditure on IT and libraries. • Alumni giving and salary levels are sometimes included as measures of standing. • In principle, to allow for size differences between institutions, revenue should be deflated by a weighted average of student and staff numbers. • Insofar as revenue per academic staff member is used to pay higher average salaries it is also a further measure of the standing of academic staff.

Who are the global peer reviewers? • chief executive officers (CEOs) of universities of high standing • chief executive officers (CEOs) of global enterprises • chief researchers/advisors of international agencies

Spillovers into the society, local communities • Universities may have other missions than scientific excellence, and only the latter is evaluated on the basis of internationally applied performance or reputation indicators. • The contributions of universities extend to their impact on society, such as contributions to public policy and technology; they may serve the local community or region by transferring technology and know how to the local actors (technical and socio-economic innovation), as well as by providing policy advice. • This role and impact is ignored in international scoreboards

Extending the range of variables to provide a more comprehensive picture of the standing/role/impact of HE institutions • by extending the range of data to include variables such as citations in government policy papers and patents taken out, • by extending the qualitative data by surveying key decision-makers outside the higher education sector, • By including books and contributions to books: the impact of books is undervalued in evaluation studies • Include the influence of the internet. These various extensions of coverage would help to redress the bias in existing measures towards more theoretical research.

Open questions • Today no robust and consistent way to measure the quality of research in HE Institutions • Nature of science is changing: there is more intersection across disciplines • while at the same time only narrow specialisations get published in Journals • How to evaluate the bridging skills/contributions to natural and social science and humanities?

Impact of ranking on HE-institutions Will ranking procedures contribute to widening of standing/status of universities and reinforce a scientific elite league? Will excessive evaluation endanger scientific research ? Research shows that citations are a bad indicator of quality of research, as indicated by a low correlation between experts scores given to institutions/subfields and citation analysis scores A balance has to be found, i. e. a combination of methods: – The use of data-systems for evaluation, – advanced bibliometric assessment procedures (citations in fields and of institutions), – expert surveys (Peer reviews)

Conclusion • International academic standing is only one criterion by which a university’s performance may be measured. • There is the need to develop other measures of performance – which have a more local focus, such as contributions to regional development, contributions to national well-being, – and detailed evaluations of teaching performance which takes account of the mix of students. A range of criteria linked to mission statements is necessary if countries are to have a diverse higher education system which caters for the full range of desires of its people and the needs of society