Development and Approbation of a Template Methodology for

Development and Approbation of a Template Methodology for National Ranking of Higher Education Institutions: 2011 outputs and next steps Larionova Marina (NTF) Perfilieva Olga (NRU HSE International Organizations Research Institute) Project “Development and Approbation of a Template Methodology for National Ranking of Higher Education Institutions” 2011 -2013

I. Project background Project “Development and Approbation of a Template Methodology for National Ranking of Higher Education Institutions” 2011 -2013 implemented by National Training Foundation in collaboration with International Organizations Research Institute of the National Research University Higher School of Economics at the request of the Russian Ministry of Education and Science. 2

Project goal • • • To develop and approbate a template methodology for ranking of Russian higher education institutions through: comparative analysis of global, national and specialized rankings; national approaches to evaluation of higher education institutions performance public and expert discussions of the draft methodology approbation of the draft methodology processing and discussion of the approbation outcomes consultations with IREG experts 3

Approbation of the developed methodology. Processing the approbation results Organizing public and expert discussions of the approbation results. Consulting with IREG experts to audit the methodology for national ranking of the higher education institutions. Amending the draft methodology in accordance with the results of the discussions. Developing recommendations on the application of the template methodology for national ranking of higher education institutions. Organizing an international conference to discuss the template methodology for national ranking of higher education institutions and the approbation outcomes. 20 20 12 Analyzing the Russian approaches (methodologies and indicators) used to evaluate performance of higher education institutions. Conducting a comparative analysis of global, national, and specialized rankings; identifying their strengths and weaknesses. Carrying out a comparative analysis of the methodologies and approaches used in international /foreign and Russian practices. Developing a template methodology for national ranking of higher education institutions. 13 20 11 Project tasks 4

The methodology approbation outcomes. The outcomes of expert and public discussions. The outcomes of consultations with the IREG experts. The draft methodology for ranking of Russian higher education institutions. The recommendations for application of the draft methodology for ranking of Russian higher education institutions. The outcomes of an international conference on the methodology for national ranking of higher education institutions. 20 12 The results of a comparative analysis of the methodologies and approaches used in international /foreign practices. The results of a comparative analysis of the methodologies and approaches used in the Russian practice. The results of a comparative analysis of the methodologies and approaches used in international/foreign practices and methodologies and approaches used in the Russian practice. A template methodology for national ranking of higher education institutions. 13 20 11 Key Outcomes 20 . 5

II. General approaches to the methodology of developing a template methodology for HEIs ranking Key principles: 1. 2. 3. 4. The methodology should provide reliable information on performance of higher education institutions and their position in rankings. The methodology should inform users of educational services on diversity of higher education institutions and education programmes providing friendly and easy-to-use information The methodology should facilitate improvement of quality and competitiveness of higher education institutions The methodology should be a source of reliable data for global and regional rankings 6

Key principles The methodology should take into account: 1. Experience and achievements of the Russian higher education institutions in the area 2. Objectives to improve competitiveness and facilitate integration of the Russian higher education institutions into global education and research environment 3. Increasing number of the Russian higher education institutions that participate/will participate in global rankings 4. Pragmatic approach to the methodology: data collected for national ranking should correlate with the data universities provide for global rankings 5. Strengths of quantitative indicator. 6. Strengths of global ranking methodologies 7

Methodology for a comparative analysis 3 levels of analysis • 1 level: Comparative analysis of methodologies on key selected parameters (target groups, key objectives, areas of evaluation, frequency, method of data collection and processing etc). • 2 level: Assessing ranking methodologies against Berlin principles on ranking of Higher Education Institutions and the IREG audit criteria and identifying key quantitative indicators • 3 level: Assessing the indicators against criteria of relevance to the Russian education system development objectives, reliability and feasibility of data collection 8

1 level: Comparing methodologies using common parameters Criteria for selection of ranking methodologies: • Rankings in with Russian universities participate or are expected to take part • Most popular rankings, which top listing is perceived as “signal” of universities competitiveness in international education and research area. • Rankings with methodologies available in open access to ensure transparency and understanding of indicators’ relevance and reliability of the obtained results. 9

Criteria for selection of ranking methodologies (continued) • • • possibility of assessing ranking methodologies against Berlin principles on ranking Higher Education institutions possibility of assessing ranking methodologies against IREG audit criteria account of diverse practices of various countries inclusion into analysis of different methodologies feasibility of application for the national HEIs ranking 10

Methodologies analyzed Global rankings • The Academic Ranking of World Universities (Shanghai Jiao Tong University ranking) • Times Higher Education Supplement (THES). • World’s Best Universities Rankings - U. S. News and World Report • U-Multirank • Leiden Ranking • QS World University Rankings Regional rankings • U-Map 11

Methodologies analyzed National rankings UK • Time Good Education Guide (UK) • Guardian Ranking, Universities league table • Guardian Ranking, Specialist institutions USA • Carnegie classification • Forbes Germany • The CHE University Ranking • CHE Excellence Ranking • The CHE Research Ranking • CHE Employability Rating 12

Methodologies analyzed Specialized rankings • Financial Times ranking of business schools • Blumberg Business Week ranking • The Economist MBA ranking • Indicators for Mapping and Profiling Internationalization 13

Position of Russian universities in global rankings QS Shanghai THE Leiden US News and QS Ranking 2011 - Moscow State University 112 - St. Petersburg State University - 251 - Bauman Moscow State Technical University 379 - Moscow State Institute of International Relations (University) - 389 - Novosibirsk State University - 400 - Tomsk State University - 451500 - Ural Federal University - 451500 - Higher School of Economics 551 -600 - Tomsk Polytechnic University - 551 -600 - People’s Friendship University - 551 -600 Ranking 2010 - Moscow State University – 74 - St. Petersburg State University – 301 – 400 Ranking 2011 2012 Moscow State University 276 -300, St. Petersburg State University – 351 -400 Ranking 2010 - Moscow State Universit y - 70 St. Petersburg State Universit y 332. - Moscow State University – 93 (global ranking, 400 universities) St. Petersburg State University – 210 (global ranking, 400 universities) Novosibirsk State University – 375 (global ranking, 400 universities) - Moscow State University – 38 (European Universities, 50 universities) 14

Definitions • Single-dimensional ranking displays vertical diversity in terms of performance by using indicators. Most existing higher education rankings take the form of a league table. • League table – a single-dimensional list going from “best” to “worst”, assigning ordinal numbers to the entities which relate only to rank and scales of difference. • Multi-dimensional ranking does not try to combine education and research rankings, for example, into single, composite measure and is often userdriven because it enables interactive display of data. • Classification is a system that allocates objects to groups on the basis of their characteristics. It shows horizontal diversity, where differences do not imply ordinary scales of “more”, “bigger” or “better”. It is aimed at showing diversity of higher education institutions. van Vught F. A. , Westerheijden D. F. Multidimensional Ranking: a new transparency tool for higher education and research // Higher Education Management and Policy. 2010. Vol. 22/3. 15

Types of analyzed methodologies Type Characteristics Methodologies Single-dimensional ranking (rankings, league tables) -Normalizes -Assigns scores - Compares higher education institutions and creates a hierarchical list of HEIs from “best” to “worst” based on composite scores - Uses single aggregate score - User-driven Shanghai, THES, QS, US News, Leiden, Times, Guardian Sp, Time Good Education Guide, Forbes, Financial Times, Bloomberg Business Week, The Economist Multi-dimensional ranking - Assesses - Compares - Displays diversity - Does not use aggregate scores - Creates hierarchical lists of higher education institutions U-multirank, CHE University, CHE Excellence, CHE Employment, CHE Research, Indicators for Mapping and Profiling Internationalization Classification - Groups objects with similar characteristics - Describes - Displays horizontal diversity - Considers various activities of higher education institutions U-Map, Carnegie 16

Development of ranking systems Source: Shin J. Ch. , Toutkoushian R. K. , Teichler U. (eds. ) University Rankings: Theoretical Basis, Methodology and Impacts on Global Higher Education. Springer, 2011. P. 14. 17

1 level of analysis: Parameters for comparing ranking methodologies 1. 2. 3. 4. 5. 6. 7. Type (global, regional, national, specialized ranking). Focus (mission, goal, objectives). Target groups: - Target customers (prospective students, parents, academics); - General public (higher education institutions, employers, media, ministries, NGO and charities, expert community). Status (independent, who is the founder etc) Publication frequency (once a year, etc. ). Basic requirements to participants Number of institutions ranked 8. Subject fields (Arts and Humanities, Engineering and Technology, Life Sciences etc. ) 9. Geographical scope 18

1 level: Common parameters for comparing ranking methodologies 10. 11. 12. 13. 14. 15. Education levels (undergraduate/postgraduate). Ranking methodology: - key principles; - indicators, weights; - procedures for data collection; - data processing methods; - data transformation into ranking. Ranking outcome (league table, multi-dimensional ranking, clusters of universities). Criticism and controversy of indicators used in a ranking. Position of the Russian universities in global rankings. Ranking reputation. 19

2 level: Assessing ranking methodologies against Berlin principles and the IREG audit criteria Criteria for assessing ranking methodologies strengths and weaknesses • Berlin principles on ranking of Higher Education Institutions • IREG Ranking Audit Criteria for assessing ranking methodologies 20

2 level: the IREG audit criteria № Criterion Weight PURPOSE, TARGET GROUPS, BASIC APPROACH 1. The purpose of the ranking and the (main) target groups should be made explicit. 2 2. Rankings should recognize the diversity of institutions and take the different missions and goals of institutions into account. 2 3. Rankings should specify the linguistic, cultural, economic, and historical contexts of the educational systems being ranked. 1 METHODOLOGY 4. Rankings should choose indicators according to their relevance and validity. 2 5. The concept of quality of higher education institutions is multidimensional and multi-perspective and “quality lies in the eye of the beholder”. Good ranking practice would be to combine the different perspectives provided by those sources in order to get a more complete view of each higher education institution included in the ranking 1 6. Rankings should measure outcomes in preference to inputs whenever possible. 1 21

2 level: the IREG audit criteria 7. Rankings have to be transparent regarding the methodology used for creating the rankings. 2 8. If ranking are using composite indicators the weights of the individual indicators have to be published. Changes in weights over time should be limited and have to be due to methodological or conceptional considerations. 2 9. Data used in the ranking must be obtained from authorized, audited and verifiable data sources and/or collected with proper procedures for professional data collection following the rules of empirical research. 2 10. The basic methodology should be kept stable as much as possible. 1 11. The publication of a ranking has to be made available to users throughout the year either by print publications and/or by an online version of the ranking. 1 12. The publication has to deliver a description of the methods and indicators used in the ranking. That information should take into account the knowledge of the main target groups of the ranking. 1 13. The publication of the ranking must provide scores of each individual indicator used to calculate a composite indicator in order to allow users to verify the calculation of ranking results. 2 14. Rankings should allow users to have some opportunity to make their own decisions about the relevance and weights of indicators 1 22

2 level: the IREG audit criteria TRANSPARENCY, RESPONSIVENESS 15. Rankings should be compiled in a way that eliminates or reduces errors caused by the ranking and be organized and published in a way that errors and faults caused by the ranking can be corrected. 1 16. Rankings have to be responsive to higher education institutions included/ participating in the ranking. 2 17. Rankings have to provide a contact address in their publication. 1 QUALITY ASSURANCE 18. Rankings have to apply measures of quality assurance to ranking processes themselves. 19. Rankings have to document the internal processes of quality assurance. 1 This documentation has to refer to processes of organising the ranking and data collection as well as to the quality of data and indicators. 20. Rankings should apply organisational measures that enhance the credibility of rankings. MAXIMIM TOTAL SCORE 2 2 180 23

Logical framework for assessing ranking methodologies against the IREG audit criteria Scale for assessment of methodologies against IREG criteria 0 – criterion is not applicable/data is not available 1 – does not comply with the criteria 2 – partially complies with the criteria 3 – fully complies with the criteria 24

25

2 level: Assessing ranking methodologies against the IREG audit criteria Position of methodologies: mean score/normalized mean score 26

2 level: Assessing ranking methodologies against the IREG audit criteria Fulfilment the IREG audit criteria 27

Limitations of methodologies (U-multirank, CHE University) • • • Resource intensity Lack of comparable data on HEIs performance Complexity of indicators and procedures used for data collection Challenges of ensuring reliability of data received from surveys Challenges of ensuring reliability and quality of collecting large volume of data Difficulties associated with processing of large volume of data 28

3 level of analysis: Assessing the indicators against criteria of relevance to the Russian education system development objectives Identification of similar/repeatable/most frequently used and relevant quantitative indicators Analysis of the most frequently used quantitative indicators on the merits of • data availability • distribution of quantitative indicators to areas of evaluation • indicators’ weights Critical assessment most frequently used quantitative indicators against criteria of relevance to the Russian education system development objectives, reliability, feasibility of data collection (see the logical framework for assessment of identified indicators in the next slide) 29

Logical framework for assessment of identified indicators for ranking HEIs methodology Area of evaluation Indicator Relevance for the Russian higher education system development objectives Reliability Availabili ty Relevance to the IREG audit criteria Relevance to the methodology Research. Input indicators Research. Output indicators Teaching and learning. Input indicators Teaching and learning. Output indicators International orientation. Input indicators International orientation. Output indicators Employment/salary Knowledge transfer. Input indicators Knowledge transfer. Output indicators Regional engagement. Input indicators Regional engagement. Output indicators Gender balance Student profile 30

3 level: List of output indicators included into experts’ assessment: Research: ¡ Number of citations per staff ¡ Number of citations per publication ¡ Field-normalized citations score. SСPP/SFCSm, SFCSm – field mean citations score ¡ Number of cited publications ¡ Total number of publications ¡ Number of publications in Nature and Science ¡ Number of publications in SSCI ¡ Number of publications in 40 specified journals ¡ Number of professional publications ¡ MNCS 1 = S (СPP/FCSm) – mean-normalized citation score ¡ Research income ¡ Number of research grants won ¡ Ratio of staff with Ph. D degrees to the number of completed Ph. D theses ¡ International awards and prizes won ¡ Doctorate production ¡ Peer reviewed other research products 31

3 level: List of output indicators included into experts’ assessment (continued): Teaching/learning: ¡ Alumni of an institution winning Nobel Prizes and Fields Medals ¡ Staff of an institution winning Nobel Prizes and Fields Medals ¡ Number of awarded Ph. D degrees per staff ¡ Ratio of awarded Ph. D degrees to awarded Bachelor degrees ¡ Income per staff ¡ Proportion of students graduated with first or second class degree ¡ Graduation rate ¡ Four-year graduation rate ¡ Freshmen to Sophomore retention rate ¡ Number of students winning nationally competitive awards ¡ Listings of Alumni in Who is Who ¡ Alumni in Forbes/CCAP Corporate Officers lists ¡ Graduate unemployment rate ¡ Percentage graduating within norm period 32

3 level: List of output indicators included into experts’ assessment (continued): International orientation: ¡ Ratio of international joint research publications per year to total number of publications per year ¡ Number of joint research publications ¡ International doctorate graduation rate ¡ Number of international research grants ¡ Income from international sources (teaching, research, contracts with international organisations) Knowledge transfer: ¡ Income from business and industry ¡ Income from knowledge transfer activities ¡ Number of university-industry joint research publications ¡ Number of CPD courses per staff ¡ Number of start-ups per staff ¡ Number of joint university-industry patents ¡ Number of cultural events (exhibitions, conferences, concerts), organised by an institution 33

3 level: List of output indicators included into experts’ assessment (continued): Regional engagement: Proportion of income from local/regional sources ¡ Percentage of graduates working in the region ¡ Number of research contracts with regional partners ¡ Number of regional joint research publications ¡ Percentage of students in internships in local enterprises ¡ Number of theses in cooperation with region Employability: ¡ Percentage of employed graduates (six month after graduation) ¡ Percentage of graduates employed through career centres ¡ Graduates salary increase ¡ 34

IV. Next steps: • • Discuss key elements of a template methodology with experts and stakeholders, including areas of evaluation, methods of data collection and indicators’ weights Approbate the template methodology for ranking of higher education institutions Conduct expert and public discussions of the approbation results Conduct consultations with IREG experts on the methodology for ranking of higher education institutions 35

Thank you for your attention! National Training Foundation www. ntf. ru 36