Understanding the Revised Standards and Guidelines for Quality

Understanding the Revised Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG) Aurelija Valeikienė, Deputy Director, SKVC (Lithuania) ENQA Expert Training 10 -11 November, 2016 Mannheim, Germany 1

Interpretation of ESG – for consistency of reviews from the point of view of ENQA, EQAR, and national authorities; for experts to achieve equal treatment of QA agencies; for agencies to shape themselves 2

Learning from the past – a number of inspirational sources, including ENQA and EQAR decisins, work of KP 3 group and current IQA group 3

Nota Bene: Revised ESGs does contain a number of new items, it is not only re-grouping of standards and text cleaning! Greater attention towards: - student-centeredness - staff (both within HEIs, and peer-review experts) - information provision and publicity (full reports!) - professionalism of agencies at all times 4

There is more for HEIs, e. g. : „The focus of the ESG is on quality assurance related to learning and teaching in higher education, including the learning environment and relevant links to research and innovation. ” “In addition institutions have policies and processes to ensure and improve the quality of their other activities, such as research and governance. “ 5

There is more for QA agencies, e. g. : Developing of professional capacities on ethics and fraud in higher education, more specifically - diploma mills and accreditation mills in order to be able to Conduct “appropriate communication with relevant authorities of those jurisdictions where they operate” and “establish the status and recognition of the institutions with which it conducts external quality assurance“ (ESG 3. 6 Internal quality assurance and professional conduct) 6

There is more for both HEIs and QA agencies: (ESG 1. 4 Student admission, progression, recognition and certification ) „Fair recognition of higher education qualifications, periods of study and prior learning, including the recognition of non-formal and informal learning, are essential components for ensuring the students’ progress in their studies, while promoting mobility. Appropriate recognition procedures rely on - institutional practice for recognition being in line with the principles of the Lisbon Recognition Convention; - cooperation with other institutions, quality assurance agencies and the national ENIC/NARIC centre with a view to ensuring coherent recognition across the country. “ 7

There is more for many actors: „The ESG should be considered in a broader context that also includes qualifications frameworks, ECTS and diploma supplement that also contribute to promoting the transparency and mutual trust in higher education in the EHEA. “ 8

„The ESG apply to all higher education offered in the EHEA regardless of the mode of study or place of delivery. Thus, the ESG are also applicable to all higher education including transnational and cross-border provision. ” Where to draw the line? ─ Is it within the mandate of the national/regional agency to assure quality of its providers abroad? Consequences? ─ Should we expect “the same” procedures or reasonably “the same” outcome? ─ Specificity of distance education QA vs. “regular” study programmes? 9

Where to draw the line? ─ Application of UNESCO/ OECD Guidelines for Quality Provision In Cross-Border Higher Education ─ Application of ENQA / APQN / ANQAHE toolkit for QA agencies 10

ESG Part II – now of equal importance in agency review process as ESG Part III 11

ESG 2. 1 Consideration of internal quality assurance “External quality assurance should address the effectiveness of the internal quality assurance processes described in Part 1 of the ESG. ” Issues to look at: Ø The limits of institutional autonomy and accountability? Ø The agency’s realm of responsibility? “Public authorities should have exclusive responsibility for the provision of the framework within which higher education and research is conducted. National qualifications frameworks and quality assurance mechanisms should be in accordance with the principles of academic freedom and institutional autonomy and take due account of the basic values and variety of purposes of higher education. ” (Recommendation CM/Rec(2007)6 of the Committee of Ministers to member states of the Council of Europe on the public responsibility for higher education and research) 12

ESG 2. 1 Continued Ø who else is on stage? “The National University Act (Organic Law 6/2001 of 21 st December 2002) sets down basic regulations on a national scale establishing the powers and responsibilities of universities, the national government and the different Autonomous Communities’ governments. This law specifies for the first time that quality assurance and quality promotion is an essential goal of higher education policy, and that the functions of evaluation, certification and accreditation belong to “the National Agency for Quality Assurance (henceforth ANECA) and the evaluation bodies determined by each Autonomous Community’s laws”. – SER by Aragon Agency for Quality Assurance and Strategic Foresight in Higher Education (ACPUA), 2015 Ø In case of multiple actors, are there no loopholes? “As outlined by the review panel, the Danish Accreditation Institution is recommended to take appropriate action, so far as it is empowered to do so, on the following issues: ESG 2. 1: To advocate for integrating all aspects of part I of the ESG into the accreditation criteria when the envisaged reform of the accreditation system takes place. ” – ENQA Board decision and letter to Danish Accreditation Institution (AI), 2016 13

ESG 2. 1 stresses institutions’ autonomy and responsibility for quality of their programmes and other provision. The QA agencies are meant to support HEIs in achieving effectively functioning internal QA. The question is “HOW? ” and “TO WHAT EXTENT? ” - Forms of dialogue with institutions (written guidelines, training events. . ) - Consultancy - Benchmarking - Quality labels - Certification (ISO…) - Rating vs. ranking 14

ESG 2. 1 Continued… Issues to look at: Ø If agency applies multiple procedures, how do they correlate? Ø Can the agency provide mapping of its procedures to ESG Part I? How was the mapping done? Ø What is the core of the agency’s criteria and indicators? Are we getting to measure effectiveness? Ø How do things de jure and de facto match? Impact? Ø Evidence of addressing ESG part I in agency’s reports? 15

ESG 2. 2 Designing methodologies fit for purpose “External quality assurance should be defined and designed specifically to ensure its fitness to achieve the aims and objectives set for it, while taking into account relevant regulations. Stakeholders should be involved in its design and continuous improvement. ” To consider: Ø Involvement of stakeholders – academics, students, employers – assumes a large consultative process on design / improvement of EQA Ø Managing interests if stakeholder views on aims and objectives differ – then what? Ø European Approach for Quality Assurance of Joint Programmes is promoted, but it is not a requirement to use one, since this depends on local legislation and several actors (incl. agencies) involved. Ø Flexible ways of external quality assurance when HEIs are able to demonstrate their effectiveness? Risk assessment? 16

ESG 2. 3 Implementing processes External quality assurance processes should be reliable, useful, pre-defined, implemented consistently and published. They include - a self-assessment or equivalent; - an external assessment normally including a site visit; - a report resulting from the external assessment; - a consistent follow-up. ● If the site visits are not a part of the processes – clear reasons? ● There could be various approaches to follow-up, it is up to the agency to determine the nature of the follow-up in light of its mission and as appropriate in its context, BUT the next review should not be interpreted as follow-up! ► Good practice – recommendations, methodologies for HEIs provided by the agency to support follow-up; special events. 17

ESG 2. 4 Peer-review experts “External quality assurance should be carried out by groups of external experts that include (a) student member(s). ” - what is the role of students? How does an agency demonstrate that the students are equal members of the panel? Does it mean they have the same responsibilities as other experts? Are they equally participating in the decision making process? - How wide is the range of experts? Who are in panels depending on the nature of the process? Who are in the pool of the agency? - Is it clear, what the expert selection criteria for each QA activity are? - How it is guaranteed that experts have no conflict of interests? Equal opportunity policy enacted? ► Good practice for cross border QA activities: to mix local and foreign experts (from a variety of national origins / experiences) ► Good practice – foreign experts in various QA activities in small 18 countries.

ESG 2. 5 Criteria for outcomes “Any outcomes or judgements made as the result of external quality assurance should be based on explicit and published criteria that are applied consistently, irrespective of whether the process leads to a formal decision. ” Agencies should have tools to ensure consistency of decision making/outcomes. Reports should demonstrate how the agency ensures consistency of outcomes. Important – training for experts how to apply the criterion (understanding and interpretation) Balance – taking into account the specific context of the institution vs. consistency 19

ESG 2. 6 Reporting “Full reports by the experts should be published, clear and accessible to the academic community, external partners and other interested individuals. If the agency takes any formal decision based on the reports, the decision should be published together with the report. ” - The publication of summary reports rather than full reports does not fulfill the requirement of the standard. - Full reports should be published also for cross-border QA activities if those activities are within the scope of the ESG (even if voluntary). - promotion of excellence? ► Good practice – it is up to the agency to prepare a summary report, but this could be recommended (added value). ► Good practice – it is not compulsory to give the opportunity to HEIs to point out the errors of fact in the reports, but the benefit is 20 underlined as it improves reliability.

ESG 2. 7 Complaints and appeals “Complaints and appeals processes should be clearly defined as part of the design of external quality assurance processes and communicated to the institutions. ” - consistently handling any dissatisfactions, e. g. requests to change a reviewer - possibility to appeal any formal decision – an agency must have a predefined process for that; but what happens if formal decisions are taken elsewhere? - even if the procedure leads to recommendations and no decisions anywhere, there should be a possibility to contest the outcomes or the procedures - pre-court institutions vs. courts - number of cases and their outcomes – is it “healthy”, does it convince, that the process is really working, not just formally in 21 place?

ESG Part I: not to be addressed directly, but via ESG 2. 1 22

ESG 1. 1 -1. 10 – not exactly work of QA agency, but: - ESG 1. 1 Policy for QA should include systemic attention to fraud - ESG 1. 2 Design and approval of programmes – reference to framework of qualifications of the EHEA - Implementation of ESG 1. 3 Student-centered learning, teaching and assessment – an area of work for the next decade for many countries - ESG 1. 4 Student admission, progression, recognition and certification – a challenge of relating QA and recognition, working with ENICs/NARICs - ESG 1. 5 Teaching staff – also talking about administrative staff! - ESG 1. 6 Learning resources and student support – need for further attention - ESG 1. 7 Information management – further developing area - ESG 1. 8 Public information – our monitoring and recommendations several years in a row - ESG 1. 10 Cyclical external quality assurance – establishing communication with other EQAR-listed agencies 23

ESG Part III: applies both to „regular“ activities and crossborder quality assurance processes 24

Comparison of ESG-2005 and ESG-2015 has a practical consequence both for agencies and review panels, as it has to be documented in the report 25

ESG 3. 1 Activities, policy and processes for quality assurance “Agencies should undertake external quality assurance activities as defined in Part 2 of the ESG on a regular basis. They should have clear and explicit goals and objectives that are part of their publicly available mission statement. These should translate into the daily work of the agency. Agencies should ensure the involvement of stakeholders in their governance and work. ” - Emphasis on the regularity - agencies should themselves conduct external QA activities on a regular basis. Organisations that only occasionally organise reviews of institutions or programmes do not comply with the standard. QA should be the main/one of the main activities of the agency. - linking of mission, strategy and annual work plans? - publicity? - stakeholder involvement in all stages and aspects? 26

ESG 3. 2 Official status “Agencies should have an established legal basis and should be formally recognised as quality assurance agencies by competent public authorities. ” - This can be by-passed if there is a requirement in the law to be first registered on EQAR in order to be formally recognised by a (national) public authority. - any kind of recognition abroad? - formal mandate to conduct cross-border activities - participation in international organisations 27

ESG 3. 3 Independence “Agencies should be independent and act autonomously. They should have full responsibility for their operations and the outcomes of those operations without third party influence. ” Look up for EQAR interpretation! - Where an agency's independence is not obvious from its structures and status, it should be considered in greater detail how operational independence is safeguarded in practice. - If the agency has other activities than external QA (e. g. seminars, consultancy), adequate policies and processes should be in place to safeguard independence of the respective organisational units in performing their QA functions. 28

ESG 3. 4 Thematic analysis “Agencies should regularly publish reports that describe and analyse the general findings of their external quality assurance activities. ” - It should be stressed that reports must be about general findings of EQA activities not about any other kind of information or the performance of an agency: - programme level - institutional level - national context, international context - past-performance vs. forward look, improvement of policies and processes - must be different from yearly activity reports. 29

ESG 3. 5 Resources “Agencies should have adequate and appropriate resources, both human and financial, to carry out their work. ” - Agencies should demonstrate a balance between financialhuman resources and the volume of their activity. - Agencies should have enough resources to carry on their mission and to improve continuously (e. g. investment in staff development). - Comprehensive understanding – resources include physical conditions, material and financial resources. 30

ESG 3. 6 Internal quality assurance and professional conduct “Agencies should have in place processes for internal quality assurance related to defining, assuring and enhancing the quality and integrity of their activities. ” - There should be clear separation between ESG activities and other activities - Agency’s activities out of the scope of the ESG: to avoid confusion, agencies have to be open and honest that they are doing this other way. If they do not follow the ESG in some activities, how do they make this clear to public? - formal mechanisms in place and actual implementation? - managing risks and reputation? 31

ESG 3. 7 Cyclical external review of agencies “Agencies should undergo an external review at least once every five years in order to demonstrate their compliance with the ESG. ” - is it mandatory by legislation or voluntary? - how did the agency use the results of previous external review? - if some panel recommendations were not followed upon, why? 32

No progression AD INFINITUM! QA agencies cannot be held hostages of circumstances beyond their reach. Agencies are not the only actors in national/regional education stirring systems. higher HEIs are autonomous entities! The context is important. 33

Quality of a QA agency ≠ quality of a national / regional external QA system Quality of a QA agency ≠ quality of HEIs and their provision 34

Triangulation method to be used: • analysis of written documentation (not only procedures, but content as well); • juxtaposition with oral evidences on practices (agency staff and stakeholders); • forming experts’ own opinion, discussing and taking judgment. 35

Joint responsibility of the expert panel in arriving at weighted judgments by consensus! 36

Thank you! aurelija@skvc. lt avaleikiene@gmail. com 37

ESG 2. 1 Mapping of ESG-2015 implementation: what evidence do we expect? 38

Addressing internal quality assurance within HEIs: HOW? Ø Developing a set of guidelines, but beware of directing and “coaching” Ø Assisting institutions in development of quality units Ø Development of various procedures and running them on the same system (programme level + institutional level) Ø Specific procedures for IQA: • quality audit • Accreditation / evaluation of teaching staff • Focus on LOs (subject-specific agencies) Ø Including in the panel a reviewer with specific expertise in quality management 39

Quality assurance is not a goal in itself. Similarly so – the external review of QA agencies. Essential principle of the ESG: fitness for purpose! 40