CONSISTENCY OF THE REVIEWS ENQA IQA GROUP Sandra

CONSISTENCY OF THE REVIEWS ENQA – IQA GROUP Sandra Marcos Ortega. ACSUCYL (Spain) 2 nd ENQA Reviewers’ Seminar Manneheim 10 -11 November 2016 1

INDEX Ø WHAT´S ENQA IQA GROUP Ø MISSION Ø OUR WORK Ø SOME FINDINGS Ø WHAT´S NEXT ØDISCUSSION 14. 6. 2021 2

WHAT´S ENQA- IQA GROUP Ø Mandate for the ENQA –IQA group. April 2015 Ø Permanent working group in charge of the internal quality assurance of the agency external reviews. Ø At the service of the Board Ø Six members Ø Aurelija Valeikienė, SKVC (Lithuania) Ø Núria Comet, AQU Catalunya (Spain) Ø Ole Espen Rakkestad, NOKUT (Norway) Ø Julien Lecoq, HCERES (France) ØOana Sarbu, ARACIS (Romania) ØSandra Marcos, ACSUCYL (Spain) Chair 14. 6. 2021 3

ENQA-IQA GROUP MISSION To analyse and provide knowledge on the outcomes of the ENQA external review process (and particularly the selfevaluations and evaluation reports) 14. 6. 2021 replace txt View menu > Header and footer 4

OUR WORK Since April 2015. 3 meetings - Definition of the Work Plan - Familiarising with EQAR interpretation - Understanding and interpretation of ESG 2015 - List of possible evidences. . . 14. 6. 2021 replace txt View menu > Header and footer 5

OUR WORK ENQA understanding and interpretation of ESG 2015 Ø Proposal of ENQA understanding and list of evidences. Ø Analysis of SAR and ERR Ø Two Agencies analysed. First evaluations against new ESG: ØACPUA ØECCE 6

OUR WORK Template for IQA-group’s analysis of the review reports – ECCE and ACPUA ENQA understanding (IQA group’s initial document) Agency interpretation in the SARs (match with ENQA´s understanding? Any additional? ) Evidence provided in the SARs (indicate the number) How the criteria were interpreted by the panel (match with ENQA´s understanding? Any additional? ) How the panel considered the evidences in the external review report? (Everything fully covered? Any additional evidence? Something not considered? ) IQA group´s observation (proposed changes in the initial document) ESG Part 3 3. 1 Activities, policy and processes for quality assurance Agencies should undertake external quality assurance activities as defined in Part 2 of the ESG on a regular basis. They should have clear and explicit goals and objectives that are part of their publicly available mission statement. These should translate into the daily work of the agency. Agencies should ensure the involvement of stakeholders in their governance and work. Emphasis on the regularity - agencies should themselves conduct external QA activities on a regular basis. Organisations that only occasionally organise reviews of institutions or programmes do not comply with the standard. QA should be the main/one of the main activities of the agency. 3. 1. Activities, policy and processes for quality assurance (Evidences) 1. Mission statement, description of values or principles of work. 2. How the mission is presented by the agency to the evaluated bodies. 3. Strategic Plan 4. The Annual Work Plan stating objectives, activities, units/persons responsible, timeline, budget. 5. Monitoring procedures of annual work plan implementation. Corrective actions taken. 6. Web page 7. its mandate. . Match with ENQA´s interpretation? Yes/No If no, how it was interpreted: Additional comments: Match with ENQA interpretation? Yes/No: If no, how it was interpreted: Additional comments: Evidence provided in the SARs (indicate the number): Additional comments: Everything fully covered? Yes/No: If no, specify what wasn´t not considered? Any additional evidence? Yes, No. If yes, please specify: Additional comments: 7

OUR WORK ENQA understanding and interpretation of ESG 2015 Ø Interviews with secretaries involved in external evaluations using new ESG - GUIDING QUESTIONS 1. Transition to the revised ESG: what are the ESG standards the agencies, in your opinion, find the most difficult to understand/to comply with? 2. Which were the most challenging ESG standards to evaluate, from the panel points of view? Why? 8

SOME FINDINGS FROM OUR ANALYSIS: • Difficult to get consistency: differences in the interpretation between the members of the group when making the analysis • Differences in understanding of some standards by Agencies (SAR) • Similar interpretation by the experts but not in the final decisions • Similar interpretation by the Agencies and experts as the first proposal of the ENQA IQA group • 2. 1 and 3. 4 need more reflection on it 9

SOME FINDINGS FROM INTERVIEWS: • Important the context of the Agency • Well appreciate to prepare the review with the secretariat • Interesting that someone understand the local language • Share responsibilities between different members of the panel • 2. 1 and 3. 4 the most difficult • Mission is not always translated into daily work 10

FIRST FINDINGS: SOME EXAMPLES 3. 4. THEMATIC ANALYSIS Agencies should regularly publish reports that describe and analyse the general findings of their external quality assurance activities. ENQA UNDERSTANDING The reports/thematic analyses published by the agencies should be based on findings from their EQA-activities, NOT on other kinds of information about the agency, the performance of the agency, or on data from the higher education sector in general. The reports/analyses should be clearly distinguishable from the agencies’ yearly activity reports. Expected evidence in the agencies’ SAR • Procedure/process description for the production of thematic analyses • Examples of analyses done: reports, analyses, magazine articles, journals, books etc. • Description/documentation of how the agency’s thematic analyses are typically disseminated 11

FIRST FINDINGS: SOME EXAMPLES ECCE Compliance: The ECCE President’s annual report to the general council is published on the ECCE website each year. Additionally, the President prepares reports on ECCE’s activities for the general newsletter of the ECU (‘Backspace’). This report is also available on ECCE’s website. Evidences: No evidences highlighted in the SAR Interpretation of the Standard by ECCE: Understood as annual activities report and accountability reports for Council. Panel Conclusion: Substantially compliance 12

FIRST FINDINGS: SOME EXAMPLES ACPUA Compliance: Ø…producing reports that are useful for HEIs and for the society as a whole, …Evaluation reports. Ø…summary reports on global results at the end of every programme accreditation process. Ø…carrying out studies upon request of the Aragon Government about different key topics, e. g. employability of university graduates, connections of higher education with other education levels, learning outcomes, etc, … Ø…annual report with information about all its activities and results. Ø Report on the status of EQA in Spanish universities, …, summarizing global results of all Spanish QA agencies in different evaluation processes. Ø…design of a new University Information System, defining a set of indicators to reflect the reality with accuracy and reliability and allowing comparative research about the different university systems. Ø… ACPUA’s Board of Directors approved the “Basis for an Excellence Programme” aimed at fostering and recognizing excellence and best practices in the Aragon university system (for both institutions and programmes). Ø Organizes monthly seminars, … as they offer an open forum to reflect, discuss, share ideas and perspectives around quality and improvement in a non-evaluating environment. Ø- Organizes meetings and sessions on different QA topics of interest. It also attends forums and working sessions organized by other institutions … 13

FIRST FINDINGS: SOME EXAMPLES Interpretation of the Standard by the Agency: - To some extend the interpretation match with the Standard - Still including annual activities report , evaluation reports, … Evidences: A list of evidences of each report and seminar organized and describe in the SAR. Panel Conclusion: Substantially compliance 14

WHAT´S NEXT To polish the list of evidences and the interpretation TO IMPROVE CONSISTENC Y ON REVIEWS Meeting in January 2017 Work on a draft of guiding questions and a list of evidences FUTURE… Work on the analysis of other SARs and ERR to improve consistency 15

WHAT´S NEXT Guiding questions for Agencies EXPECTED OUTCOMES Guiding questions for experts List of evidences 16

Thank you!

DISCUSSION 1. DO YOU THINK IT WILL BE USEFUL THE DOCUMENTS WE ARE WORKING ON. 2. IN YOUR OPINION, WHAT WILL BE MORE USEFUL : A LIST OF GENERAL EVIDENCES FOR ALL CRITERIA OR A LIST FOR EACH ONE SEPARATELY? 3. DO YOU FIND ANY CRITERIA MORE DIFFICULT TO UNDERSTAND THAN OTHERS? 18