WRITING A GOOD EVALUATION REPORT SUPPORT TO STRENGTHENING

WRITING A GOOD EVALUATION REPORT SUPPORT TO STRENGTHENING THE HIGHER EDUCATION SYSTEM IN AZERBAIDJAN OCTOBER 2019 Training session N 0. 2 Eliane Kotler Michelle Houppe

General Steps of Procedures 1. Self-assessment process 2. Submission of the application and self-assessment report 3. Review of the documents by the Agency 4. Composing the experts group 5. Experts work before the site visit 6. Visit of the experts group 7. Joint report of the experts group 8. Comments by HEI on the factual errors 9. Decision by the Committee (with the HEI present) 10. Follow-up procedures

WRITING A GOOD REPORT TITRE PRINCIPAL DE LA PRÉSENTATION / Prénom Nom / JJ/MMM/AAAA PAGE 3

WRITING A GOOD REPORT 1 2 SER Visit 3 External Report TITRE PRINCIPAL DE LA PRÉSENTATION / Prénom Nom / JJ/MMM/AAAA PAGE 4

WRITING A GOOD REPORT SER visit report The report relies on the SER: • to confirm ou infirm the self-analysis (double check) • assess the quality of the whole SE approach TITRE PRINCIPAL DE LA PRÉSENTATION / Prénom Nom / JJ/MMM/AAAA PAGE 5

WRITING A GOOD REPORT A - THE CONTENT The report must be useful - Allow the evaluated program to assess its position (national environment) - Help to take decision and manage the program REQUISITE: è give an appreciation about the quality of the program è rely on facts and figures (for example with tables in attachment) è give a clear and verified picture, thanks to the visit è formulate recommendations TITRE PRINCIPAL DE LA PRÉSENTATION / Prénom Nom / JJ/MMM/AAAA PAGE 6

WRITING A GOOD REPORT N+x years today N-x years The information gathered (SER + visit) allows the evaluators to: • Assess the quality of the program • Evaluate its trend, its trajectory • Give recommendations for the future TITRE PRINCIPAL DE LA PRÉSENTATION / Prénom Nom / JJ/MMM/AAAA PAGE 7

WRITING A GOOD REPORT B - THE WRITING precise, concise descriptive respectful repetition Well argued redundant TITRE PRINCIPAL DE LA PRÉSENTATION / Prénom Nom / JJ/MMM/AAAA PAGE 8

KEY POINTS OF THE REPORT HAS TO BE OBJECTIVE, EVIDENCE BASED, USEFUL • OBJECTIVE • EVIDENCE BASED • USEFUL No personal jugement The report has to provide references to the SER, to datas given The recommendations have to be based on the strengths and weaknesses. They have to be realistic, feasable and understandable. METHODOLOGY - Sum up your notes and compare to your needs - Draft your contribution : Ø No personalization, no direct critical view on a person TITRE PRINCIPAL DE LA PRÉSENTATION / Prénom Nom / JJ/MMM/AAAA PAGE 9

STRUCTURE OF THE REPORT I. Institution Identity Sheet II. Introduction which precises the main features of the programme and its position within the institution III. On-site visit description Presentation of the institution's self-evaluation approach Composition of the panel of experts On-site visit description III. Analysis of the study programme: 1. Programme aims and learning outcomes 2. Curriculum design 3. Teaching staff 4. Facilities and learning resources 5. Study process and students‘ performance assessment 6. Programme management The strengths and the weaknesses have to be highlighted for each evaluation area Recommendations have to be provided IV. CONCLUSION General Strengths, Weaknesses and Recommendations TITRE PRINCIPAL DE LA PRÉSENTATION / Prénom Nom / JJ/MMM/AAAA PAGE 10

STUDY PROGRAMME ANALYSIS 6 evaluation areas. For each evaluation area: Comments (analyses) A short description of the situation in the study programme, based on elements from the self-evaluation report and on findings from the site-visit indicating the compliance and/or not-compliance with standards. Strengths A list of strengths – examples of good practices, excellent achievements, innovative solutions. . Weaknesses A list of weaknesses – problem areas (and non-compliances, if applicable) to be dealt with. Recommendations of how to address the weaknesses and on further improvements, for elimination of the deficiencies found, as well as long-term recommendations for further improvement of the study programme. Evaluation results TITRE PRINCIPAL DE LA PRÉSENTATION / Prénom Nom / JJ/MMM/AAAA PAGE 11

EVALUATION STANDARDS IN A NUTSHELL s rd o yw Ke WHAT DO YOU NEED TO ASSESS ? WHAT ARE YOU TRYING TO DEMONSTRATE ? Ø IN ORDER TO EVALUATE IF THE PROGRAM IS: • compliant: A • mostly compliant: B • partially compliant: C • non-compliant: D TITRE PRINCIPAL DE LA PRÉSENTATION / Prénom Nom / JJ/MMM/AAAA PAGE 12

1. PROGRAMME AIMS AND LEARNING OUTCOMES rds w ey o K Internal & external coherence Relevance with the context / environment TITRE PRINCIPAL DE LA PRÉSENTATION / Prénom Nom / JJ/MMM/AAAA PAGE 13

2. CURRICULUM DESIGN s y Ke rd wo Content is relevant Adapted and innovative TS C E TITRE PRINCIPAL DE LA PRÉSENTATION / Prénom Nom / JJ/MMM/AAAA PAGE 14

3. TEACHING STAFF y Ke rd wo s qualified adequate stable research-oriented fairly recruted TITRE PRINCIPAL DE LA PRÉSENTATION / Prénom Nom / JJ/MMM/AAAA PAGE 15

4. FACILITIES AND LEARNING RESOURCES Adequate to the needs Sufficient funding TITRE PRINCIPAL DE LA PRÉSENTATION / Prénom Nom / JJ/MMM/AAAA PAGE 16

5. STUDY PROCESS AND STUDENTS’PERFORMANCE ASSESSMENT Student-centered approach: support, mobility, research Clear assessment T N E D U S T S S E C C U S TITRE PRINCIPAL DE LA PRÉSENTATION / Prénom Nom / JJ/MMM/AAAA PAGE 17

6. PROGRAMME MANAGEMENT y Ke rd wo s The structure deals with its responsibilities in terms of: Accountability (providing datas) Quality assurance Involvement of stakeholders Partnerships TITRE PRINCIPAL DE LA PRÉSENTATION / Prénom Nom / JJ/MMM/AAAA PAGE 18

DRAFTING SCHEME y Ke finding facts rd wo s solution TITRE PRINCIPAL DE LA PRÉSENTATION / Prénom Nom / JJ/MMM/AAAA PAGE 19

PROVISIONAL REPORT DRAFTING 1. EACH EXPERT WRITES A COMMENT ABOUT HIS OWN FIELD OF INTERVIEW: • • Not too long (length to be defined) Clear Objective: evaluative but evidence-based Argued 2. THE PRESIDENT OF THE COMMITTEE READS THE ENTIRE REPORT (AND SOMETIMES REWRITES SOME PARTS OF IT) 3. THE REPORT IS SIGNED BY THE PRESIDENT The name and quality of the experts are precised on the report (at the beginning or at the end) TITRE PRINCIPAL DE LA PRÉSENTATION / Prénom Nom / JJ/MMM/AAAA PAGE 20

BEST PRACTICES • AVOID CONDITIONAL SENTENCES, DELIVER A CLEAR MESSAGE • DO NOT BE TOO PRESCRIPTIVE; • AVOID USING SPECIFIC CULTURAL CONTEXTUAL REFERENCES; • DO NOT EXTEND SENTENCES ACROSS MANY LINES. TITRE PRINCIPAL DE LA PRÉSENTATION / Prénom Nom / JJ/MMM/AAAA PAGE 21

BEST PRACTICES • EVALUATION REPORT SHOULD NOT REPEAT INFORMATION PROVIDED IN THE SER: NO COPY/PASTE • EVALUATION REPORT SHOULD NOT BE DESCRIPTIVE BUT ANALYTICAL • FACTUAL INFORMATION ABOUT THE STUDY PROGRAMME SHOULD BE CORRECT • NOT ONLY WEAKNESSES BUT ALSO STRENGTHS SHOULD BE EMPHASIZED • ENOUGH ATTENTION SHOULD BE DEVOTED TO THE CONCLUSIONS • ATTENTION TO BE PAID ABOUT HOW RECOMMENDATIONS OF PREVIOUS EVALUATION WERE IMPLEMENTED. TITRE PRINCIPAL DE LA PRÉSENTATION / Prénom Nom / JJ/MMM/AAAA PAGE 22

THE EVALUATION REPORT DRAFTING -1 THE REPORT HAS TO BE THE RESULT OF A COLLEGIAL ANALYSIS. AT THE END OF THE ON-SITE VISIT, THE EXPERTS HAVE A FINAL DEBRIEFING (3 HOURS) TOGETHER TO DEFINE THE STRENGHTS, WEAKNESSES AND RECOMMENDATIONS. THEN, EVERY EXPERT CAN DRAFT HIS/HER OWN PART, TAKING INTO ACCOUNT THIS COLLEGIAL POSITION TITRE PRINCIPAL DE LA PRÉSENTATION / Prénom Nom / JJ/MMM/AAAA PAGE 23

THE EVALUATION REPORT DRAFTING -2 • Provisional report drafting Total: 1 month • Internal editorial review : • Give your opinion/feedback on the others’ contribution • Each expert contributes to the conclusion of the report • Provisional report last drafting TITRE PRINCIPAL DE LA PRÉSENTATION / Prénom Nom / JJ/MMM/AAAA PAGE 24

THE EVALUATION REPORT DRAFTING -3 • Provisional report sent to HEI 10 days • The HEI makes comments : • Only factual mistakes that have to corrected • No comment about the quality of the evaluation • Final report drafting • Contribute to the accreditation proposal • Final report sent to HEI • Publication of report + comments on the website TITRE PRINCIPAL DE LA PRÉSENTATION / Prénom Nom / JJ/MMM/AAAA PAGE 25

BEGINNING OF THE REPORT, 2 EXAMPLES 1 - PRESENTATION OF THE INSTITUTION’S SELF-EVALUATION APPROACH After the first contacts with ANB staff, by …. , xxx University set up a dedicated committee around the Head of Education Department and the head of School of Education, in order to monitor the self-evaluation process. Five groups of staff were created, and 10 meetings were held between march and april 2019. The self-evaluation report, dated june 2019, is a rather consistent document, with some redundancies, and some lacking information. All in all, it is rather well organized, and clearly written. It follows the various areas and fields of the AZ external evaluation standards framework, and addresses all criteria. However the indicators corresponding are not always informed. The SWOT analysis of each criteria is uneven: some criteria have been analysed in a deep way, others are less complete. TITRE PRINCIPAL DE LA PRÉSENTATION / Prénom Nom / JJ/MMM/AAAA PAGE 26

BEGINNING OF THE REPORT, 2 EXAMPLES 1 - PRESENTATION OF THE INSTITUTION’S SELF-EVALUATION APPROACH After the first contacts with ANB staff, by …. , xxx University set up a dedicated committee around the Head of the Department, The Deputy Dean of the Faculty and the Head of the department of Az …, in order to monitor the self-evaluation process. Seven groups of staff were created, and 10 meetings were held between april and may 2019. The self-evaluation report, dated june 2019, is a rather consistent document, with some redundancies, and some lacking information. It is nevertheless well organized, and clearly written. It follows the various areas and fields of the AZ external evaluation standards framework, and addresses in detail all criteria. It reveals the excellent state of mind of the institution as regards to the process of selfevaluation, and its good understanding and awareness of the defined criteria. The SWOT analysis of each criteria is quite complete and detailed, and seems to have been an opportunity for a deep self-critical analysis. The number of demonstrations and regulations listed in annex is consistent. The well-developed quality management culture of the programme seems to be an important asset for a pertinent self-evaluation approach. TITRE PRINCIPAL DE LA PRÉSENTATION / Prénom Nom / JJ/MMM/AAAA PAGE 27

RECOMMENDATION EXAMPLE : TEACHING STAFF Weakness : Difficulty in Az in finding good teachers, related to the fact that salaries outside the university are higher than salaries inside the university, which discourages many researchers. Recommendations : - One solution to the issue of lack of good teachers would be to recruit more professors coming from the professional sector to work part-time at the university Weakness: Lack of good teachers Recommendations : - The institution has to find solutions to solve this very important problem TITRE PRINCIPAL DE LA PRÉSENTATION / Prénom Nom / JJ/MMM/AAAA PAGE 28

RECOMMENDATION EXAMPLE : STUDENTS MOBILITY PROGRAMMES Weakness : Mobility is indeed very difficult for students incoming students as well as outgoing (0 incoming students / 3 outgoing in 2017 -2018). The SER provides explanations for this issue that are not very convincing : the obligation for the students to work to earn money. After the visit, it appears that there are other reasons such as the low level of foreign language and the low financial support. Recommendations : - a strong support should be provided for the practice of EN language (increase the number of teaching hours in oral practice) - the university officials should develop Erasmus+ projects Weakness: Mobility is indeed very difficult for students incoming students as well as outgoing. The SER provides explanations for this issue. Recommendations : - The institution has to find solutions to solve this very important problem

WORKSHOP – ASOIU COMPUTER ENG. CRITERIA 1 WRITE A REPORT ABOUT CRITERIAS: • 1. 1 • 1. 2/1. 3 • 1. 5 • 1. 6 TITRE PRINCIPAL DE LA PRÉSENTATION / Prénom Nom / JJ/MMM/AAAA PAGE 30

HOW TO WRITE: 1/FINDINGS 2/ANALYSIS 3/RECOMMENDATIONS 1. 1 • EVERY INFORMATION REQUIRED ABOUT THE TEACHING DOCUMENTS IS PROVIDED AND AVAILABLE ON THE WEBSITE. HOWEVER THE AIMS OF THE STUDY PROGRAM ARE NOT REALLY DESCRIBED. • AFTER THE VISIT, IT APPEARS THAT… • THE COMMITTEE SUGGESTS TO PROVIDE A PRECISE DESCRIPTION OF THE GOALS OF THE PROGRAMME AND EMPLOYMENT OPPORTUNITIES FOR GRADUATES, IN ORDER TO INFORM STUDENTS AND STAKEHOLDERS. THE BEST WAY WOULD BE PUBLICITY ON THE WEBSITE AND DURING JOB FAIRS. TITRE PRINCIPAL DE LA PRÉSENTATION / Prénom Nom / JJ/MMM/AAAA PAGE 31

HOW TO WRITE THE REPORT 1. 2/ 1. 3 • THE EXPECTED LEARNING OUTCOMES ARE SPECIFIED, WITH A LOT OF DETAILS. THE COMMITTEE WONDERS IF ALL THESE COMPETENCIES WERE ACTUALLY ACQUIRED AT BA LEVEL. THE COMPARISON WITH MA COMPETENCIES SHOWS … • THE LINKS WITH THE LABOR MARKET ARE NOT SPECIFIED IN A CONVINCING WAY. • AFTER DISCUSSION WITH EMPLOYERS, THE COMMITTE BELIEVES THAT… TITRE PRINCIPAL DE LA PRÉSENTATION / Prénom Nom / JJ/MMM/AAAA PAGE 32

HOW TO WRITE THE REPORT 1. 5 • THE REGIONAL CONTEXT IS NOT ANALYZED: AS A CONSEQUENCE, THE COMMITTEE CANNOT EVALUATE IF THE LEARNING OUTCOMES ARE ON LINE WITH THE DEVELOPMENT OF THE BUSINESS ACTIVITIES IN THE AREA. • AFTER MEETING WITH THE MEMBERS OF THE BOARD, THE COMMITTEE RECOMMENDS TO DEVELOP PERIODICAL INTERNSHIPS AND PRACTICAL LESSONS. A FINAL REPORT COULD BE EVALUATED BY A MIXED JURY PANEL: TEACHING STAFF AND PROFESSIONAL TOGETHER. TITRE PRINCIPAL DE LA PRÉSENTATION / Prénom Nom / JJ/MMM/AAAA PAGE 33

HOW TO WRITE THE REPORT 1. 6 • EMPLOYABILITY IS A REAL CONCERN FOR THE PROGRAMME. HOWEVER, THE COMMITTEE DOES NOT KNOW HOW THE CAREER CENTER OPERATES IN AN EFFECTIVE WAY. • AFTER THE VISIT, … • THE COMMITTEE RECOMMENDS TO PUBLISH JOB AND TRAINING OFFERS, TO TRAIN STUDENTS TO WRITE CV AND BEHAVE DURING JOB INTERVIEWS. TITRE PRINCIPAL DE LA PRÉSENTATION / Prénom Nom / JJ/MMM/AAAA PAGE 34

CONCLUSION: 1. STRENGTHS 1. A QUITE GOOD USE OF THE WEBSITE, WHICH NEEDS TO BE FURTHER DEVELOPED 2. THE COMPETENCE APPROACH IS EFFECTIVE 3. THE ISSUE OF EMPLOYABILITY IS TAKEN INTO ACCOUNT TITRE PRINCIPAL DE LA PRÉSENTATION / Prénom Nom / JJ/MMM/AAAA PAGE 35

CONCLUSION: 2. WEAKNESSES 1. THE LABOR MARKET IS NOT SUFFICIENTLY INVOLVED IN THE PROGRAM: • Design of the program • Update of the content • Internships opportunities 2. THE GOALS OF THE PROGRAM ARE NOT DEFINED IN A PRACTICAL AND SATISFACTORY WAY TITRE PRINCIPAL DE LA PRÉSENTATION / Prénom Nom / JJ/MMM/AAAA PAGE 36

CONCLUSION: 3. RECOMMENDATIONS • THE GOALS OF THE PROGRAM HAVE TO BE DEFINED MORE CLEARLY AND MADE PUBLICLY AVAILABLE. • AN OVERALL VIEW OF THE GLOBAL PICTURE NEEDS TO BE PROVIDED. • THE LINKS WITH EMPLOYERS NEED TO BE STRENGHTENED: INVITE THEM TO STAND IN THE BOARD/COUNCIL OF IMPROVMENT, IN JURY PANELS, OR TO GIVE LESSONS. TITRE PRINCIPAL DE LA PRÉSENTATION / Prénom Nom / JJ/MMM/AAAA PAGE 37