OAAA Oman Academic Accreditation Authority OAAA Institutional Standards

OAAA Oman Academic Accreditation Authority (OAAA) Institutional Standards Assessment Workshop A training module in preparation for Institutional Standards Assessment 4 June 2015

Aims of the Workshop OAAA To provide participating HEI staff with an overview of the OAAA’s role in Institutional Standards Assessment To provide a guide for the process of the Institutional Standards Assessment and its link to sections of the Standards Assessment Manual To provide staff with a clear idea of their role in the Institutional Standards Assessment process To provide guidance on how to carry out a selfassessment and prepare an Institutional Standards Assessment Application including the methods of analysis and ADRI To provide participating HEI staff with a chance to ask questions and clarify understanding

Program OAAA Session 1 8. 30 -8. 45 Introduction OAAA’s role in Institutional Accreditation Session 2 8. 45 -9. 30 Institutional Standards Assessment Manual and Process How to read the Standards Assessment Manual (SAM) Standards Assessment Process Timeline Session 3 How to Carry Out a Self-assessment/Prepare an 9. 30 -10. 30 Institutional Standards Assessment Application (ISAA) ISAA Template overview 10. 30 -10. 45 Break Session 4 Activity 1 10. 45 - 11. 45 Types of evidence to review when preparing an ISAA Session 5 Methods of Analysis/Using ADRI for 11. 45 -12. 45 Introduction of methods of analysis for Institutional Standards Assessment including the use of ADRI as an analytical tool to support self-assessment 12. 45 -1. 30 Lunch Session 6 1. 30 -2. 30 Session 7 2. 30 -3. 15 Activity 2 Rating criteria and standards Recap and Tips Features of a good ISA product and process Preparing for a smooth ISA Session 8 3. 15 -3. 45 Questions Opportunity for OAAA to provide further clarifications Dr Salim Radhawi CEO, OAAA Dr Anna Scopaz Quality Assurance Consultant Dr Salim Radhawi CEO, OAAA Team Dr Tess Goodliffe Deputy CEO, Technical Affairs OAAA Team Ms Susan Trevor. Roper Senior Quality Assurance Expert OAAA Team

OAAA Roles & Responsibilities Developing a system for institutional and program accreditation Accrediting public and private HEIs and academic programs Conducting Quality Audits of HEIs Establishing a procedure for recognising foreign programs offered in Oman Signing mutual recognition Mo. Us with external QA agencies Updating and maintaining the OQF

HEI QA Processes OAAA HEI Licensure First cycle commenced 2008 HEI Accreditation Stage 1: Quality Audit ≤ 4 years HEI Accreditation Certificate Met HEI Accreditation Stage 2: Standards Assessment Appeal Not met KEY 1 -2 years on Probation Start/End Met Process Document HEI Standards Reassessment HEI Accreditation Terminated

OAAA Difference between Quality Audit and Institutional Standards Assessment Quality Audit Standards Assessment Evaluates the HEI’s effectiveness against its own stated Mission and goals against a set of nine national standards which are applied to every HEI operating in Oman Formative in nature Summative in nature Results in Recommendations, Affirmations and Commendations which help to develop the emerging HE sector in Oman Results in Accreditation Outcome Quality Audit Report made public Standards Assessment Report not made public Accreditation Outcome and ratings against standards and criteria made public

OAAA Any questions?

OAAA Session 2: Institutional Standards Assessment Manual and Process A training module in preparation for Institutional Standards Assessment

OAAA Session aims • To gain an understanding of the structure of the Standards Assessment Manual (SAM) • To learn how to make best use of the SAM • To learn about the key milestones in the Institutional Standards Assessment process

OAAA Standards Assessment Manual (SAM) Part A Institutional Accreditation Overview Part B Institutional Standards Part C Accreditation Outcomes and Ratings against Standards and Criteria Part D The Self-assessment Part E The Institutional Standards Assessment Part F Methods of Analysis Part G Appendices See SAM Introduction, Table 1

OAAA SAM Part B Institutional Standards STANDARD 1: GOVERNANCE AND MANAGEMENT Governance and management of the HEI is ethical and ensures implementation of academic and non-academic systems and functions which support achievement of the HEI’s Mission and Vision and the protection of academic standards. Governance and Management structures, processes and mechanisms for accountability are appropriate. . . 9 Standards Only address those standards which are relevant to your HEI Criterion 1. 1: Mission, Vision and Values The Mission, Vision and Values are appropriate, have been developed in consultation with stakeholders, formally approved and guide the HEI in all its activities. The HEI has Mission and Vision statements which clearly define the HEI’s purpose, whom it serves and what it intends to accomplish; and these align with the national priorities of Oman. The HEI has a defined set of institutional Values. Indicators a. The Mission, Vision and Values effectively guide the HEI, are consistent with the HEI’s purpose and its ability to meet the national priorities of Oman, and community expectations. b. The governing body has formally approved the Mission, Vision and Values. c. Key stakeholders have been consulted and support the Mission, Vision and Values. d. The Mission, Vision and Values are readily accessible and effectively communicated to stakeholders. e. The Mission, Vision and Values are regularly reviewed and reaffirmed or amended as appropriate in order to maintain relevance and effectiveness in guiding the HEI. 79 Criteria Address those criteria which are relevant to your HEI (state why others are Not Applicable) 4 -6 Indicators per criterion. Provided as guidance NOT requirements

OAAA SAM Part C • Accreditation Outcomes and Ratings against Standards and Criteria Met (ratings 2, 3, 4) Partially Met (rating 1) Not Met (rating 0) Excellent Good Met Not Met Accredited with Merit in one or more standards Accredited On Probation Not Accredited Criteria Ratings Standards Ratings Accreditation Outcomes See SAM section 5

OAAA SAM Part D The Self-assessment • HEIs are expected to undertake a selfassessment well before the ISAA submission date • Self-rate HEI performance against criteria; use this to inform standard ratings • Commentary must match ratings • Respond to Quality Audit Report Commendations, Affirmations and Recommendations (CARs) See SAM section 10

Workshop Activity OAAA The ISA indicative timeline highlights key tasks for: – – OAAA Executive Officer Panel HEIs Identify which activities are HEI responsibilities We’ll check throughout the session

OAAA Prior to ISAA submission SAM Part E The Institutional Standards Assessment ISAA submission Preliminary Meeting Planning Visit SA Visit Report v 5 OAAA Board approves outcome Outcome posted on OAAA website See SAM section 3. 4

OAAA ISAA Submission • Ensure all (relevant) criteria have commentary and ratings • Ensure all Supporting Materials (SMs) are attached • Ensure declaration is signed • Nominate a Contact Person (this may happen earlier) See SAM section 11. 4

OAAA Receives ISAA • Undertakes a ISAA completion check – If not complete the ISAA will be returned to the HEI – If complete, ISAA and SMs forwarded to Standards Assessment Panel • Invoice issued See SAM section 14. 3

OAAA Preliminary Meeting About 6 weeks after the ISAA is submitted • Standards Assessment Panel meets to discuss ISAA (international Panel Members via teleconference) • Provides opportunity to focus attention on ISAA • Panel discusses preliminary ratings and commentary • Ensures Panel Members are very clear about tasks and expectations • Helps with planning for rest of the Standards Assessment See SAM section 18. 2

OAAA Planning Visit About 7 weeks before the Standards Assessment Visit Provides an opportunity: • To facilitate the overall Standards Assessment Visit • To request additional information • To identify individuals Panel would like to meet • To discuss logistics See SAM section 18. 5

OAAA Standards Assessment Visit About 7 weeks after Planning Visit Provides an opportunity for the Panel to verify whether the HEI’s ISAA is: • Adequate in addressing all relevant standards and criteria • Comprehensive in scope • Accurate and complete • Defensible in ratings against criteria and standards See SAM section 20

Draft Reports OAAA Draft Report v 1 Draft report v 2 Draft Report v 3 Draft Report v 4 Prepared based on Panel’s provisional ratings and preliminary comments Written after the Standards Assessment Preliminary Meeting and incorporates Panel discussion Written on last day of Standards Assessment Visit. Panel reaches consensus on standard and criteria ratings and provide precise, evidence-based commentary to support ratings Cross-checked against existing evidence. Undergoes internal and external moderation

Draft Reports OAAA Draft Report v 5 Draft Report v 6 Final Report • Feedback from internal and external moderation is incorporated • Report v 5 sent to HEI for feedback on factual inaccuracies; and to OAAA Board approves accreditation outcome and ratings against standards and criteria Report sent to HEI under embargo for 10 days See SAM section 21. 6

Public Reporting OAAA 2 weeks after Final Report sent to HEI • Accreditation Outcome and ratings against the standards and criteria posted on OAAA website • Accreditation Certificate awarded to successful HEI See SAM section 21. 6

OAAA Prior to ISAA submission After Accreditation Outcome Release ISAA submission Preliminary Meeting Planning Visit SA Visit Report v 5 OAAA Board approves outcome Outcome posted on OAAA website Stakeholder Feedback Sought OAAA seeks feedback from stakeholders on all aspects of the Institutional Standards Assessment process See SAM section 23

OAAA Any questions?

OAAA Session 3: Preparing the Institutional Standards Assessment Application (ISAA) A training module in preparation for the Institutional Standards Assessment

OAAA Session Aims To be aware of how to complete the ISAA To be aware of what self-rating involves To understand how to determine the standard rating To understand how to determine the Accreditation Outcome To understand how fees are calculated 27

OAAA Institutional Standards Assessment Application (ISAA) • The template will be available electronically • The same template will be used by the Standards Assessment Panel – The HEI provides commentary explaining how it has met each criterion and why its self-rating against each criterion is appropriate – The Panel provides commentary against each criterion 28

OAAA Institutional Standards Assessment Application (ISAA) • The ISAA includes: – Introductory section in which the HEI provides an overview of its history and context – Declaration Form – Standards 1 -9 & all criteria (HEIs provide commentary against each relevant standard and criteria) – A list of Supporting Materials – Summary Data Tables (see SAM Appendix P) 29

OAAA Required Supporting Materials must be submitted with the ISAA • Institutional establishment or licensing documentation (such as Decree establishing the HEI or similar) • Official licenses for all programs offered • Current catalogue (prospectus or similar) • Most recent HEI Annual Report • A campus map See SAM section 13. 3. 5 30

OAAA Declaration to be signed by HEI’s most senior representative stating that: • the information contained in the application is complete and accurate • the HEI meets all national laws and requirements not specified in OAAA Institutional Standards • the HEI abides by health and safety regulations and the OAAA holds no responsibility in this regard (this is still under discussion) See SAM section 11. 2 31

HEI Overview OAAA Include a brief history, campus location/s, a general description of the HEI and its context, and any special characteristics it may have. The description should include the academic and general structure of the organisation and a complete list of programs being offered, including details of their licensing and accreditation status (and if the HEI is not the body awarding the qualifications, which entity is the awarding body). Relevant data should be included in the Summary Data Tables. The overview should provide a context for the HEI. It should be brief and succinct See SAM section 13. 3. 1 32

Standards 1 -9 OAAA • In the template, each standard and criterion is written out in full: Standard text Rating HEI Commentary Criterion text Rating HEI Commentary • Commentary includes HEI responses to Affirmations and Recommendations in the Quality Audit Report and how Commendations have been maintained or further strengthened 33

OAAA Completing the ISAA • For each criterion provide a commentary which describes how the HEI has met its requirements. • The commentary should refer to evidence. • After commentary has been provided against each criterion, provide a brief overarching summary statement on overall performance against the standard. • Provide an explanation against the standard statement (there is no need to reiterate commentary already provided against each criterion) See SAM section 11. 3. 2 34

OAAA Standard Rating & Commentary STANDARD 2: STUDENT LEARNING PROGRAMS BY COURSEWORK Rating Academic standards are maintained through the implementation of the HEI’s planned, well-managed approach to the design, delivery and assessment of all student learning by coursework programs. The HEI has clearly defined generic graduate attributes and program-specific learning outcomes which align with the Oman Qualifications Framework, the HEI’s Mission and Vision and stakeholder expectations. Program delivery is supported by effective use of appropriate teaching and learning methods and assessment is governed by soundly-based regulations, policies and procedures. The HEI’s processes support a culture of academic integrity. The HEI has a systematic approach to the monitoring of graduate destinations and employment that is used to inform the review of student learning by coursework programs. Commentary [Insert a very brief summary of how the HEI meets the standard] 35

OAAA Criterion Rating & Commentary Criterion 2. 1: Graduate Attributes and Student Learning Objectives Rating The HEI has defined generic graduate attributes which reflect stakeholder expectations and which graduates from all programs are required to attain. These are effectively communicated to all stakeholders and incorporated into program development, design and delivery. The HEI’s systems and processes for designing, developing and approving programs ensures that the academic standards of awards are set at an appropriate level and that program-specific learning outcomes are appropriate for the award and align with the Oman Qualifications Framework. The process for assessing student attainment of generic graduate attributes and program-specific learning outcomes is clearly defined and implemented. Commentary [Describe how the requirements of the criterion have been met. Provide links to Supporting Materials that support claims made in the commentary] 36

OAAA Evidence used to support commentary statements • The HEI should provide evidence to support the application • List all evidence referred to in the commentary in the table at the end of all nine standards. Consecutively number each item of supporting materials Supporting Material # Description of SM SM 001 Institutional License SM 002 Program License(s) SM 0023 Current catalogue See SAM section 11. 3. 7 37

OAAA Summary Data Templates Evidence to support claims made in the ISAA can be provided using the Summary Data Templates • • Number of students by program, year of study and gender Number of students by program, year of study and mode Attrition, progress and completion rates Number of staff by department, year, employment status and gender • Number of staff by academic department, year, employment status and nationality • Number of staff by academic department and highest qualification held • Number of staff by administrative department, year, employment status and nationality See SAM Appendix P

OAAA Workshop Activity • Refer to the worksheet which provides examples of panel commentary which support ratings ranging from 4 to 0 (see SAM p. 70) • Identify which panel commentary relates to which rating

Criterion Rating Definition OAAA Rating 4 (Met) Description Definition: Provision or practice consistently meets the requirements of the criteria and exceeds requirements most of the time. Characteristics of provision or practice: i. Results are of high quality and significantly exceed the requirements of the criterion ii. Results against the criterion are achieved through a deliberate and innovative approach iii. Results against the criterion are sustainable and have significantly improved over time iv. Results against the criterion are indicative of best practice See ISAM section 5. 1 40

OAAA Example of Rating 4 A revised innovative academic advising system was successfully implemented and has been shown to have had a significant impact on retention and identifying ‘at risk’ students; the system has received significant positive feedback, been periodically reviewed for effectiveness, indicating continuous quality improvement. Rating against criterion 6. 5, page 52 41

Criterion Rating Definition OAAA Rating 3 (Met) Description Definition: Provision or practice consistently meets the requirements of the criterion and exceeds the requirements some of the time. Characteristics of provision or practice: i. Results consistently meet the requirements of the criterion and exceed the requirements some of the time ii. Results against the criteria are achieved through a deliberate approach which is consistently implemented iii. Results against the criterion are sustainable and have improved over time See SAM section 5. 1 42

OAAA Example of rating 3 As part of its continuous improvement system, the HEI has introduced a revised academic advising system in response to feedback; there is a comprehensive handbook and training for staff and students which has been consistently implemented; the system has had a positive impact; and the HEI has clear plans for how the system will be evaluated. Rating against criterion 6. 5, page 52 43

Criterion Rating Definition OAAA Rating 2 (Met) Description Definition: Provision or practice meets the requirements of the criterion most of the time. Characteristics of provision or practice: i. Results meet the requirements of the criterion. ii. Results against the criterion are achieved through a deliberate approach which is implemented most of the time. iii. Results against the criterion are sustainable. See SAM section 5. 1 44

OAAA Example of rating 2 The HEI has implemented an effective formal academic advisory system which overall, supports students in meeting their educational goals; the system has been evaluated and improvement plans have been implemented in most departments Rating against criterion 6. 5, page 52 45

OAAA Rating Criterion Rating Definition Description Definition: Provision or practice does not fully meet the requirements 1 (Partially Met) of the criterion but the HEI has demonstrated an appropriate commitment to meeting the requirements of the criterion in the future; not fully meeting the requirements of the criterion does not have a significant impact on the overall achievement of the standard. Note: the combined effect of two Partially Met criteria should not have significant impact on the overall achievement of the standard (under discussion) See SAM section 5. 1 46

OAAA Example of rating 1 The HEI has implemented a new institution-wide system for academic advising which replaced adhoc, faculty based approaches to academic advising. While the new system is designed to improve support to students, and students Rating against criterion 6. 5 expressed a high degree of satisfaction with the new system to Panel members, it has not yet been evaluated for effectiveness. The HEI’s approach to evaluation is clearly specified and will be used for the evaluation of the HEI’s academic advising system at the end of the academic year. Rating against criterion 6. 5, page 52 47

OAAA Criterion Rating Definitions Rating 0 (Not Met) Description Definition: Provision or practice does not meet the requirements of the criterion. Definition: Not Applicable This criterion is not relevant to the HEI’s context. See SAM section 5. 1 48

OAAA Example of ‘Not Met’ While the HEI has a policy for academic advising, the system has not been implemented throughout the HEI and has not been monitored or evaluated for effectiveness Rating against criterion 6. 5 Example of Not Applicable Criteria 3. 1 – 3. 6 do not apply to XX HEI as it is classified as a College and does not deliver Student Learning by Research Programs (see SM 011 HEI license approval). 49

OAAA Standard Rating Criteria rating informing standard rating Description Standard rating Most of the criteria are rated 3, at least one criterion is rated 4, and no criteria are rated 1 or 0 Excellent Most of the criteria are rated 3, no more than two criteria are rated 1 and no criteria are rated 0 Good Most of the criteria are rated 2, no more than two criteria are rated 1 and no criteria are rated 0 Satisfactory More than two criteria are rated 1 or one criterion is rated 0 Not met The standard is not applicable to the HEI’s context NA ‘Most’ in this context means more than 50% See SAM page 71 50

OAAA Determining the overall rating for the standard (1) Criterion Rating 6. 1 2 6. 2 3 6. 3 2 6. 4 2 6. 5 2 6. 6 2 6. 7 3 Standard 6 ? ? ? Satisfactory 51

OAAA Determining the overall rating for the standard (2) Criterion Rating 5. 1 3 5. 2 3 5. 3 3 5. 4 0 5. 5 3 5. 6 4 Standard 5 Not met ? ? ? 52

OAAA Determining the overall rating for the standard (3) Criterion Rating 8. 1 4 8. 2 4 8. 3 4 8. 4 4 8. 5 4 8. 6 4 8. 7 3 8. 8 1 8. 9 4 8. 10 4 Standard 8 Good ? ? ? 53

OAAA Determining the overall rating for the standard (4) Criterion Rating 8. 1 2 8. 2 3 8. 3 2 8. 4 2 8. 5 3 8. 6 3 8. 7 3 8. 8 3 8. 9 3 8. 10 4 Standard 8 Excellent ? ? ? 54

OAAA Determining the overall rating for the standard (5) Criterion Rating 4. 1 2 4 4. 3 2 4. 4 2 4. 5 4 4. 6 3 4. 7 2 4. 8 3 4. 9 NA Standard 4 ? ? ? Satisfactory 55

OAAA Determination of Accreditation Outcome Description Outcome Most of the standards are rated Good and at least one standard is rated Excellent. No standards are rated Not Met Accredited with Merit Most of the standards are rated as a minimum Satisfactory and one or more standards are rated as Excellent Accredited with Merit in one or more standards Most of the standards are rated either Satisfactory or Good Accredited One or more standard is/are rated as Not Met One or more standard is/are Not Met (following Standards Reassessment) ‘Most’ in this context means more than 50% On Probation Not Accredited See SAM page 71

OAAA Determination of Accreditation Outcome, Example (1) Standard Rating 1 Satisfactory 2 Excellent 3 NA 4 Good 5 Satisfactory 6 Satisfactory 7 Good 8 Satisfactory 9 Good Accreditation Outcome Accredited with Merit in Student ? ? ? Learning by Coursework 57

OAAA Determination of Accreditation Outcome, Example (2) Standard Rating 1 Satisfactory 2 Excellent 3 NA 4 Good 5 Good 6 Satisfactory 7 Good 8 Good 9 Good Accreditation Outcome ? ? ? Accredited with Merit 58

OAAA Determination of Accreditation Outcome, Example (3) Standard Rating 1 Satisfactory 2 Excellent 3 NA 4 Good 5 Excellent 6 Satisfactory 7 Good 8 Not met 9 Good Accreditation Outcome ? ? ? On Probation 59

OAAA Determination of Accreditation Outcome, Example (4) Standard Rating 1 Good 2 Good 3 NA 4 Good 5 Good 6 Good 7 Good 8 Good 9 Good Accreditation Outcome ? ? ? Accredited 60

OAAA Submitting the ISAA • • ISAA written in the language of instruction Professionally typeset, soft cover format 8 hard copies and e-version on data sticks Supporting Materials to be submitted electronically (on data stick or similar) • Information referred to on a website must: – be accessible by OAAA – not be hindered by internal firewalls – be printable and can be saved and downloaded See SAM section 11. 4

OAAA Fees • Royal Decree 54/2010, Article 18 permits OAAA to charge fees • Fees based on HEI institutional classification and number of full-time equivalent student enrolments at the end of the last academic year • Number of students must be verified by an appropriate independent body • Number of FTE = Number of ( FT + 0. 5 PT)

OAAA Fees (cont. ) • University: 25, 000 OMR + 2 OMR x total number FTE enrolments • University College: 20, 000 OMR + 2 OMR x total number FTE enrolments • College: 15, 000 OMR + 2 OMR x total number FTE enrolments • Accreditation Outcome will not be released until full payment has been received See SAM section 11. 5

OAAA Any questions? 64

OAAA Session 4: Activity 1 Types of evidence to review when preparing the ISAA A training module in preparation for the Institutional Standards Assessment

Workshop Activity Aims • To learn how to use the indicators as tools for self-assessment of the criteria • To understand the types of questions to ask when undertaking a self-assessment of the criteria • To understand the types of documents that should be reviewed as part of the selfassessment of the criteria

Workshop Activity • Each group has been given a set of suggestions to guide the development of a self-assessment against the criteria. Answer the following: • • For which criterion is the guidance relevant? What other questions could be asked? What other documents could be reviewed? How can your HEI demonstrate that its processes and practices related to this criterion are effective?

Workshop Activity Reporting Back • Which criteria relates to which set of suggestions for self-assessment? • How easy did you find the exercise? • What was the most challenging part of the exercise? • How will this help you prepare for the ISA?

OAAA Session 5: Methods of Analysis Using ADRI for Institutional Standards Self-Assessment A training module for preparation for the Institutional Standards Assessment

OAAA Aims and objectives of this session • Understand the different methods of analysis used in Standards Assessment • Understand the importance of evidence in supporting an HEI’s claims • Understand use of ADRI as a tool in selfassessment • Appreciate the reasons why a robust method of analysis is important see SAM Part F, particularly section 25 70

Some Methods of Analysis Issues OAAA Using Statistics and Types of Evidence Quantitative evidence Qualitative evidence Gaining a Comprehensive Picture Gaining Confidence in the Evidence Reaching Conclusions

OAAA Using statistics Standards Assessment outcomes are based on consideration of evidence. There are different types of evidence, each with its own methods of presentation and collection. A notable distinction is between Quantitative and Qualitative evidence. All types have something useful to offer in reaching a comprehensive conclusion. Therefore, Standards Assessment is a mixed method exercise.

Gaining a Comprehensive Picture OAAA Saturation – a method used to explore an issue until no new information about it comes to light. Triangulation – a method for strengthening the analysis using combination of: Multiple original source of data (e. g. students, staff, other stakeholders) Multiple methods of data collection (e. g. surveys, interviews, literature) Different types of data Process Mapping – a method for depicting the steps in a process and their relationships.

Gaining Confidence in the Evidence OAAA In reaching a rating decision, Panel Members must have confidence that the evidence is not only comprehensive, but also valid, reliable and honest! Examples of methods for gaining confidence: Non-attribution Rule Discourage ‘rehearsing’ for interviewees Random Interviews Non-attributable surveys (e. g. student evaluations of teaching) Independence/externality (of survey analysis, program reviews etc. )

OAAA I A R D Using ADRI for ISA

OAAA What is ADRI? • ADRI (or a similar tool) helps to facilitate structured and systematic self-assessment. • ADRI is used to show that achievements have not come about by chance. • ADRI can show that activities are supported by systems and are sustainable. • ADRI focuses on opportunities for improvement rather than weaknesses – the emphasis is on seeking opportunities to enhance the HEI’s provision. • ADRI can be used at all levels of the HEI – institutional, department, program, class. • ADRI is used by HEIs and by the Standards Assessment Panel.

What is ADRI? EN M PR O VE PP R O A C YM S LO LT EN U EP ES H • What strategies, structures and processes have been developed? • Has the approach been benchmarked against best practice? • What performance indicators have been developed to track progress? T R D • What has been achieved? • How is this measured against goals, targets and performance indicators? • Do these results show that the standard or criterion has been met? A IM monitored and reviewed? • What has been learned? • How will improvements be made for next time? T • How are activities OAAA • Have strategies, plans etc been put into practice? • What is the extent of their implementation?

OAAA Using ADRI in Standards Assessments The HEI implements systems and processes for academic advising which provide effective advice and support to students and to assist them in achieving their educational goals. Criterion 6. 5 How do you know if your program meets this criterion? What results/evidence support your claim? How do you know if what you do is “good enough”?

ADRI Reviewing for Internal Academic and External Advising Processes ADRI Cycle of Quality Assurance OAAA EN T Evidence examples: • Academic advising policy • Staff handbook • Student handbook • Evidence of benchmarking M A PR O VE PP EP LO O A C T EN ES Evidence examples: U • Academic advising LT S data • Evidence of student satisfaction • Evidence of impact e. g. at risk student register, student progression rates YM Academic advising D R R Criterion 6. 5: IM Evidence examples: • Evidence of review of academic advising system e. g. external/internal reports • Input obtained from staff and students • Resulting action plans • Reports on progress against action plans H Evidence examples: • Ongoing staff and student feedback • Staff and/or student training materials • Records of academic advising Reviewof of the System It canis. Internal be applied toquality any topic – in thisused case, analysisinof an ADRI a model assurance bythe agencies many External Review of the System HEI’s academic countries around advising the world

OAAA Conclusion • Quality assurance requires a suitable method for analysing the effectiveness of the systems in place to show they meet external standards. • The ADRI method is a useful tool. It can be used by small work teams, internal review committees and external review panels. • Because ADRI combines an assessment of the quality system with a constructive analysis, it is not something extra to do, but rather a better way of doing what we should be doing anyway.

OAAA Session 6: Activity 2 Rating Criteria and Standards A training module for preparation for the Institutional Standards Assessment

OAAA Aims of the activity • To practice applying ratings to criteria • To gain an understanding of the types of processes and practices and characteristics are associated with the ratings • To develop an understanding of the types of evidence required to justify ratings

Workshop Activity Select any standard • Refer to the criterion rating definitions and characteristics provided in Table 3 SAM section 5 • For each criterion related to the selected standard provide bullet points on the types of processes and practices that would normally be in place to support the criteria ratings Rating 4 3 2 1 0

Workshop Activity Reporting Back • Without discussing specific ‘real life examples’ what was the most challenging aspect to deciding on a rating? • How difficult was it to determine the type of evidence required to support the rating? • How did you achieve consensus within the group?

OAAA Any questions?

OAAA Session 7: Institutional Standards Assessment Recap and Tips A training module for preparation for the Institutional Standards Assessment

OAAA Session Aims • Be reminded of the concept of self-assessment • Identify features of a ‘good’ institutional standards assessment application (ISAA) • Reflect on features of a ‘good’ institutional standards assessment (ISA) self-assessment process • Identify practices that will support a smooth ISA Visit 87

OAAA What is Self-Assessment? • In the context of quality in higher education, the term “self-assessment” is used interchangeably with “self-evaluation” and “self-study”. • Core definition: “The process of critically reviewing the quality of one’s own performance and provision” • Takes place in the context of external review; but is based on the principle that primary responsibility for quality and quality assurance rests with the HEI • Involves: – a process of self-reflection in the form of critical evaluation by the HEI being reviewed (based on evidence and analysis; identifies strengths and areas for improvement; evaluates performance against external standards ) and – preparation of a document reflecting the self-assessment: the Institutional Standards Assessment Application (ISAA) based on Harvey, L. , 2004 -14 Analytic Quality Glossary 88

OAAA Workshop Activity Think like a Panel member! Scenario • You are a member of an OAAA Standards Assessment Panel, for the Stage 2 Institutional Accreditation: Standards Assessment of an HEI. • As a Panel Member, from your perspective, what are the features of a ‘good’ ISAA? To help you carry out your role, what do you need from the HEI’s ISAA? • Please discuss in your group and list 10 features of a ‘good’ ISAA.

OAAA The Institutional Standards Assessment Application (ISAA) The Product 90

OAAA General Features of a ‘Good’ ISAA • Complete, and the layout adheres to the template • Demonstrates that the HEI meets the institutional standards • Includes the HEI’s response to Quality Audit CARs • Self-evaluative; analytical; based on evidence; demonstrates use of ADRI • Clear; readable; understandable; ‘makes sense’ to an external reader • A ‘single voice’: the ISAA is edited to a high standard • A ‘clear voice’ : internal consistency in the content • All key evidence (supporting materials - SMs) is submitted with the ISAA and clearly labeled (numbered, titled, dated) • No SM submitted more than once; clear cross-references to other sections / supporting materials 91

OAAA Features of a ‘Good’ ISAA: Commentary and Use of Evidence • • Commentary relates explicitly to each criterion Open and honest / not ‘promotional’: avoids marketing language Concise: as long as needed and no longer Appropriate balance between description and analysis; description should not greatly outweigh analysis • Evidence used to: • compare the HEI’s performance against the criteria and standards • identify strengths and areas for improvement, in order to apply ratings • justify evaluative statements made in the commentaries • Draws on evidence that already exists in the HEI • Selective in use of evidence • Evidence used is: valid; reliable, accurate and relevant 92

OAAA Any additional features of a ‘good’ ISAA? 93

OAAA The Institutional Standards Assessment (ISA) Self-Assessment The Process 94

OAAA Workshop Activity Reflection on ISA Self-Assessment Reflect on your ISA self-assessment process to date. List 5 features of a ‘good’ institutional self-assessment process

OAAA Features of a ‘Good’ ISA Self. Assessment Process (1) • Ownership and inclusivity: based on the principle that the main responsibility for quality and quality assurance rests with the HEI • Based on an understanding of the cyclical, two stage institutional accreditation process (Quality Audit – Standards Assessment) and the concept of continuous quality improvement • Builds on the Stage 1 Quality Audit Process (with special attention to CARs) • Uses a planned approach (timeline; tasks; roles and responsibilities; procedures etc. ) which is informed by the experience of previous self -review activities • Appropriate balance of attention to evaluation (self-assessment) vs improvement • Effective communication: all relevant stakeholders kept “in the loop” • Support for all involved: training; acknowledgement of time and effort 96

OAAA Features of a ‘Good’ ISA Self. Assessment Process (2) Effective Project Management E. g. • Steering Committee and working groups established • Time frame determined and resource requirements (time, budget etc. ) identified and addressed • Staff training conducted (e. g. in ADRI) • Task broken down: clear responsibilities for evidence gathering and drafting the ISAA established • Clear procedures for: approving ISAA drafts; circulating drafts; getting feedback on drafts • Responsibilities for bringing all the sections together into a coherent overall ISAA and for final editing established. See SAM section 10. 2 97

OAAA Any additional features of a ‘good’ ISA self-assessment process? 98

OAAA Preparation for the Institutional Standards Assessment (ISA) Visit 99

OAAA Panel Room Layout See SAM section 20 How can you prepare for the ISA visit? 100

Preparation for the ISA Visit (1) OAAA • Use lessons learnt from the Stage 1 Quality Audit process (ADRI) • Contact Person to establish and maintain a close and effective working relationship with the OAAA Executive Officer (EO) • There are no ‘stupid questions’; seek timely clarification from the EO on any points that arise • Be well prepared for the Planning Visit: allocate time in the days beforehand to: – Prepare a written response to the Points for Clarification – Consider the request for additional Supporting Materials and being to collect these ready for submission – Consider the draft ISA Visit Program and be prepared to contribute constructively to finalizing this during the meeting – Decide on the key venues and logistics for the visit in preparation for consideration by the OAAA See SAM section 20 + Appendix F 101

OAAA Preparation for the ISA Visit (2) OAAA identifies the profile of interviewees, not named individuals. In populating the ISA Visit program, adhere to expected requirements to avoid last minute complications: E. g. – Line managers are interviewed separately from their staff (also applies to “in situ” interviews) – Interviewees are interviewed only once (unless special circumstances apply) – Key roles undertaken by named individuals are indicated (e. g. Chair of Disciplinary Committee) – Representatives of external stakeholder groups such as employers and student placement hosts are selected appropriately: relevant role vs seniority – Interviewees who are Arabic rather than English Language users are identified – The EO is informed of any changes in a timely manner 102 See SAM section 29

OAAA Preparation for the ISA Visit (3) • Develop an approach to contacting and briefing all interviewees; it is in the interest of the HEI that all interviewees identified by the OAAA Standards Assessment Panel are available for interview in accordance with the Standards Assessment Visit Program • Allow sufficient time for ensuring the availability of: – Governance representatives: (e. g. senior government officials; chairs of members of governing bodies) – External Stakeholders (e. g. employers, student-placement hosts; advisory group members, guest lecturers etc. ) – Affiliate representatives (may include external examiners) – Alumni – Students ( identify back ups) • Ensure interviewees are well informed: make use of the briefing sheets provided (translate as required) SAM Appendices I, J, and K

Preparation for the ISA Visit (4) OAAA • Ensure everyone understands what Public Submissions and Random Interviews are; the Public Submissions notification in the local press is a standard part of the process for all HEIs See SAM Section 20 + Appendix G • Ensure all interviewees understand the general format and protocols of the interviews See SAM section 29 E. g. – Most will be interviewed as part of a group and will enter the interview room as a group – Interviewees should bring their name cards with them to the interview – No note taking or recording of the interview is permitted – Interviewees may refer to documents but Panel Members are not permitted to receive documents from them – Students are not being evaluated as individuals – Post interview, interviewees should not report what they (or other people) said during the interview • Trial Standard Assessment Visits: If conducted, avoid the pitfalls. Coaching of interviewees will be detected by the Panel and will hinder the process See SAM section 12

OAAA During the Standards Assessment Visit 105

During the ISA Visit OAAA Support the smooth running of the ISA visit by: • • Disciplined time keeping Effective management of interviewees by HEI Contact Person available at all times IT support readily available to Panel (through the Contact Person) • Panel’s need to work confidentially respected • Catering for the Panel sufficient but not excessive • Additional points? Panel Departure: • Differs from Stage 1 Quality Audit; no feedback on preliminary findings or conclusions (management of expectations) • Photo opportunity 106

OAAA The OAAA Institutional Accreditation process encourages and recognises excellence through its rating scale and accreditation outcomes with Merit: The OAAA is keen to “catch HEIs doing things well”! Any Questions? 107

OAAA Session 8: Q&A A training module for preparation for the Institutional Standards Assessment

Aims of the Workshop To provide participating HEI staff with an overview of the OAAA’s role in Institutional Standards Assessment To provide a guide for the process of the Institutional Standards Assessment and its link to sections of the Standards Assessment Manual To provide staff with a clear idea of their role in the Institutional Standards Assessment process To provide guidance on how to carry out a selfassessment and prepare an Institutional Standards Assessment Application including the methods of analysis and ADRI To provide participating HEI staff with a chance to ask questions and clarify understanding

Any questions?