Development of Key Performance Indicators for the Engineering

Development of Key Performance Indicators for the Engineering Education Programs in Jordan: University of Jordan as a case study by Issam S. Jalham Professor of Industrial Engineering, former vice president of Higher Education Accreditation Commission in Jordan, University of Jordan, Amman 11942 Jordan

Abstract In Jordan most of universities/colleges of technology offer an engineering education and are anticipated to continuously improve their performance to prepare high quality engineers. So, it is necessary to construct wellstructured and up-to-date performance indicators for the programs. In this study, each of the ABET eight criteria was divided into sub-criteria and each sub-criterion was given suitable KPIs. 37 key performance indicators (KPIs), which are outcome-based and compatible with ABET’s accreditation criteria were developed. Then a questionnaire including the suggested KPIs was mailed to the stakeholders. Based on the analysis, the developed 37 KPIs are highly recommended to all engineering programs in Jordan.

Introduction After a huge and complex survey, Educational and vocational training systems in Jordan can be classified into the following: 1. University Education The university designation is supported by evidence of rigorous standards. One of the key defining characteristics is substantial engagement in the conduct of research. Credible research activity is important also for the external reputation of the system of Higher Education as a whole, since Universities are at the top of the system and the international standing of Universities is heavily dependent on the quality of their research. A University must have at least three major fields of study (Bachelor, Master and Doctorate degrees) and a significant research component. 2. University College and/or Community College Education It is expected that this education shares some of the key characteristics of a University. These characteristics include research activity relevant to local and national needs; appropriate research facilities; and, programs up to and including the Master’s degree in at least two broad fields of study. 3. High School Certificate leading to university entrance (Tawjihi) 4. Technical education (Examples: industrial, hotel and tourism, agricultural, house economy, and vocational educations ) 5. Compulsory Basic Education 6. Pre-school Education (Examples: Nursery or Kinder Gardens) 7. Vocational training programs

Relationship among educational and vocational training systems

Background Latest statistics in english Jordan has made significant progress in higher education, a fact reflected in the sharp increase in the number of public and private higher education institutions over the last four decades. At present there are 10 public universities, 20 private universities, and 44 community colleges. Latest statistics according to MHE

Referring to diagrams above: ■ Concern has continued to grow with the increase of the number of higher education institutions, students, and faculty members. (without control over this process) ■ This led to concern about accreditation and quality assurance (QA) of higher education in Jordan. ■ Reform of higher education became inevitable by controlling this growth & Complying with the accreditation and QA regulations. 12/7/2020 Professor Issam S. Jalham, Vice president-HEAC 6

► As a guarantee for a minimum standard of quality and excellence, a legislation was promulgated in 1998 to establish the Accreditation Council (AC). ► AC primary focus was private universities. Public universities were not subjected to accreditation as their enrollment continued to exceed capacity levels and issues of quality became prominent. (Public universities are now subjected to accreditation but they are given a grace period) ► Public and private pressure heightened the demand on universities and concerned public officials to achieve international standards toward Jordan becoming a regional/global center for higher education delivery. 12/7/2020 Professor Issam S. Jalham, Vice president-HEAC 7

► To that end, in June of 2007, the (AC) was dissolved, and the Higher Education Accreditation Commission (HEAC) was established in its place by law. HEAC became a financially and administratively independent entity and attached to the Ministry of Higher Education. ► In 2009 HEAC became a financially and administratively independent entity and attached to The Prime Minister. 12/7/2020 Professor Issam S. Jalham, Vice president-HEAC 8

The scope of HEAC’s jurisdiction encompasses: 1. Mandate over private and public universities, including all and any foreign higher education institutions affiliated with Jordanian universities. 2. Responsibility for the National Testing Center, which designs and conducts standardized tests in all fields. 3. The advancement of sound means and methods to ensure that higher education institutions are in full compliance with all its regulations (accreditation & QA). 4. (Ranking of the Universities in the future) 12/7/2020 Professor Issam S. Jalham, Vice president-HEAC 9

World ranking of universities, arises today, is a challenge now because of high competitiveness all over the world. So seeking international accreditation may help universities compete. ABET (Accreditation Board for Engineering and Technology) accreditation is considered as a way for competitiveness at the world level.

Indicators are defined as "individual or composite statistics that reflect important features of a system, such as education, health, or economy". It is also known as key performance indicators (KPI’s) that help an organization define and measure progress towards organizational goals. Once an organization has analyzed its mission, identified all its stakeholders, and defined its goals, it needs a way to measure progress toward those goals. KPIs are those quantifiable measurements.

Performance indicators identified should be scientific and practical. Based on an extensive review of literature, the authors identified 22 KPIs for ETE programs which meet the following criteria: 1. Alignment with the institution missions 2. Linkage to overall Engineering and Technology Educational program objectives 3. Access to specific information regarding program problems 4. Measurement of observed behavior rather than perceptions 5. Reliability and validity 6. Access to analytical links 7. Feasibility of implementation 8. Addresses a broad range of audiences

Significance of ABET accreditation: The following points show the importance of ABET accreditation: 1. Verifies that their educational experience meets the global standard for technical education in their profession. 2. Enhances their employment opportunities—multinational corporations require graduation from an accredited program. 3. Supports their entry to a technical profession through licensure, registration, and certification—all of which often require graduation from an ABET-accredited program as a minimum qualification. 4. Paves the way for them to work globally, because ABET accreditation is recognized worldwide through international agreements, and many other countries’ national accrediting systems are based on the ABET model. 5. Directly involves faculty and staff in self-assessment and continuous quality improvement processes

All programs seeking accreditation from ABET must demonstrate that they satisfy all of the following eight general criteria for BSc level programs: 1. Students, 2. Program Educational Objectives, 3. Student Outcomes, 4. Continuous Improvement, 5. Curriculum, 6. Faculty, 7. Facilities, and 8. Institutional Support.

Methodology To be able to evaluate the eligibility of an engineering program for ABET accreditation, the current status of the program should be evaluated and Self-Study Report (SSR) should be prepared. The evaluation of the program should start from the development of the suitable KPI’s for the program. In this study, each of the ABET eight criteria was divided into sub-criteria and each sub-criterion was given a suitable KPI’s. To achieve the aim of translating stakeholders level of satisfaction, different approaches can be used: Surveys, Observations, Interviews and Documentary Analysis.

4. Questions of surveys must provide the needed information for the study. Since the surveys were conducted to help assess the current status of the department, asked questions were mainly based on the criterion posed by ABET, developed sub-criteria and their thirty seven KPI’s. 5. Collected data were analyzed using SPSS software.

To fulfill the objective of this study: 1. KPI’s were developed by distributing a questionnaire, which was designed, and verified, to the stakeholders of the program. 2. The stakeholders of each program were taken as final year students, industry, public sector, alumni, Jordan Engineers Association, and faculty. 3. A five Likert scale survey was adopted during the assessment to collect the needed data. When responding to a Likert item, respondents specify the level of agreement on a symmetric agree/disagree scale for a series of statements. This type of surveys is easy and convenient to collect data. Likert items are used to measure respondents’ attitudes to a particular question or statement. For an analysis purposes, the data it is usually coded as follows: 1. Strongly agree, 2. Agree, 3. Neutral, 4. Disagree, 5. Strongly disagree.

Results and Discussions After the analysis of the results of this study, 37 KPIs Adopted by Stakeholders. These 37 KPIs are shown in Tables 1 -8 below. To be more precise and realistic, each criterion was divided into subcriteria. A KPIs were assigned to each sub-criterion as it is shown in tables 1 -8. As an example, let us take the sub-criterion 2 of criterion 3 (student outcomes) for the course metallurgical processes, offered by the Industrial Engineering Department at the School of Engineering/University of Jordan. As it is indicated, the KPIs are the course assessment by faculty (CAF) and the course assessment by student (CAS). The results are shown in Figure 2 and Figure 3 (a) and (b). These forms are specially designed to suite the courses.

Table 1. KPI’s of Criterion 1: students No. Subcriterion 1 Acceptance and transfer criteria 2 3 4 5 6 Advising system Evaluation system Prerequisites of graduation project Internship Graduation requirements KPI Cumulative average of high school exam, University regulations requirements Student interviews, surveys Survey Reports, survey Survey Table 2. KPI’s of Criterion 2: PEO No. Subcriterion 1 Development of vision, mission, objectives, …. . etc 2 Dissemination of vision, mission, objectives, …. . etc 3 Consistency of PEO’s with the mission of the institution 4 Review process of PEO’s KPI Participation of stakeholders (documentation) No. of the web site visitors Relationship table Review results as an input to improve study plan and course contents (documentation)

Table 3. KPI’s of Criterion 3: Students outcomes No. Subcriterion KPI 1 Relationship of Student Outcomes to Program Educational Relationship table Objectives 2 Evaluation of student outcomes CAS, CAF Table 4. KPI’s of Criterion 4: Continuous improvement No. 1 2 3 4 Subcriterion KPI Description of the assessment CAF processes Examples of data collection specific exam questions, student portfolios, internally processes developed assessment exams, senior project presentations, nationally-normed exams, oral exams, focus groups, industrial advisory committee meetings, or other processes that are relevant and appropriate to the program Summaries of the results of the Coursereport evaluation process Results of the valuation processes Documentations and minutes ofmeetings for the student outcomes have been systematically used as input in the continuous improvement of the program

Table 5. KPI’s of Criterion 5: Curriculum No. 1 2 3 4 5 6 Subcriterion Study plan Guidance plan The curriculum aligns with the program educational objectives the prerequisite structure of the program’s program meets the requirements in terms of hours and depth of study for each subject area (Math and Basic Sciences, Engineering Topics, and General Education) Course file content KPI Completed table 5 -1 Available, no of students aware of it Relationship table flowchart No. of hours covered by the study plan course syllabi, textbooks, sample student work, etc. (documentation) Table 6. KPI’s of Criterion 6: Faculty No. 1 2 3 Subcriterion Qualification, Ranks, and size Professional development Authority and responsibility 4 5 Research and patents Dedication to the courses they teach KPI Faculty : student ratio, workload No. of attended conferences and workshops Regulations, participation in course creation, modification, and evaluation. Contribution in revision of program educational objectives and student outcomes, and their role in the attainment of the student outcomes. Description of the roles of others on campus, e. g. , dean or provost, with respect to the areas above. No. of publications and patents Survey

Table 7. KPI’s of Criterion 7: Facilities No. 1 3 Subcriterion Sufficiency of Offices, Classrooms, and rest rooms to serve program objectives, mission, and outcomes of the program Laboratories (Sufficiency of supervisors and technicians, Sufficiency of equipment, safety signs, maintenance program, guides for conducting the experiment) Computing Resources 4 Library and e-library 2 KPI Ratio of No. of Offices, Classrooms, and rest rooms to no. of students Ratio of supervisors and technicians to the no. of students Reports and site visits Accessibility, no. of workstations, servers, storage, and networks including software Ratio of supervisors and technicians to the no. of students No. of books for each course, no. of software linces Table 8. KPI’s of Criterion 8: Institutional support No. Subcriterion 1 Leadership adequacy to ensure the quality and continuity of the program and its involvement in decision that affect the program 2 Program Budget and Financial Support 3 Professional Development of the administrative, instructional, and technical staff 4 Faculty Hiring and Retention process 5 Strategies used to retain current qualified faculty 6 Support of Faculty Professional Development KPI Reports, minutes of meetings Regulations, and reports Development plan, no. of staff attending courses. Regulations, reports of cases No. of sabbaticals, No. of supportive travel, workshops, seminars, etc. ,

Conclusions KPI is a measurable factor of extreme importance to the program in achieving its strategic goals and in presenting its effectiveness. Based on the analysis, the developed 37 KPIs are highly recommended to all engineering programs in Jordan. Also, special forms should be designed to achieve the purpose.