LEARNING GAIN LINKING STUDENT ATTAINMENT WITH ENGAGEMENT SATISFACTION

LEARNING GAIN: LINKING STUDENT ATTAINMENT WITH ENGAGEMENT, SATISFACTION AND WELLBEING Dr Amanda Callaghan (Project Lead) Cesare Aloisi (Project Officer) 1 Copyright University of Reading LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT

White Paper May 2016 • Proposed that the TEF would consider teaching and learning excellence across three main aspects: • Teaching Quality, • Learning Environment, • Student Outcomes and Learning Gain.

DEFINITION • Learning Gain can be described as the ‘distance travelled’ by students during their studies, demonstrable by an improvement in knowledge, skills, work-readiness and personal development between two points in time. Mc. Grath et al 2015 RAND 4 LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT

HEFCE Learning gain project The impetus for this project comes from BIS who asked HEFCE to ‘consider whethere are better indicators, such as measures of student engagement, to provide information on what a high quality student experience looks like’. Outcomes of this project will feed into ongoing debates about the quality and impact of higher education, and how we evidence the value of investment in it. http: //www. hefce. ac. uk/lt/lg/projects/ 5

UNIVERSITY OF READING LEARNING GAIN PROJECT – MIXED METHODS • A combination of methods and indicators to track improvement in performance, for example through a combination of grades, student learning data and student surveys. • A three-year longitudinal project to test and evaluate a range of available methodologies and to draw conclusions on what might be the right combination of instruments for the measurement of learning gain. 6 LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT

INTRODUCTION The University of Reading project will be modelling the effectiveness of using data that measure: • student attainment, • student engagement, • student satisfaction, • student wellbeing and • critical thinking skills. Taken together they present a global view of what students gain in knowledge, skills, work-readiness and personal development. and perceived learning gain to predict attainment. LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT

LEARNING PYRAMID Distance from academic discipline Discipline-specific skills • Breadth and depth of knowledge Academic and research skills Professional and “employability” skills Personal effectiveness and self-awareness Social skills • Ability to learn through research and enquiry • Entrepreneurship • Use/applicability of discipline in real-world contexts • Desire for self-improvement, intellectual curiosity, commitment to lifelong learning • Reasoning and problem-solving skills • Human, social, civic and environmental responsibility LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT

WHAT MEASURES WOULD YOU USE? Sheet in your pack. Of the tools you have access to, what would you use to measure: • • • Discipline specific Academic and research skills Employability Self-awareness Social skills LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT

MEASURES WE CHOSE Skill type Measures Discipline-specific Grades and progression (from SITS data) Perceived Learning Gain Academic and research, including “critical thinking” Grades and progression CLA+ Need for cognition scale (NCS) UKES Professional and “employability” Internal survey from Careers Office Destinations of Leavers from HE (DLHE) Personal effectiveness and self-awareness Social UKES Student wellbeing survey (WBS) NCS UKES WBS LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT

DISCIPLINE SPECIFIC: GRADES AND PROGRESSION Differences between student grades at two points in time. 11 LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT

DATASET 1754 students from 51 programmes of studies: - Undergraduate Full-time Graduated in 2014/15 (pass and fail) On time From a 3 -year course = 75% of UG FT students who graduated on time in that year = 55% of UG students who graduated in that year LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT

Programme level SITS data LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT

Student level SITS data Programme 1 LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT

PERCEIVED LEARNING GAIN • Students rate themselves against a set of criteria linked to the learning outcomes of modules and programmes. • End of each academic year and on initial arrival. • Based on methods used by a medical education researcher at a German University (Dr Raupach). • http: //www. evasys. co. uk/newsreader/evaluating-learningoutcomes-with-evasys. html • Schiekirka et al 2013. Estimating Learning Outcomes From Pre- and Posttest Student Self -Assessments: A Longitudinal Study. Academic Medicine: • March 2013 - Volume 88 - Issue 3 - p 369– 375 15 LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT

ACADEMIC AND RESEARCH CLA+ • The Collegiate Learning Assessment (CLA+) is a US performance based test that measures critical thinking, problem solving, analytic reasoning, and writing skills • The CLA+ uses a Performance Task and a series of selected-response questions measure these higher order skills • No prior knowledge of any particular field is necessary in order to perform well • The principal goal is to provide an objective assessment about the critical-thinking skills a student possesses as they enter and exit college LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT

60 min 30 min 17 LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT

CLA+ APPROACH AND ISSUES • Randomised sampling from a selection of modules • Cross-sectional and longitudinal approach • First administration: ceiling effect: • 6 students out of 39 are already Advanced and cannot progress any further • With +1 point in the essay and +1 correct answer in the MCQ 31 out of 39 students would sit in the highest two levels. LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT

ISSUE ENGAGEMENT WITH SURVEYS ETC • How do you incentivise students to engage when they see no benefit for themselves? 19 LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT

CLA+ ISSUES LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT

Year 1 Autumn Year 2 Spring Autumn Spring Year 3 Autumn Longitudinal comparison CLA 1 40 P 2 UGs 40 P 1 UGs matc hing CLA 1 160 P 1 UGs CLA 2 Spring CLA 1 X-se 40+” 160” P 3 UGs ction al 160 P 2 UGs CLA 2 LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT

THE UK ENGAGEMENT SURVEY (HEA) • It is the only undergraduate survey in the UK to focus on student engagement. • Aimed at non-finalists (early warning). 22 LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT

WELLBEING AND NEED FOR COGNITION • Student Wellbeing Project ( Department of Economics Uo. R, 2011) is a longitudinal study to measure how student wellbeing, performance, productivity and satisfaction with university provision are related. • Need for Cognition Scales (NCS) is a personality instrument of metacognition intended to measure the extent to which an individual desires to engage in a cognitive activity. It can be used in parallel with the CLA+ to track development in critical thinking skills. The NCS measures predisposition to think critically rather than apply critical thinking skills in prescribed situations. 23 LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT

EXAMPLE VALIDATION MODEL • Measures of psychological well-being, financial security, sense of belonging and engagement are entered as level 1 time-changing predictors. 24 LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT

SAMPLING APPROACH All students Many students SITS data UKES + WBS Careers A few students CLA+ Perceived LG LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT

MEASURING THE IMPACT OF ENGAGEMENT ON EMPLOYMENT • Analysis of graduates from 11/12, 12/13 and 13/14 • Completion of placement activities (Reading Internship Scheme, Mini, Micro, Maxi – 1 year placement, UROP), RED award, study abroad, gender, ethnicity, fee status, disability, home Faculty, degree studied, degree classification, age on entry. • Placement data – criticism of correlation = academically better students more likely to engage in placements but also more likely to get a good job. 26 LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT

ACKNOWLEDGEMENTS • Thanks to Dr Karen Ayres (Statistics) for analysis of employment. • HEFCE for funding under the Learning Gain Project 27 LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT