Network of Deputy Vice Chancellors Pro Vice Chancellors

Network of Deputy Vice. Chancellors, Pro Vice. Chancellors and Vice. Principals 17 -18 May 2017 Grand Connaught Rooms, London Sponsored by

Learning Gain Follow us on Twitter at @hefce / @rdsmithuk #learninggain Richard Smith Senior Policy Adviser, HEFCE HEA PVC Network London, May 17 2017

Interest grows in learning indicators and learning gain “HEIs actively compete for well informed, discerning students, on the basis of price and teaching quality, improving provision across the whole sector, within a framework that guarantees minimum standards. ” Securing a sustainable future for higher education , 2010, Browne et al “…what best predicts educational gain is measures of educational process: what institutions do with their resources to make the most of whatever students they have. ” Dimensions of Quality, 2010, Graham Gibbs 45% of students “did not demonstrate any significant improvement in learning” during their first two years of college Academically Adrift: Limited Learning on College Campuses, 2011, Richard Arum & Josipa Roksa Process variables “…characterise what is going on in teaching and learning” • Class size • Class contact hours, independent study hours and total hours • Quality of teaching: • experience and training • research record • judged by students • Research environment • Level of intellectual challenge: • level of curriculum • depth of approach to studying • student engagement • Formative assessment and feedback • Other, incl. reputation, peer ratings, student support, QA processes

“In a world in which machines came to dominate tasks involving core cognitive processing, the importance of, and skill premium attached to, non-cognitive skills is likely to rise. The high skill - high pay jobs of the future may involve skills better measured by EQs than IQs, by jobs creating social as much as financial value. Yet our education system, at present, has a strongly cognitive slant. Perhaps in future that will need to change, with as much effort put into cultivating social CVs as academic ones. ” Nov 2015 - Andy Haldane - Chief Economist at the Bank of England

Institutional Pilot Projects Higher Education Learning Gain Analysis HEFCE’s Learning Gain Programme Raising and building understanding and awareness National Mixed Methodology Project Horizon scanning

2015 -16: institutional pilot projects Methods Grades - Differences between student grades at two points in time Surveys - Students self-report on what they have learned (etc. ) through a survey Standardised Tests - Measure acquisition of generic or specialised skills Other qualitative methods - Including personal development portfolio and/or reflection on skills gap Mixed Methods - Using a range of tools / methods to track performance (RAND 2015) Skills Knowledge Workreadiness Dimensions of learning gain Personal development

2016 -17: national project launched Critical-thinking & problem-solving Attitudes towards study experience Engagement 12 test questions exploring respondents ability to: Three sets of survey questions exploring: • • summarise the main conclusion (CT) assess the impact of additional evidence (CT) apply principles (CT) detect reasoning errors (CT) identify similarity (PS) find procedures (PS) make relevant selections (PS). source: Cambridge Assessment 2016 -17 (1 st administration) 1. 2. Critical-thinking & problem-solving Attitudes towards study experience • Positive attitude toward Literacy (6 questions) • Openness to diversity (7 questions) • Academic motivation (8 questions) source: Pascarella/Wabash 2016 -17 (2 nd administration ) 1. 2. 3. Critical-thinking & problem-solving Attitudes towards study experience Engagement • course emphasis on critical thinking • interaction with staff • time spent between various academic and non-academic activities source: HEA/UKES questions 1, 5 and 18 2017 -18 (3 rd administration) 2018 -19 (4 th administration ) 2022? Link results to graduate outcomes? TEST REPEATED

Higher Education Learning Gain Analysis (HELGA) Investigating the trade-off between burden and validity through: Start- and end-points in existing administrative data Methods for making comparisons across institutions/departments Approach: Applying proven methods Linked administrative datasets Comparison against richer measurement approaches Drawing on academic expertise - Learning Gain Expert Group

What are we learning?

Thank you for listening r. smith@hefce. ac. uk

Next HEA PVC Network 22 -23 November 2017 Imperial College, London 12 03/10/2020

Learning Gain and Social Mobility: The Contribution of the LEGACY Consortium Christina Hughes Sheffield Hallam University christina. hughes@shu. ac. uk

15. 2% BME Attainment Gap ECU Equality in Higher Education Statistical Report 2015 http: //www. ecu. ac. uk/publications/equality-highereducation-statistical-report-2015

30% Earnings differential for males by socio- economic status (courtesy Anna Vignoles University of Camridge)

24% Earnings differential for females by socio -economic status (courtesy Anna Vignoles University of Camridge)

The Social Mobility Imperative “Lifting others up through education”

• 18 Russell Group universities and one post-92 • Using a range of methodological approaches, four work streams led by: • Cambridge: developing an instrument to measure learning gain across cognitive, metacognitive and affective dimensions; • Warwick: measuring the impact of Realise 2 Strengths development tool on student employment outcomes; • Birmingham: measuring the impact of study abroad; work abroad; and remaining on campus on • Nottingham: measuring the progression of students’ career adaptabilities amongst the ‘less engaged’. www. legacy. ac. uk @Legacy. LGProject

• • Student engagement in these initiatives takes persistence (and incentives); Female students and medics seem to be the most engaged; The less engaged remain less engaged; Yet, we have 6, 000+ student responses in 11 institutions for the Cambridge survey; 431 student responses in 7 institutions for the Nottingham work; 439 student responses in 6 institutions for the Warwick work plus 316 oneto-one coaching sessions; 61 qualitative interviews of ‘work abroad’ students in 7 institutions for the Birmingham work. What we have learnt so far: Student Engagement

• Considerable time spent to organise the fieldwork. • Healthy response rate is due to persistence of partners and flexible redistribution but: • Response rate good for online survey, but not sufficient if system-wide exercise. • Response seems (Cambridge work) higher for Medicine than other subjects • Importance of using incentives to engage students • Importance of communicating with student participants to maintain engagement; . What we have learnt so far: Methodology 20

• • • • • • • • • • • Data linkage: HESA and DLHE Item 1: HE students 2015/16 with links to the client dataset where applicable by Instance identifier Link marker Student registration population (used to define the student population) Qualifiers population (used to define the qualifiers population) First year marker UKPRN of HE provider Age at 31 August (full) Gender Nationality (full) Domicile (Country of domicile) Geographic region of domicile (Government office region for UK / Geographic region for non-UK) Ethnicity (full - only applicable for UK domiciled students) Parental education Socio-economic background State school marker Disability (full) Course aim (full) Highest qualification on entry (Postgraduate (excluding PGCE), PGCE, First degree, Other undergraduate qualification, Other qualification, Level 3 qualification (including A levels and Highers), Qualifications at Level 2 and below, No formal qualification, Not known, Not applicable) Level of study (Postgraduate / First Degree / Other Undergraduate) Year of programme Expected length of study (<5 weeks, =>5 weeks and <24 weeks, =>24 weeks and <=1 year, >1 year and <=2 years, >2 years and <=3 years, >3 years and <=4 years, >4 years and <=5 years, >5 years and <=6 years, >6 years and <20 years, Not known or expected length of study programme is greater or equal to 20 years) Mode of study Highest GCSE English grade Highest GCSE Maths grade Planned employability, enrichment and pastoral activity (applicable to England) Planned timetables hours for learning activities (applicable to England) Total tariff score (applicable to students holding level 3 tariff bearing qualifications) Qualifiers age at 31 July (full) Qualification obtained (full) Level of Qualification (Postgraduate / First Degree / Other Undergraduate) Mode of qualification (full-time / Part-time) Principal subject of study Full person equivalence (FPE) Item 2: Student mobility data for those who link to the client dataset 2015/6 by Instance identifier (used for linking to the main dataset) Mobility duration Mobility location Mobility scheme Mobility type Analysing Outcomes for Social Disadvantage: LEGACY Contributes

INSTITUTIONAL DEVELOPMENT OF LEARNING GAIN CAN MAKE A STEPCHANGE IN OUTCOMES OF THOSE ALWAYS-ALREADY-DISADVANTAGED We Measure What we Value: Are we?

Programme Lead: Project Leaders: Cambridge: Professor Christina Hughes Christina. Hughes@shu. ac. uk Birmingham: Professor Jan Vermunt jdhv@cam. ac. uk Professor Anna Vignoles av 404@cam. ac. uk & Dr. Sonia Ilie isi 22@cam. ac. uk Eluned Jones E. G. Jones@bham. ac. uk Nottingham: Dr. Nalayini Thambar Nalayini. Thambar@nottingham. ac. uk Warwick: Anne Wilson A. E. Wilson@warwick. ac. uk Programe Support: Programme Manager: Sunil Maher Sunil. Maher. 1@warwick. ac. uk Research Fellow: Dr. Heike Behle Heike. Behle@warwick. ac. uk Web: Twitter: www. legacy. ac. uk @Legacy. LGProject FIND OUT MORE ….

Next HEA PVC Network 22 -23 November 2017 Imperial College, London 24 03/10/2020