21042015 Electronic Management of Assessment Electronic management of

21/04/2015 Electronic Management of Assessment

Electronic management of assessment Overview » Introductions » Jisc Electronic management of Assessment (EMA) project and study » Headline findings so far » Prioritising challenges - activity » From challenges to solutions » Solution scoping - activity 2

Electronic management of assessment 3 Electronic management of assessment (EMA) project Background and context » Recently completed – a 3 year programme exploring technology-enhanced assessment and feedback » 4 main themes: › Feedback and feed forward › Enhancing student employability › Strategies for change › Electronic management of assessment (EMA) http: //bit. ly/jisc-assess and http: //bit. ly/jiscdsaf CC BY-NC-SA 2. 0 jackhynes

Electronic management of assessment 4 Electronic management of assessment (EMA) project Background and context » EMA - benefits and drivers are clear » Challenges surfacing relating to process, practice and technology » Working in collaboration with the Heads of e-Learning Forum (He. LF) and UCISA to maximise the benefits technology can offer in the management of assessment CC BY-NC-SA 2. 0 ©comedynose via Flickr

Electronic management of assessment 5 The EMA study Aims » To gather examples of practices, policies & process as well as approaches to implementation » To identify the challenges around EMA, in terms of pedagogy/culture, process or technology and to identify existing solutions where they can be found » To provide interim guidance to institutions on the state of play and effective practice with approaches to implementation » To make recommendations to institutions, systems providers and Jisc on priorities to address CC BY-NC-SA 2. 0 ©comedynose via Flickr

Electronic management of assessment Information sources » He. LF EMA surveys 2011, 2012, 2013 » Jisc assessment and feedback programme (2011 to 2014 involving over 30 institutions) » Other Jisc learning and teaching related projects » Literature review » Online questions (April 2014) – 90 responses from 70 different institutions (65 HE and 5 delivering HE in FE) covering all four UK nations » Think Tank (May 2014) 32 institutions represented » Interviews with learning providers 6

Electronic management of assessment 7 Headline findings » Nuanced responses but “The use of technology is now a fundamental part of broad trends visible the support for assessment » Localised initiatives but and feedback practice across beginning to scale up » System integration is a key the sector, but there are few examples of fully integrated problem area approaches to supporting the » EMA exposes variability in whole assessment and business processes feedback lifecycle” Jisc, 2014

Electronic management of assessment 8 Lots of localised initiatives How extensively has your organisation applied EMA? 120 100 N/A Small scale pilots Large scale pilots Mandated locally Mandated organisation wide Don't know 80 60 40 20 0 e-Submission e-Marking e-Feedback e-Exams

Electronic management of assessment Loose system integration 9

Electronic management of assessment Variation in business processes 10

Electronic management of assessment Pain points 11

Electronic management of assessment What causes the pain? » Resistance: many stakeholders take narrow view; little joined-up thinking » Scaling up: may expose technology issues & ‘harden’ staff attitudes » Feedback: storage & access issues; good practice can be hard to scale » Practice: may be simple but process workarounds cause frustration & resentment » Technology: limited range of core products how hard can it really be? » Formative approaches: less well developed than summative » Anonymity: both ends of spectrum motivated by student pressure 12

Electronic management of assessment A Lifecycle Model 13

Electronic management of assessment Top 10 Challenges 14

Electronic management of assessment Top 10 Challenges 15

Electronic management of assessment Activity - the challenges How? 1. Review the challenges as 2. 3. 4. identified in the sheets on your tables (or visit the blog) Individually or in pairs tick the challenges that are causing you pain As a table, discuss and rank which you think are the biggest barriers (1 = most important, 5 = least important) Share on your tables any enablers/solutions to these problems 16

Electronic management of assessment 17 Where are we now? » Review of the EMA landscape across UK HE and FE completed. » Identified and prioritised the challenges around EMA » Workshops in Dec 2014 and Jan 2015 explored possible solutions to these prioritised challenges http: //ema. jiscinvolve. org/

Electronic management of assessment 18 Where next? Phase 1 Landscape March – July 2014 review – Landscape review –challenges providing initial prioritised challenges identification of prioritised challenges and outline of the state of play with regards to EMA Moving from challenge to solutions – 4 projects going forward Ongoing capture of resources – ‘EMA toolkit’ Phase 2. 2 Oct – Aug 2015 Ongoing development of resources for the ‘toolkit’ based on the assessment and feedback

Electronic management of assessment Four projects » EMA Requirements Map » Feedback hub » Reliable submissions » EMA systems integration web resource » Assessment & feedback toolkit 19

Electronic management of assessment EMA requirements map » Project to identify, validate and specify key EMA requirements and workflows, through development of generic models that represent the diversity of practice. » Providing clarity and transparency on workflows (particularly from submission to return of grades). Helps systems suppliers better support good pedagogic practice and helps institutions review their own practice. » Develop a visual way of presenting workflows; map current systems to workflows; and engage with suppliers and developers to fill gaps. 20

Electronic management of assessment Feedback hub » System independent tool for aggregating, organising and presenting feedback at a programme level, for both staff and students. Also enables interaction around feedback between staff and students. » Students benefit from aggregated view of their feedback to support self -reflection on progress; lecturers see a more holistic view of students’ progress and better understand an individual’s progression to identify where intervention or support is needed. 21

Electronic management of assessment Systems integration » Resource to address lack of interoperability between marking systems and student records systems and subsequent need for ‘workarounds’ by staff. » Helps institutions find solutions to EMA systems integrations issues via improved workflows and by sharing existing integrations (use cases and code). » Gaps can be surfaced and prioritised for potential development. » A community of practice might support this resource. 22

Electronic management of assessment EMA toolkit » Web-based toolkit of resources: practices, processes, policies, technical integrations etc based around the assessment and feedback lifecycle. » Examples of solutions , enabled by technology, based on pedagogy and underpinned by research, that can be repurposed for local contexts. » Addresses the question of ‘what does good assessment design look like? ’ and enhances assessment literacies of staff and students. » Navigation routes through the toolkit e. g. by identified workflow and stakeholder groups (academic staff, administrative staff, student 23

Electronic management of assessment Activity Choose one project area to work on The aim is to help to shape the detail of these projects. Discuss: » What would you like to see from this ‘solution’ and how might you use it? » What do you have that you can contribute? » How can we best reflect the needs of HE and FE within this project? » Sign up to get involved: http: //ema. jiscinvolve. org/wp/2015/04/05/solution-developmentis-under-way-can-you-help/ 24

Electronic management of assessment Activity Choose one project area to work on: The aim is to help to shape the detail of these projects. Discuss: » What would you like to see from this ‘solution’ and how might you use it? » What do you have that you can contribute? » How can we best reflect the needs of HE and FE within this project? › Toolkit: http: //bit. ly/EMAyork › EMA requirements map http: //bit. ly/sysyork › Feedback hub scoping activity http: //bit. ly/feedyork » Sign up to get involved: http: //ema. jiscinvolve. org/wp/2015/04/05/solution-development-isunder-way-can-you-help/ 25

Electronic management of assessment Find out more: » Lisa Gray lisa. gray@jisc. ac. uk » Landscape report available from: bit. ly/jisc-ema » Join the conversation on the blog: ema. jiscinvolve. org/ and on twitter #jiscassess » Join the mailing list: www. jiscmail. ac. uk/tech-enhanced -assessment » Guides, case studies, videos and other resources on assessment and feedback: bit. ly/jisc-assess 26

Any questions?