Issues in Categorising Learning Designs HELEN WALMSLEYSMITH STAFFORDSHIRE

Issues in Categorising Learning Designs HELEN WALMSLEY-SMITH STAFFORDSHIRE UNIVERSITY

Overview Reasons for categorising learning activities Taxonomies of learning activity types Using taxonomies to categorise learning activities Development of e. DAT based on e. Design Template Content analysis and Inter-rater reliability Updated e. DAT and consultation activity Discussion

Why Categorise? Representation, comparison, evaluation, sharing…

Existing Taxonomies Ulster: receives, debates, experiments, creates, explores, practices, imitates and meta-learns Laurillard: Acquisition, Discussion, Investigation, Practice, Collaboration, Production, JISC Pedagogical Vocabularies Project reports identified that there is a ‘lack of common understanding of, and shared vocabularies for, pedagogical practice amongst teachers, system developers, learning technologists and e-learning researchers. ’ Mod 4 L project identified issues with representation of learning design

Using taxonomies to categorise Examples of issues: ◦ Exercise using the Learning Designer to map activities to teaching and learning activity types illustrated difficulties ◦ AMP (Assessing MOOC Pedagogies) involved extensive group discussion to reach agreement on categorisation of MOOC ◦ Open University programme of mapping activities to types also found the process challenging

Initial e. DAT Based on principles in e. Design Template: ◦ Open / closed activities ◦ Tutor managed / student managed ◦ Variety of interaction ◦ Variety of assessment

Content Analysis Process aims to increase validity: ◦ Codebook or categories created terminology ◦ Text analysis units determined – ‘activity segment’ ◦ Raters trained in use of categories ◦ Each rater looks at all learning activities (or sufficient overlap) ◦ Raters work independently ◦ Inter-rater reliability measured

Inter-rater Reliability Analysis using Krippendorff’s Alpha: Unlike percent agreement and Cohen’s Kappa, Krippendorff’s alpha is a measure of the disagreements in a set of data Where Do = observed disagreement and De = expected disagreement This takes chance into account as well as the magnitude of the misses. In addition, it applies to any number of observers, any number of categories, any metric or level of measurement, incomplete or missing data and large and small sample sizes alike. Acceptable levels of IRR =. 6 to. 8

e. DAT in development… Updated following pilots Activity: Look at e. DAT and sample activities Review categories Try categorising independently Compare and discuss

Bibliography Charlton, P. , Magoulas, G. , & Laurillard, D. (2012). Enabling creative learning design through semantic technologies. Technology, Pedagogy and Education, 21(2), 231– 253. Currier, S. , Macneill, S. , Corley, L. , Campbell, L. M. , & Beetham, H. (2006). Vocabularies for Describing Pedagogical Approach in e-Learning: A Scoping Study. In DCMI ’ 06 Proceedings of the 2006 international conference on Dublin Core and Metadata Applications: metadata for knowledge and learning. Falconer, I. , Beetham, H. , Oliver, R. , Lockyer, L. , & Littlejohn, A. (2007). Mod 4 L Final Report: Representing Learning Designs. Retrieved February 15, 2016, from http: //www. jisc. ac. uk/media/documents/programmes/elearningpedagogy/mod 4 lreportfinal. pdf Krippendorff, K. (2011). Computing Krippendorff’s Alpha-Reliability. Philadelphia: Annenberg School for Communication Departmental Papers. Laurillard, D. (2012). Teaching as a Design Science: Building Pedagogical Patterns for Learning and Technology. Abingdon, UK: Routledge, Taylor & Francis Group.

Rienties, B. , Toetenel, L. , & Bryan, A. (2015). “Scaling up” learning design: impact of learning design activities on LMS behavior and performance. In Proceedings of the Fifth International Conference on Learning Analytics And Knowledge - LAK ’ 15. Swan, K. , Day, S. , Bogle, L. , & Prooyen, T. van. (2015). Metaphors for Learning and the Pedagogies of MOOCs. In 2015 Meeting of the American Educational Research Association, Chicago, IL. University of Ulster. (2008). Hybrid Learning Model. Retrieved March 20, 2014, from http: //cetl. ulster. ac. uk/elearning/hlm. php Walmsley, H. (2015). The e-Design Template: A Pedagogic Guide for e-Learning Designers. In M. Maina, B. Craft, & Y. Mor (Eds. ), The Art & Science of Learning Design. Rotterdam: Sense Publishers.