Action Research Meets Outcomes Assessment UAs VALA and

Action Research Meets Outcomes Assessment: UA's VALA and the D 2 L LMS n n n John Baker, President and CEO, Desire 2 Learn Inc. Garry Forger, Technology Coordinator, The University of Arizona Jennifer Franklin, Instructional Dvlp & Assessment Specialist, The University of Arizona Copyright 2004. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

Setting the stage…

Two perspectives… “Teaching is the intentional arrangement of events so that appropriate learning will happen. ” Robert Menges and Ann Austin (1994), AERA "Instructional Design is a field that prescribes specific instructional actions to achieve desired instructional outcomes; the process decides the best methods of instruction for enacting desired changes in knowledge and skills for a specific course content and learner population. Barbara Seels and Rita Richey (1994), AECT

So, how do we… Bridge the gap between teaching and course development to help faculty and the staff who support them to be able to intentionally arrange events in e-learning environments so that appropriate learning is most likely to happen?

Some background… n University of Arizona Context n Learning Technologies Center n FIPSE-LAAP VALA Project

Optimistic view …

Another view… C. J. Bradford

“The people in the picture know there is a problem. The game isn't working. However, what the people cannot perceive is that the crucial knot that needs to be untangled is outside their dimension of reality. ” www. stringfigurestore. com

VALA and the Desire 2 Learning Management System Action Research Meets Outcomes Assessment at the University of Arizona NLII-EDUCAUSE Meeting, January 2004

Presenters: n John Baker, President and CEO, Desire 2 Learn Inc. n Garry Forger, Assistant Director, Learning Technologies Center, University of Arizona n Jennifer Franklin, Ph. D. , Instructional Development & Assessment Specialist, Learning Technologies Center, University of Arizona

Session Agenda: n Presentation Goals & Topics 1. 2. 3. 4. 5. n Why VALA, what is it, and how will it work What is the VALAnator? What does it take to make a “VALAnator”? So why Desire 2 Learn? So why UA? , D 2 L Speaks… Discussion

Wish we could discuss today, but probably can’t… n n n The larger “ontological soup” Tower of EDspeak Babel/Babble: and communities of discourse The culture of teaching and learning in higher education – Sof. T-Sof. TL The problems of implementation in rapidly changing technology environment

Why VALA?

VALA Premise… There exist optimal instructional methods, strategies, and tactics to enhance and ensure learning and which can be identified by studying learning outcomes and their relationship to factors such as: – Learner differences – Content and task differences – Instructional treatment differences

Sources of Knowledge for College Teaching* 1. Formal preparation in a discipline or field instruction on the content or subject matter we teach and, sometimes, on teaching our content 2. Observations, demonstrations, and “stories” of others 3. Personal experience through the “practice” of teaching (personal teaching knowledge and theories) 4. Drawing on others’ systematic research and theory on learning and teaching (scholarly teaching) 5. Contributing to systematic research and theory on learning and teaching (scholarship of teaching) * G. Roger Sell (2003)

Some Impediments to Effectively Choosing Instructional Strategies Lack of knowledge/skills to make new choices in the design and development process Lack of feedback to signal the need to reconsider a choice in a course that is already underway lack the incentive, specific skills and resources (time, technical support, staff support) needed to conduct empirically sound evaluation studies or educational research

and … Habit…AOTBE. . We teach as we were taught and when creating web-based courses we try to replicate face to face instructional methods in elearning Few of us are exposed to a wide range of examples of strategies and tactics for teaching in distance courses and we have few constructs to guide us in adopting and adapting them when we do encounter them. We evaluate student learning but we rarely have time to systematically evaluate the effectiveness of the instructional choices we make.

So, What Is VALA? The Virtual Adaptive Learning Architecture is n system n concept n strategy n project

VALA Goal 1 n Promote reflective practice in teaching and course development in e-learning contexts by enabling inquiry ranging from formative evaluation to informal or formal research concerning how to “arrange events…”

VALA Goal 2… Enable LTC staff to participate in the research, development and dissemination of new knowledge related to the effects of web-based instructional methods, strategies and tactics on learning outcomes and the relationship of those effects to individual differences in learners

VALA Goal 3… n Contribute to the development of metadata and standards related to “intentionally arranging events” that can productively connect to SCORM/IMS and, ultimately, Educational Modeling Language (EML) research

VALA Goal 4… n And ultimately reach full implementation and dissemination of VALA technology related to the FIPSE-LAAP grant focused on learner-characteristics and adaptive instruction in web-based courses where it all started.

How will VALA work?

VALA System Components … 1. A set of resources for faculty and staff training, development, and support 2. Resources for knowledge R&D and management including the VALA Thesaurus 3. A software tool set to support inquiry (research/action research/evaluation, aka the “VALAnator” 4. An organizational development plan 5. Technology infrastructure including the Desire 2 Learning LMS and other e-learning tools.

1. Training and Development and Support Resources… n Will include a collection of “just in time” learning objects* tailored for faculty These include online collections of selected – conceptual frameworks for describing learner/strategy interactions, typologies and models for instructional development models (e. g. , NLII supported MOATS Project) – situated examples of instructional strategies, tactics and methods – typologies of individual differences and instruments for measurement * Learning objective is faculty development and training in systematic instructional development techniques.

2. Knowledge R&D and Management – The VALA Thesaurus, an annotated, illustrated thesaurus of terms related to “intentionally arranging events” provides coherently structured, theory-sensitive terminology along with annotations including definitions, references and citations, and web-based examples of learning objects scaled from simple learning objects to full scale courses and programs. – The VALA ontology, a prototype supporting the Thesaurus that we will assess as a model for extensions to SCORM/IMS in order to further Educational Modeling Language (EML) research and development

3. Inquiry support system We’ll come back to this item.

4. Organizational Development Strategies To assure vitality and sustainability develop opportunities to: Disseminate n Evaluate n Enhance and revise n

OD Strategies… n Organize faculty user group, build community Layer in Academic Offerings n Scout and promote VALA & D 2 L to n Specific outreach to University organizations promoting the Scholarship of Teaching and Learning. Outreach to the larger communities of practice and research n n – Graduate Teaching Program (a 10 unit certificate program on college teaching for graduate students) – Sierra Vista Ed Tech Masters Special Problems Course. – grant funded or special projects needing technical support formative or summative evaluation – Innovative faculty – Parnter with Academic Co. Labs (SCORM/IMS

5. Technology Resources A partial list: n Desire 2 Learn LMS n Web. CT and Blackboard n Old Pueblo Moo n Coursebuilder (middleware) n VIRAGE, a searchable video library n Polis n Multimedia Board System n Statistical tool sets

3. The VALAnator n a wizard that assists the user in conducting studies using assessment, tracking, and user progress data within a SCORM/IMS compliant LMS in order to support users in developing more effective instruction and evaluating courseware and learning objects.

Rationale for the VALAnator… If a system were available to easily “automate” the process needed to design, develop and implement an evaluation or research study within a web-based course and which did not require a high degree of research or assessment expertise from faculty, more faculty would be able and willing to conduct pilot and small scale evaluation or research studies in the context of everyday instruction in web-based courses.

It would enhance user skills and/or mitigate lack of skills in n n Research methods and study designs and protocols Test and measurement skills Ethics and IRBs Data competencies (develop datasets, conduct statistical analyses, use computer systems and stat packages, presentation/reporting/charting/graphing * Research, Evaluation, Test and Measurement, Assessment

Broadly VALAnator will … 1. Query the user and define purpose and scope of study a. b. c. d. 2. 3. 4. Typology of research purposes Typology of research methods, for each Typology of study designs and their requirements with respect to external and internal validity, sampling and assignment, statistical methods Typology of instructional variables Define “a study”, map instructional variables to LMS data sources, test for requirements of research method/study design Conduct a study Acquire, analyze, and report results.

then… 1. 2. 3. 4. 5. 6. Identify an instructional issue State as series of single sentence questions State as hypothesis and null hypothesis of what would and would not be true Operationalize study in terms of actual variables to be measured (dependent, intervening, covariate) Classify type of inquiry Catagorize variables by type of measurement (nominal, ordinal, interval, ratio)

Translate concepts to “variables” and get data Goal Elements Intention to Instructional Objectives Arrangement to operationalized descriptions of methods Events to activities, processes, sequences Appropriate learning to learning outcomes Data Sources Ask instructor Map to D 2 L Tools and Content Map to D 2 L Assessment and User Progress Data

Compare and contrast… Same word used many ways in the contexts of soft… fuzzy… competing paradigms. . Ill structured categories and hierarchies. . . unreliably used to tag documents. A nice tidy, epistemologically “crisp” classification system Nearly any page of the Eric Thesaurus

“Arranging” Processes Classes of variables* Design & Deploy & Admin & develop deliver Maintain partial contexts agents Teachers, TAs, etc. Learners intentions Strategies methods tactics events “syllabus” ISD, artifacts Blueprints, do Learning cs outcomes Course admins; curriculum committtees reflection “showtime” teaching, learning Register, enroll, review Evaluate SET, P&T

Theory filters Cognitivist n Behaviorist n Constructivist And a Metatheoretical strategy to match scale of theory and problem, since theories blend and overlap at different levels of instruction – e. g. Macro (course) to micro (one concept) n

WHY D 2 L: LMS Requirements n n n Need a fully functional easy to use SCORM/IMS compliant LMS Need to be able to get data in and out (open database architecture) Need conditional release of content tied to ways to categorize learners in order to assign them to groups Need ways to track learners Needed a product and a vendor open to innovation

Why U of A and VALA? n D 2 L speaks…