Canvas User Experience UX Studies Presented today by
Canvas User Experience (UX) Studies Presented today by: Amy Garbrick and Maggie Frederick, Penn State University Core Research Team: ● Amy Garbrick: Director of Learning Design (College of IST) ● Andrea Gregg: Manager of Instructional Design (World Campus) ● Ronda Reid: Instructional Designer (College of IST) ● Vicki Williams: Manager, Assessment & Evaluation Research (ITS)
Canvas User Experience (UX) Studies Where It Started…with one college and it grew. . . � A simple thought: College of Information Sciences and Technology (IST) wanted to find out how students approached the IST course design in Canvas � Decided to test students: College of IST came up with a 8 tasks for standard course items and then recorded students as they were going through the tasks 2
Definitions: Canvas User Experience (UX) Studies @ PSU 10 min Overview � e. Learning Shops = Learning Design Units within a college/campuses that design and develop online courses; All support online some also support Resident Instruction (RI) courses � World Campus = Delivery unit for courses designed by e. Learning Shops; Also has it’s own Learning Design unit that creates courses for Colleges/Campuses who do not have their own e. Learning Shop or the capacity 3
2 different efforts: Canvas User Experience (UX) Studies @ PSU 1. Talk-alouds � Round 0: College of IST Student Interns � Round 1: College of IST WC Students � Round 2: Cross-Unit UX Study -- EMS, Liberal Arts, World Campus 10 min Overview 2. User Survey � Hot off the press results � Student design preferences � 5, 000 World Campus students received survey 4
Canvas User Experience (UX) Studies @ PSU 10 min Overview Tell us what you think… Students in the talk-aloud sessions most often went to: 1. 2. 3. 4. Syllabus Modules Schedule Homepage 5
Canvas User Experience (UX) Studies @ PSU 10 min Overview Here’s what our students said. . . Students in the talk-aloud sessions most often went to: 1. 2. 3. 4. Syllabus Modules Schedule Homepage 6
Modules, Modules! Canvas User Experience (UX) Studies @ PSU 10 min Overview “Modules are the primary source of information regarding courses, understanding that is paramount. ” ~Subject 1 “Pretty much everything is under the modules. That's typically where I would go. “ ~Subject 2 7
Canvas User Experience (UX) Studies @ PSU 10 min Overview Tell us what you think. . Students in the survey said the most important component on the Syllabus was: 1. 2. 3. 4. Grading Weights Grading Scale List of Course Materials Course Schedule 8
Canvas User Experience (UX) Studies @ PSU 10 min Overview Here’s what our students said. . . Students in the survey said the most important component on the Syllabus was: 1. 2. 3. 4. Grading Weights Grading Scale List of Course Materials Course Schedule 9
Canvas User Experience (UX) Studies @ PSU 10 min Overview Tell us what you think. . Students in the survey said the most important component in the Course Schedule was: 1. 2. 3. 4. Grades Assignments Due Dates Lesson Date Ranges 10
Canvas User Experience (UX) Studies @ PSU 10 min Overview Here’s what our students said. . . Students in the survey said the most important component in the Course Schedule was: 1. 2. 3. 4. Grades Assignments Due Dates Lesson Date Ranges 11
Why it’s Important to Talk to Learners “[A]sking students about their experiences of e-learning gives surprising, individual, and frequently contradictory results and this in itself is good reason for listening to students more often and more thoroughly if we are to avoid making assumptions about their experience from our teacher-centred view. ” (Sharpe & Bennefield, 2005) 12
PSU Who we are 13
14
Recent Research WC LMS Requirements (2013, qual. ) What are World Campus requirements for the LMS? 2 Int. & 4 focus groups - w/ WC students (12), faculty (13), staff (12) Identified requirements (e. g. mobile, easy, discussion, social presence) Report presented to TLT, Bb, WC; Used in LMS pilots Online Student Interface Study (2015, quant. ) What are student perceptions in navigating online courses? Sub-committee of the DMT; OAR survey; 387 respondents Mostly +; Critiques: locations for due dates, assignments Student Interface Task Force using report to make recommendations Socient Study (2015, quant. ) What improvements needed for outstanding student exp. ? Socient survey: 1649 respondents Mostly positive w/ areas for improvement (e. g. career services, advising) LD improvements: usability, tech consistency, instructor involve. 15
What We’ll Cover Today � Canvas User Experience (UX) Studies � Talk-alouds � Round 0: College of IST Student Interns � Round 1: College of IST WC Students � Round 2: Cross-Unit UX Study -- EMS, Liberal Arts, World Campus � User Survey � Hot off the press results � Student design preferences � 5, 000 World Campus students received survey 16
Canvas User Experience (UX) Studies 17
Canvas User Experience (UX) Studies Where It Started…with one college and it grew. . . � A simple thought: College of Information Sciences and Technology (IST) wanted to find out how students approached the IST course design in Canvas � Decided to test students: College of IST came up with a 8 tasks for standard course items and then recorded students as they were going through the tasks 18
(Task 1) You have a question on the Lesson 07 Web Usability essay assignment. How would you find out more about this assignment? Examples of What We Asked (Task 2) Your instructor sent an email reminding you to complete the Lesson 03 readings. How would you locate your reading assignments? (Task 3) You would like to receive a weekly email notification regarding course announcements. How would you set-up this notification? 19
Tested the Test (aka Canvas UX Round 0): Bring in the Guinea Pigs Image: Megan van der Elst (http: //jezebel. com/hipster-guinea-pig-professor-boo-was-cute-before-it-1595022520) 20
Canvas UX Round 1: IST World Campus Students �Recruited 5 undergrad IST World Campus students �Willing to come to University Park for the study �IRB Approved study �Gave them a $50 gift certificate to Amazon. com � 4 male, 1 female: � 1 student: aged 20 -23 � 1 student: aged 24 -29 � 2 students: aged 30 -39 � 1 student aged 40 -49 21
CROSS UNIT Canvas UX Round 2: Cross-Unit Canvas UX Study IST WCLD EMS LIBERAL ARTS TLT Our Team: Ronda, Tao, Tracy, Mike, Maggie, Vicki Jenn, Amy, Andrea The combination of prior studies with the Canvas transition has provided opportunities for more collaborative research and efforts to “get it right. ” 22
IST, EMS, Liberal Arts, WCLD, w/ TLT Canvas UX Round 2: Cross-Unit Canvas UX Study � Maggie Frederick -- College of Earth & Mineral Sciences (EMS) � Jennifer Gray & Tracy Wallace -- College Liberal Arts) � Mike Brooks -- World Campus, Learning Design unit (WCLD) � Each “shop” recruited 5 WC learners; each shop used own course design 14 new participants + 5 from IST = 19 in total 23
IST, EMS, Liberal Arts, WCLD, w/ TLT Canvas UX Round 2: Cross-Unit Canvas UX Study �Completed set of tasks, used talk-aloud protocol �Audio-, video-, and screen-recorded � IRB Approved study �Gave them a $50 gift certificate to Amazon. com � 15 cards were funded by central ITS 24
Canvas UX Round 2: Cross-Unit Canvas UX Study �Currently undergoing data analysis on audio, video, and transcripts � Should have more results soon �But, anecdotally, did see some of the same themes we saw in IST’s Canvas UX Round 0 and Round 1 testing 25
“Not too bad, easy once you get familiar” ~ Subject 1 (Task 2) Canvas UX Round 0 and Round 1: “Easy to find. ” ~Subject 5 (Task 4) What Did We Find in IST Testing? “Once I understood what modules were it was fairly easy to find. ” ~Subject 4 (Task 1) Image: Staples. com 26
Canvas UX Round 0 and Round 1: Modules, Modules! http: //mgoblog. com/mgoboard/dantonio-elicits-sympathy-jamie-samuelson-detroit-free-press 27
Canvas UX Round 0 and Round 1: Modules, Modules! “Modules are the primary source of information regarding courses, understanding that is paramount. ” ~Subject 1 “Pretty much everything is under the modules. That's typically where I would go. “ ~Subject 2 28
“That is really simple to find and I find that a lot easier than the current method of ANGEL…” ~Subject 4 Canvas UX Round 0 and Round 1: “I think this is a big improvement over ANGEL. ” ~Subject 1 Canvas vs. ANGEL “That's a lot nicer than the current system. ” ~Subject 5 Image: Smeal 29
User Survey 30
Mixed Methods Analysis Quantitative Canvas User Survey �Descriptive and tests for significance �Reliability Qualitative �Open-ended survey responses analyzed in NVivo for themes & patterns in responses �Produce Word clouds / trees, etc. 31
Survey Canvas User Survey • Survey developed initially in collaboration with IST faculty Jess Kropczynski to identify user behaviors and preferences in Canvas screen design • Modified by cross-unit team for this round of data collection 32
Participants Canvas User Survey ● Sent to almost 5, 000 students whose home campus is WC and who took at least one SP 16 Canvas course ○ Landing page preferences (e. g. lion, recent activity, syllabus) ○ Syllabus organization preferences (e. g. full, compressed, separate schedule) ○ Navigation menu preferences (e. g. concise, full) 33
34
Canvas User Survey: Key Findings Image: http: //iwantmyshowback. com/wp-content/uploads/2011/05/news-hot-off-the-press-whats-new. jpg 35
Home Page Canvas User Survey: Key Findings Given the four alternatives you were shown previously, please indicate which Home page you prefer by dragging and placing your selections in order of preference Modules Syllabus Recent Activity Welcome 36
Syllabus Canvas User Survey: Key Findings Given the three Syllabus alternatives, please indicate your preference for each by dragging your preferences in the order you prefer them Syllabus View A Syllabus View B Syllabus View C 37
Canvas User Survey: Key findings 38
Course Schedule on Syllabus Page Canvas User Survey: Key Findings 39
Course Schedule: Print Canvas User Survey: Key Findings 40
Most important to see in the Syllabus: Canvas User Survey: Key Findings � Course Schedule � Grading Weights � Grading Scale � List of Course Materials Most important components in the Course Schedule: � Due Dates � Assignments � Lesson Date Ranges 41
When you view your Syllabus, what information is most important to you? Canvas User Survey: Key Findings Answer Average Value Standard Deviation Responses Course Schedule 5. 55 0. 78 511 Grading weight of assignments (how much each assignment is worth) 4. 92 1. 05 490 Grading scale (what percentage 4. 78 is necessary for an A in the course) 1. 22 482 List of Course Materials (books, 4. 54 e. Reserves, software, etc. ) 1. 29 473 Instructor Information 4. 13 1. 36 440 Resources 4. 13 1. 41 391 42
Navigation Canvas User Survey: Key Findings � Assignments was the most important item to have on the Navigation menu, followed by Grades and Modules. � Students felt they did not need to see Files, Pages, Conferences, or People in a Navigation menu and were least sure of the meaning of the Pages or Conferences items. 43
Canvas User Survey: Key Findings 44
Sense of Community Canvas User Survey: Key Findings The majority of the students felt the Nittany Lion graphic gave them a sense of inclusion in the larger Penn State community, but interaction with the instructor and classmates also contributed. 45
In Closing… “If all of us are committed to developing innovative programs and policies for education, we need to marshal all of our forces and deploy all methodologies that are available to us — qualitative as well as quantitative research; the best randomized controlled trials as well as the best ethnographies, case studies, and so on. ” (Sullivan, President of UVA, @ AERA Plenary address, 2014) 46
Thanks for your time! Questions? Maggie Frederick mas 960@psu. edu Amy Garbrick amyg@psu. edu 47
- Slides: 47