Improving Assessment Efforts Integrating Educational Technology Wednesday March

Improving Assessment Efforts Integrating Educational Technology Wednesday, March 16, 2011, 8: 30 – 9: 15, Seville West EDUCAUSE Midwest 2011, Chicago, Illinois Kenneth Janz, Ph. D. AVP and CIO Winona State University Ken Graetz, Ph. D. Director, TLT Winona State University

Overview of Presentation • • Overview of Winona State University (WSU) e-Warrior: Digital Life and Learning Program Assessment of Educational Technology e-Warrior Assessment Plan Key Findings Ongoing Assessment Challenges Questions

Founded in 1858 (Normal School) Public State Assisted Winona, Minnesota 9, 000 students

Required Participation Dual-platform lease program since 1997 $480 semester

e-Warrior Video Goes Here

Winona is Unique……. . • Space Utilization (Currently at 93%) – Anytime, Anywhere Learning – Computer Labs • Computer Self-Efficacy • Access to Software • Innovative Pedagogy – e-books – enhanced communications • Faculty Can Count On What Students Can Bring To Class

Interesting Data…… Faculty Recruitment Attracting Students Awards Rankings The federal IPED’s report indicates Winona State’s five-year undergraduate graduation rate leads all other Mn. SCU Universities by at least 10 percent (IPED, 2009). 1997 WSU graduation rate was just over 40%.

*Core based on Fink (2003, p. 60 – 101)

What Makes the Target Hard to Hit? Greater academic integration New devices Better wireless access Shifting pedagogies More applications Aging laptop programs Budget cuts Device backlash

Where Are We Going • Winona State University Technology Master Plan • Runs July 2010 – June 2013 • Framework for integrating and prioritizing technology related needs

Research Questions Based on Conceptual Model *Core based on Fink (2003, p. 60 – 101)

Measuring How We Are Doing Building Longitudinal Metrics • • • Student Surveys Faculty Surveys Alumni Surveys Faculty / Student Focus Groups Collect Course Information Presentation / Publications Existing Student Success Data Facilitating Activities Unit HLC Assessment

Multiple Measures (1 of 2) • • Annual Faculty and Student Technology Survey Faculty and Student Training Counts Faculty and Student Training Evaluation Survey Alumni Survey Faculty Portfolio Archival Log Data Help Desk Satisfaction Survey Faculty and Student Focus Groups

Multiple Measures (2 of 2) • Review Selected Course Information • Review Selected Course Assignments • e-Warrior Digital Life and Learning Program meeting attendance and minutes • Showcase Survey • Presentation and Publication Counts • Number of inquiries and visits to learn about the program. • Existing Student Success Data • Educause Core Data • Application and Course Specific Measures

Key Findings and Measures

Student-Related Questions • Are students satisfied with the tools? – Do first year students find the program “lame” – Do Seniors become disillusioned with the program? – Are students adding tools not provided through the program? • Do students use the tools for academic purposes extensively? – Do they use the tools more for academic vs entertainment? – Do they use the tools inside the classroom? – Does academic tool use vary by year in school? • How do students evaluate their instructors’ tool use? – Do perceptions vary by year in school?

Student satisfaction (5 -point scale) Item Mean I use my laptop frequently for class related purposes. 4. 38 The use of a laptop is important to my academic success. 4. 37 I am satisfied with the software provided in the program. 4. 10 The laptop provided is of good quality. 4. 01 I was adequately trained to use my laptop. 3. 93 The program policies are clear. 3. 93

What grade would you give the e-Warrior Digital Life and Learning Program overall? • “A” = 38% overall • “B” = 53% overall • Both are satisfied, but Seniors grade the program lower A B C Total D First Year Seniors F 0 20 40 60 80 100

On average, approximately how many hours per day do you use your laptop? • Normally distributed, averaging 5 -6 hrs/day • No differences between first years and seniors More than 12 hrs/day 9 -12 hrs/day 7 -8 hrs/day 5 -6 hrs/day 3 -4 hrs/day Total First Year Seniors 1 -2 hrs/day Less than 1 hr/day 0 20 40 60 80 100

Of the total average time that you use your laptop during a typical day, how much of the time do you use it for class-related purposes? • Positively skewed, averaging 3 -4 hrs/day • No differences between first years and seniors • Use their laptops for nonacademic purposes about 2 hrs/day More than 12 hrs/day 9 -12 hrs/day 7 -8 hrs/day 5 -6 hrs/day 3 -4 hrs/day Total 1 -2 hrs/day First Year Seniors Less than 1 hr/day 0 20 40 60 80 100

On average, in the courses you use your laptop while in class, how frequently do you use it in class? • Students use laptops Every time the course meets in class 2 -3 courses per term Most of the time the course meets • Students do not use laptops in class everyday About half the time the course meets • Seniors use laptops in class Sometimes, but not regularly more freq. than first year Total First Year Seniors Never 0 20 40 60 80 100

Application frequency (5 -point scale) Item Mean Access course materials 4. 68 Communicate with your instructors 4. 06 Collaborate with other students in your classes 4. 02 Create class-related presentations 3. 97 Take a quiz or exam 3. 84 Take notes 3. 51

Added Software • Communication Tools – Skype – Yahoo Messenger • Creativity/Productivity Tools – – – • Final. Cut Express Audacity Google Chrome Picasa Evernote Soundflower Entertainment Tools – VLC Player • Academic Tools – – La. Te. X Dim Sum Google Earth Nook Study, other ebook readers

Use D 2 L • I feel that all instructors should use the D 2 L system as it is a great way for students to know where they stand in the course, because they are able to view their grades regularly. Therefore, it would be nice if instructors would update grades for students as well on the site. Another reason D 2 L is a great system is that it helps with contacting classmates, and is great for instructors to update with all of the information from each class. • I like D 2 L but a lot of the Minne professors don't seem to use it much. Too much paper still handed out!! not green

More Engagement • Using photos or You. Tube clips to supplement their lesson is one thing that a couple of teachers have done, that I found very beneficial. • Instructors need to understand that just reading off slides does not teach students effectively. Instead, incorporate examples or class activities or even more hands on learning. If it is a proven fact students learn more from hand on activities, you think the college would require material to be taught through hands on techniques. • Don't just read right off the slide you made, it's boring and no one learns anything.

Less Technology in the Classroom? • Thank you finally a question I have a strong opinion on. If your going to require laptops and charge as much as you do for them, no professor should be allowed to say you can't use them to take notes in class. I am well aware of the distraction they present. Figure something out, you cant be a laptop school if you have to take notes in one or two classes each semester by notebook. • I, personally, would prefer the instructors to use less technology because it is easier for me to learn by doing something myself, rather than a computer/calculator to do it for me.

Relationship With GPA • Positive correlation between cumulative GPA and frequency of laptop use in class; moderated by year in school – First year: r = +. 19 – Senior: r = +. 23

Student-Related Questions • Are students satisfied with the tools? – Do first year students find the program “lame” – Do Seniors become disillusioned with the program? – Are students adding tools not provided through the program? • Do students use the tools for academic purposes extensively? – Do they use the tools more for academic vs entertainment? – Do they use the tools inside the classroom? – Does academic tool use vary by year in school? • How do students evaluate their instructors’ tool use? – Do perceptions vary by year in school?

Faculty-Related Questions • Are faculty satisfied with tools and services? • Are faculty using tools in their courses effectively?

Item Mean Technology Support (i. e. , contacted TLT with a specific technical question about a particular tool) 4. 54 Technology Consultation (i. e. , contacted TLT for general advice on the application of technology) 4. 68 Faculty and Staff Workshops (i. e. , attended a live session in person or online) 4. 36 Customized Student Training (e. g. , requested development of student workshops or online training) 4. 71 Online Learning Materials (e. g. , accessed movies or other online help provided through TLT) 4. 50 Project Support (i. e. , worked with TLT staff on a specific, technology-related academic project) 4. 64 Blog, Wiki, and Moodle Services (i. e. , used a blog, wiki, or Moodle course on TLT servers) 5. 00 Special Application Services (i. e. , developed/used a custom or special application on TLT servers) 4. 80

Item Mean D 2 L Course Merge/Copy Service (i. e. , requested D 2 L course sections be combined or copied) 4. 37 Multimedia Lab Services (e. g. , used TLT video editing and production services) 4. 67 Video and Audio Streaming Support (i. e. , requested assistance with streaming recorded events) 4. 20 Special Event Planning and Support (e. g. , conference, festival) 5. 00 ITV Technical Support (e. g. , TLT assisted scheduling or connecting an ITV meeting) 4. 77 Class//Meeting Room Hotline (i. e. , requested immediate technical assistance in a campus room) 4. 66 Multimedia Equipment Checkout (e. g. , borrowed camera, projector, headset, clickers) 4. 71

Faculty Exchange Video Goes Here

Ongoing Assessment Challenges • IMS Analytics • Assessment Related to Faculty Development

Questions