Automated marking of individualised spreadsheet assignments the impact

Automated marking of individualised spreadsheet assignments: the impact of different formative self-assessment options Paul Blayney University of Sydney Australia p. blayney@econ. usyd. edu. au Mark Freeman University of Technology Sydney Australia mark. freeman@uts. edu. au

Outline 1. 2. 3. 4. 5. 6. Objectives & motivation Relevant research Context & CAA tool Method Results Conclusion

Research objectives & motivation for the paper n Objective n n To demonstrate how spreadsheets can be used to provide formative feedback to students To investigate the effects on students of different types of formative feedback (usage restrictions) Effects on academics and the university Motive n n n Higgins 2002 – lack of feedback research Improve academic productivity Improve student motivation, knowledge and skills

Relevant Literature n n n IT and learning - no significance difference n Russell (99) Limited impact when IT introduced into learning if no change in assessment n Biggs (99), Alexander & Mckenzie (98) New opportunity set for CAA n Scanned paper MCQ n Specialist software (e. g. Web. MCQ, Question. Mark) n Standardised course mgmt. system (Web. CT)

CAA Opportunities 1. 2. 3. 4. 5. 6. 7. 8. 9. Assign assessment Clarifying assessment Complete assessment Deliver formative feedback Submit summative assessment Mark assessment Generate feedback Deliver feedback Evaluate assessment effectiveness

The CAA tool Overview n n Worldware …Spreadsheet software (e. g. Excel) Excel VBA (Visual Basic) incorporated to provide feedback and mark numerical problems Problem solving application n n Financial accounting Management accounting Tax Finance Etc (maths, stats, physics)

CAA Opportunities 1. 2. 3. 4. 5. 6. 7. 8. 9. Assign assessment Clarifying assessment Complete assessment *** Deliver formative feedback *** Submit summative assessment Mark assessment *** Generate feedback *** Deliver feedback Evaluate assessment effectiveness

Research Method n Students n n Pre and post course surveys Review of unsolicited comments from online discussion forum Analysis of learning behaviour from files Staff n n n Academics n VBA specialist reflections n Users interviews & surveys Department/Faculty n Teaching & learning specialist interview n Head of department interview Uni n IT interview

What a student does 1. 2. 3. 4. 5. Receives weekly assignment file Completes it Self-assesses draft answers (different options) Submits final weekly assignment Gets mark

Demographics of students surveyed n Gender: n Age: < 21 = 92% n 1 st language: English 49% n Male 53% Female 47% 21 -25 = 7 >25 = 1% Other 51% Working for $’s – hours/week: none 1 -5 6 -10 > 10 37% 15% 23% 25%

Student perceptions overall Pre Post Feedback crucial Pseudo self-assessment gauge ability Pseudo self-assessment taken serious 94% 82% n/a n/a Self-assessment motivated learn Self-assessment motivated try Productive way to learn Non-threatening Confidence in CAA Workload for summative value OK CAA developed accounting problem solving CAA developed spreadsheet skills CAA more enjoyable n/a 76% 64% n/a n/a 60% 82% 76% 79% 55% 64% 61% 73% 75% General 1. 2. 3. VBA 1. 2. 3. 4. 5. 6. 7. 8. 9. .

Demographic analysis of student pre-course perceptions that “feedback is crucial” Gender 96. 5% (M) 97. 2% (F) p=0. 51 Age 97. 2% (<21) 92. 9% (>21) p=0. 05 Language 98. 4% (ESB) 95. 3% (NESB) p=. 01 Working hours 96. 4% (<6) 97. 3% (>6) p=0. 47

Demographic analysis of student post-course perceptions n Gender table 3 n n Native language – table 4 n n No significant differences Age – table 5 n n No significant differences except less confidence in accuracy with older students (p=0. 04) Working hours – not reported in paper n No significant differences except less motivation to learn with working students (p=0. 01) and lower confidence in program’s marking accuracy (p=0. 08)

Analysis of free-response comments solicited from survey Best Things n= 294 Things to Improve n = Self assessment gauges ability and allows practice reinforcing concepts 39% Greater assessment value 32% Easy marks 16% Marking accuracy 18% Novel way to learn 12% Instructions & format 15% Flexibility of IT assignment 8% Don’t require formulas 11% Spreadsheet skills developed 4% More feedback 2% Other 21% Other 170 22%

Analysis of free-response unsolicited comments n n Online discussion forum (Blackboard) Bulk n Administrative n n n File location on network? Digital drop box workings? Ad hoc n n Content Assignment marking system

Student learning behaviour with different self-assessment options n Ave. mark Limited number of self assessments (max 10) allowed (n = 873 or 10. 1%) All entry cells must be attempted before self assessment allowed (n = 442 or 5. 1%) Unlimited self-assessments (n = 7, 285 or 84. 8%) Option 1 students performed best: table 7 (p=. 0002) 86% 67% 81%

Usage of Self Assessment Details of Self Assessment Use (1 student for 1 assignment) 97 times !!

What the academic does n Steps to produce self-assessable files 1. 2. 3. 4. 5. 6. 7. 8. Design question Create Excel spreadsheet file Add VBA forms and code modules Create unique files for students Update VBA code in Master Marker file Provide unique files to students Receive files submitted by students Run Master Marker file. Post results.

Academic staff (developer) reflections and perceptions n The Developer n n Accountant by training (academic since 1985) Lotus 123 user from 1983 to 2000 Close working relationship with two regimes of IT staff The CAA Tool n Initial system developed over 10 years ago n n n Converted to Excel in 2001 Used in business education n Lotus 123 worksheet files and macros UG & PG and Professional Education E. g. Financial Accounting, Management Accounting, Tax, Information Systems Assignment submission n n Email CMS Digital drop box

Academic staff (developer) reflections and perceptions n Benefits n n n n Improve learning Productivity Opportunity for reduced teaching load General support – colleagues and HOD Opportunity for creativity and challenge Research opportunity Contributing to department productivity and reputation Costs n n n Exposed to vagaries of IT Time consuming trouble shooting student errors When outsourced… n n Exposed to poor academic practice Reliance on others’ expertise

Other academic users’ perceptions Free response comments from survey (n=8) support interview feedback Best Things to Improve Timely feedback to students 80% Digital drop box assignment submission 80% Allows regular assignments without costly marking 60% Marking accuracy 60% Developer’s familiarity with content facilitated integration 40% More feedback as to why 40% answers are wrong

Other academic users’ perceptions: potential diffusion of this particular CAA innovation Rogers (95) 5 factors effecting the mainstreaming of innovation (n=8) Strongly agree Agree Undecided Disagree Strongly Disagree Relative advantage 88% 12% - - - Trialabiltiy 25% 12% 63% - - Compatibility 25% 50% 25% - - Complexity 25% 62% - 13% - - 36% 64% - - Factor Visibility

Effects on Department / Faculty n Benefits n n n Learning reputation for innovation Improved productivity Intra-departmental cooperation Enhanced morale by supporting innovation Plagiarism disincentive n n Costs n n n Via file individualism Individual data Linked cells Need to protect against intellectual property exit “Black box” Back up re reliance on developer Exposure to system reliability (CMS, Internet, viruses) System capacity requirements

Effects on University n Benefits n n n Boundary testing IT reputation Learning reputation Productivity Breakdown of academic boundaries Costs n n Assessment innovation is high risk IT assessment innovation is even riskier

Conclusions n n n Confirmation that feedback is crucial Best learning where students had a limited number of self-assessments Limitations n n Specialist VBA required Sydney University context Student propensity to circumvent Controlling assignment difficulty

Conclusions n Future research n n Effect of different self-assessment options on summative assessment Closer analysis of self-assessment and different learner attributes Relationship between self-assessment assignments and assessment value and assignment frequency Self-assessment options and learner confidence