SOPHISTICATED TASKS IN EASSESSMENT WHAT ARE THEY AND

SOPHISTICATED TASKS IN E-ASSESSMENT WHAT ARE THEY? AND WHAT ARE THEIR BENEFITS? Andrew Boyle Research & Statistics team QCA

Importance of e-assessment n Important in major government initiatives n e. Learning strategy n White Paper on 14 – 19 education n QCA’s e-assessment vision n n E-assessment commonplace by 2009 Currently developing KS 3 ICT test n CAA well established in UK HE

Importance of sophisticated tasks n Substantially different task types in the near future n From pencil-and-paper tests and from early CAA n Tomlinson report/14 – 19 White Paper n E-assessment not limited to multiple-choice testing n Potential to test learners in both structured and unstructured environments n Both short and long answer questions n Ability to use techniques (sic) such as video clips n Assess a wider range of knowledge, skills and understanding than possible at present

Possible e-assessment futures n ‘Intelligent measurement’ n Sophisticated assessment methods (e. g. simulations) become more practicable n ‘Reinvented’ learning and assessment n Radically different assessment tasks n Integration of learning and assessment n Highly dynamic and adaptable content

Examples of sophisticated tasks n World Class tests n Key Stage 3 ICT test n TRIADs n NAEP-TRE n Concept-mapping tasks

Taxonomies of e-assessment tasks n Parshall’s five-dimensional framework n Item format n Response action n Media inclusion n Level of interactivity n Scoring algorithm

Towards a definition? n Parshall n Five dimensions likely to be inter-related n Each dimension acts as a continuum n No simple-to-sophisticated cut points n Difficult to find a definition n Cf Mackenzie n Possible to suggest core features

Core features of sophisticated tasks n Contain media-rich stimulus material Would rule out essays – even machine-marked n Test taker interacts with stimulus in various ways n Choosing responses (various ways) n Constructing responses (writing answers, plotting data points) n Directly demonstrating capability (simulations) n Test taker produces complex work n More complex than simple CAA, not necessarily more sophisticated than pencil-and-paper tests n

Comments on core features n Must contain several features to be ‘sophisticated’ n Notion of ‘sophistication’ always linked to other aspects – not just task design Test model n Type of data produced n Marking n Reported information n Assessment purpose n n Description of tasks as ‘sophisticated’ usually amounts to claim about merits of tasks/test

Possible benefits of sophisticated tasks n Measuring different things n Addressing different assessment purposes

Measuring different things n Simple items based on a ‘dated form of psychology’ n Learner as a ‘collector of facts and skills’; added to repertoire independently n At odds with recent understanding of learning and experts’ cognition n Conventional test models at odds with test takers’ cognitive strategies n n Success in tests depends on at least two dimensions (accuracy and speed) n Test-taking strategies Models should account for varied sources of difficulty

Measuring different things (2) n Curriculum coverage n ‘Higher-order learning outcomes (HLOs)’ n n n Valued in curriculums, but not easily tested by simple item formats Simple-item tests might not represent the curriculum adequately Particularly important if modern curriculums emphasise new types of abilities, etc. n n ‘Shared rhetoric in education’ Emphasis on: Maths and Science; ICT; problem solving and communication

Definition of ICT capability n Construct measured in KS 3 ICT test n Technical and cognitive proficiency to access, use and communicate information using technological tools n Purposefully applying technology to solve problems, analyse information, develop ideas, create models and exchange information n Discriminating in use of information and ICT tools n Contrasted with ICT skills n Technical competences to carry out tasks using common software applications

Challenge for sophisticated-task users n Sophisticated tasks depart from outdated psychological models Need for new mental models n Significant challenge to researchers n Account for implications of sophisticated tasks n When specifying assessment model n During validation n

Threats to validity from sophisticated tasks n Construct-irrelevant variance n Sophistication and complexity in user interfaces - ‘nuisance variables’ n Detract from validity n Differential impact of new task styles n Learners of differing backgrounds n Different cognitive styles/dispositions

Different assessment purposes n Substantial strand of recent research addresses assessment purposes Formative assessment can improve learning n Formative assessment wide-reaching notion n n Applies to formal test instruments and to day-today teaching techniques (e. g. questioning styles)

Formative e-assessment n Potential of e-assessment to provide useful feedback on aspects of sophisticated tasks n Well-designed formative information more supportive of learning than simple right|wrong n Varied presentation styles Providing feedback in range of visual and interactive modes n Might include a wider range of learning styles n

Limits to the optimism (1) n E-assessment is expensive to develop n Costs of e-assessment shifted towards front of cycle n Large budgets necessary for test development n Especially true of sophisticated tasks n Large organisations tend to have most resource to commit to big projects n n Generally focused on summative assessment Often conservative, risk averse n Makes use of sophisticated tasks for innovative formative assessment less likely

Limits to the optimism (2) n Writing good test items is difficult. n Good quality items take a long time to produce; iterations of reviewing, editing, etc. n Teachers don’t necessarily have item writing skills n Claim that difficulty in writing good items is particularly pronounced for sophisticated tasks n Sophisticated tasks n Expensive n Slow to develop n Not easily written by non-specialist n May be difficult for teachers and learners to ‘own’ n Lack key characteristic of formative assessment n Assessment closely integrated with classroom practice

Discussion n No clear-cut definition of ‘sophisticated tasks’ n Device to distinguish from alleged reactionary use of simple question styles in early CAA n Critique of benefits of sophisticated tasks n Implies a position of informed scepticism n Central assertion is accepted n n Sophisticated tasks will soon be ubiquitous They will bring real benefits Benefits have been asserted rather than proven Need to move to informed implementation

Discussion (2) – the two benefits n Assessment of HLOs n Formative assessment n Often confounded – they are distinct n Argument stronger in case of HLOs n Empirical evidence in near future n Sophisticated tasks’ ability to facilitate formative assessment – less secure n n n Difficulty in authoring tasks – continue to be created by large central organisations Formative assessment ought be to low-tech – usable in everyday classrooms Use of day-to-day software applications formative assessment – e. g. track changes in word processors

Other work to do on sophisticated tasks n Deriving scores for each task n Measurement models n summarise scores n provide meaningful information about performance/ability, etc. n Validation of tests

Mackenzie definition n Advanced Computer-Based Assessment … n goes beyond the simple multiplechoice/response item types into the area of complex question types, adaptive branching, scenarios and simulations of real life situations or problems. In formative mode, feedback may be immediate, extensive, contextsensitive and include annotation of diagrams, links to web resources or full courseware/elearning tutorials making it an extremely powerful learning tool.