Writing assessment descriptors Ensuring validity and reliability in

Writing assessment descriptors Ensuring validity and reliability in assessment Olwyn Alexander February 2015

Outline Principles of test/assessment usefulness Focus on validity: assessment construct Focus on reliability: assessment criteria

Principles of test usefulness Validity Reliability Authenticity Interactiveness Impact/Washback Practicality Bachman & Palmer, 1996

Principles of test usefulness Validity – the test actually measures the performance it claims to measure and this is appropriate for the test purpose Reliability – measurement is consistent between different assessors and different test takers Authenticity – the tasks in the test are representative of tasks in the target situation Bachman & Palmer, 1996, Alexander et al. , 2008

Principles of test usefulness Interactiveness – the tasks in the test engage the test takers’ communicative language ability, background knowledge and strategic competence Impact – the test will affect classroom teaching and learning (and wider institutional policy – entry levels) Practicality – resources will be required to develop the test and train assessors to administer it Bachman & Palmer, 1996, Alexander et al. , 2008

Test specification Who individual/one class/all classes? Why achievement/proficiency? What construct/knowledge/skills? How format/rating scales? Bachman & Palmer, 1996

Test specification Who students in 10 -week pre-sessional classes at or near CEFR B 2 Why to measure achievement of target performance in subject-specific context What construct/knowledge/skills How 2, 000 word literature review exploring a subject-specific research question Bachman & Palmer, 1996

Test specification – construct What is a literature review? 1. Audience/Purpose/Structure 2. Level of target discipline knowledge 3. Level of critical engagement with sources Bachman & Palmer, 1996

Literature review construct 1) Involves more than linguistic knowledge and skills 2) Requires students to engage with texts in the disciplines & struggle to understand them 3) Requires evaluation and management of information 4) Raising awareness of graduate attributes 5) Target performance described at SQA level 10: exit level of UG and entry level of PG degrees

Graduate attributes • a sense of ‘research-mindedness’ enabling a wider more analytical perspective on individual practice • the ability to identify problems, formulate research questions & interpret complex data to seek answers • the ability to derive meaning from complexity and make informed judgments on the basis of evidence • an openness to learning and positive orientation to new opportunities, ideas and ways of thinking • a tolerance for ambiguity and unfamiliarity.

SCQF level 10 knowledge A critical understanding of the principal theories, concepts and principles in a subject/discipline. Detailed knowledge and understanding in one or more specialisms some of which is informed by or at the forefront of a subject/discipline. Knowledge and understanding of the ways in which the subject/discipline is developed, including a range of established techniques of enquiry or research methodologies.

SCQF level 10 application Execute a defined project of research, development or investigation and identify and implement relevant outcomes. Practise in a range of professional level contexts which include a degree of unpredictability and/or specialism.

SCQF level 10 cognitive skills Critically identify, define, conceptualise, and analyse complex/professional level problems and issues. Critically review and consolidate knowledge, skills, practices and thinking in a subject/discipline. Make judgements where data/information is limited or comes from a range of sources.

Test specification – construct What is a literature review? 1. Audience/Purpose 2. Structure overall & within sections/paragraphs 3. Level of target discipline knowledge 4. Level of critical engagement with sources Task: Choose one of the areas above and brainstorm ideas for this aspect of the construct

Audience peers, novices (non-specialists) in the community of practice lecturers (specialists) who assess research projects, dissertations, theses educated non-specialists, e. g. in funding councils, NGOs or government departments who award grants or use the findings

Purpose in-depth exploration of a specific aspect of a research area to indicate the current state of the art and suggest new research directions OR to define and limit the scope of and provide a context and framework for a piece of research. In-depth = reference to wide range of sources Context = current state of the art Framework = theoretical basis for research design

Structure Thematic comparison and classification of sources, based on clear criteria usually specified in advance. Relates key papers & ideas to each other, e. g. shows which make similar/opposite/more developed claims; which ideas stimulated by which earlier contributions Paragraphs develop from general to specific with claims supported by evidence from relevant sources. Review (& sections/paragraphs) structured from familiar to new; structure made explicit for the reader by summarising at the end of each section what has been discussed and how this links to what follows.

Subject knowledge Responsibility of student (or subject expert if joint marking opportunity). Includes recognition of key figures/papers which have moved the field on and therefore should be cited. EAP tutors do not have this knowledge Nevertheless have to be able to read a text they do not fully understand to assess whether it achieves its purpose for the stated audience through its structure and the level of critical engagement it demonstrates.

Critical engagement Not simply description/summarising (SCQF, 2012) identify, define, conceptualise, & analyse complex/ professional level problems & issues. (Bruce, 2014) evaluative judgement made within any field of human activity about some aspect, object or behaviour of that field. (Dodd, 2014) the different usefulness of knowledge…, appreciating how knowledge in its various forms holds potentially different value for different people in different places at different times.

Critical engagement Not simply description/summarising (Argent, 2013) showing awareness of different perspectives/stances (Argent, 2014) arguing cogently why some work makes a more relevant, useful or powerful contribution to the field generally or in the context of specific research. (Spencer & Alexander, 2014) relates not to finding fault (criticising) but to criteria; evaluation must be in terms of /based on specific criteria, stated explicitly.

Test specification How 2, 000 word literature review with subject-specific research question Submitted to Turnitin Assessed using customised holistic assessment descriptors Bachman & Palmer, 1996

Assessment descriptors Analytic (e. g. IELTS) or holistic (e. g. TOEFL) Holistic – quicker/easier to give an impression mark A response at this level largely achieves all of the following A response at this level is marked by one or more of the following A response at this level may reveal one or more of the following weaknesses

Assessment descriptors Task: develop a set of statements which capture the aspects outlined in the construct A response at this level largely achieves all of the following 1. Content & task achievement (audience/purpose) 2. Structure overall & use of sources & stance 3. Paragraph structure & use of language

Assessment descriptors Task: develop sets of statements which capture varying levels of achievement A response at this level largely achieves all of the following A response at this level is marked by one or more of the following A response at this level may reveal one or more of the following weaknesses

Assessment descriptors Example: task achievement and content Fully addresses the specific area (concept, problem, method) to be explored, showing achievements/gaps. Addresses the specific area in sufficient depth to cover the main points, showing some achievements/gaps The main points are covered but with some redundant ideas. Achievements/gaps are not shown clearly. Not all aspects of the area are covered and achievements/gaps in research are not shown.

Discussion Any questions & comments?