TVET Australia Assessment validation and moderation A power

TVET Australia Assessment, validation and moderation A power point presentation developed by the NQC to support information sessions on assessment, validation and moderation

Contact NQC Secretariat TVET Australia Level 22/ 390 St Kilda Road Melbourne Vic 3004 Telephone: +61 3 9832 8100 Email: nqc. secretariat@tvetaustralia. com. au Web: www. nqc. tvetaustralia. com. au

Disclaimer This work has been produced on behalf of the National Quality Council with funding provided through the Australian Government Department of Education, Employment and Workplace Relations and state and territory governments. The views expressed in this work are not necessarily those of the Australian Government or state and territory governments

Acknowledgement This presentation was designed to support the interactive information sessions that formed part of the NQC’s communication and dissemination strategy: NQC products: validation and communication. Reports and materials which focus on validation and moderation may be downloaded from the NQC website at http: www. nqwc. tvetaustralia. com. au/nqc_publications This work was produced for the National Quality Council by Andrea Bateman, Quorum QA Australia Pty Ltd Chloe Dyson, Quorum QA Australia Pty Ltd

Quality of assessment

Setting the Scene § Concerns about the quality of assessments and comparability of standards across the VET sector § OECD (2008) Reviews of VET (Australia) § NQC (2008) Industry Expectations of VET § Service Skills SA (2010) VETi. S Project

Today’s workshop § § Assessment Developing Assessment Tools Competency Mapping Simulated Assessment § Engaging industry § Assessment Quality Management Framework § Validation and moderation § System considerations § Diverse settings

NQC resources • Guide for Developing Assessment Tools • Assessment Facts Sheets • Simulated Assessment • Making Assessment Decisions • Peer Assessment and Feedback • Quality Assuring Assessment Tools • Assessor Partnerships • Systematic validation • Assessor Guide: Validation and Moderation

Session 1: What is assessment? § Purposeful process of systematically gathering, interpreting, recording and communicating to stakeholders, information on student performance.

Assessment Purposes § Evaluative § designed to provide information to evaluate institutions and curriculum/standards – primary purpose is accountability § Diagnostic § Produce information about the candidate’s learning § Formative § Produce evidence concerning how and where improvements in learning and competency acquisition are required § Summative § Used to certify or recognise candidate achievement or potential

Assessment Purposes § Assessment for learning occurs when teachers use inferences about student progress to inform their teaching (formative) § Assessment as learning occurs when students reflect on and monitor their progress to inform their future learning goals (formative) § Assessment of learning occurs when teachers use evidence of student learning to make judgements on student achievement against goals and standards (summative) § http: //www. education. vic. gov. au/studentlearning/assessment/preptoyear 10/default. htm

Session 2: Developing Assessment Tools NQC Products • Guide for Developing Assessment Tools • Assessment Facts Sheets • Simulated Assessment • Making Assessment Decisions • Peer Assessment and Feedback • Quality Assuring Assessment Tools • Assessor Guide: Validation and Moderation http: //www. nqc. tvetaustralia. com. au/nqc_publications

Impact Changes to definitions within the NQC publications § AQTF 2010 User Guide documentation; and the § Training Package Development Handbook Reliability Validity Assessment tool Validation Moderation

Key Stages – developing assessment tools § identify and describe the purposes for the assessment § identify the assessment information that can be used as evidence of competence/learning § identify a range of possible methods that might be used to collect assessment information § define the contexts for interpreting assessment information in ways that are meaningful for both assessor and candidate § determine the decision making rules § define procedures for coding and recording assessment information § identify stakeholders in the assessment and define their reporting needs.

Essential Characteristics - Assessment Tool An assessment tool includes the following components: § The context and conditions for the assessment § The tasks to be administered to the candidate § An outline of the evidence to be gathered from the candidate § The evidence criteria used to judge the quality of performance (i. e. , the assessment decision making rules); as well as the § The administration, recording and reporting requirements.

Ideal Characteristics § § § § The context Competency mapping The information to be provided to the candidate The evidence to be collected from the candidate Decision making rules Range and conditions Materials/resources required Assessor intervention Reasonable adjustments Validity evidence Reliability evidence Recording requirements Reporting Requirements

Competency Mapping § The components of the Unit(s) of Competency that the tool should cover should be described. This could be as simple as a mapping exercise between the components within a task (eg each structured interview question) and components within a Unit or cluster of Units of Competency. The mapping will help determine the sufficiency of the evidence to be collected as well as the content validity. Advice regarding competency mapping can be found in the NQC Assessor Guide: Validation and Moderation

Competency Mapping

Competency Mapping: Steps in the process § Step 1: Unpack the unit of competency to identify its critical components. § Step 2: For each assessment method, list the tasks to be performed by the candidate. § Step 3: For each assessment method, map the critical components of the unit to each assessment task. Refer to NQC Assessor Guide: Validation and Moderation

Level of specificity in mapping – Risk Assessment Risk can be determined by consideration of: § Safety (eg potential danger to clients from an incorrect judgement) § Purpose and use of the outcomes (eg selection purposes) § Human capacity (eg level of expertise and experience of the assessors) § Contextual (eg changes in technology, workplace processes, legislation, licensing requirements and/or training packages)

Decision Making Rules § The rules to be used to: § Check the quality of the evidence (i. e. the rules of evidence) § Judge how well the candidate performed on the task according to the standard expected § Synthesise evidence from multiple sources to make an overall judgement Additional advice – refer to Fact Sheet 2

Reasonable Adjustments § This section of the assessment tool should describe the guidelines for making reasonable adjustments to the way in which evidence of performance is gathered without altering the expected performance standards (as outlined in the decision making rules).

Simulated assessment § For the purposes of assessment, a simulated workplace is one in which all of the required skills are performed with respect to the provision of paid services to an employer or the public can be demonstrated as though the business was actually operating. § In order to be valid and reliable, the simulation must closely resemble what occurs in a real work environment. § The simulated workplace should involve a range of activities that reflect real work experience. It should allow the performance of all of the required skills and demonstration of the required knowledge. Ref: AQTF definition (refer to Activity Handout), Assessment Fact Sheet 1

Activity 1: Engaging Industry § In your groups discuss what input employers (you might wish to specify a vocational area) could provide to develop valid assessment tools and processes. § For the following scenarios, note down 2/3 questions you could ask employers and how the responses will inform the development or review of assessment tools and/or processes. Relevant NQC support materials: • Industry Enterprise & RTO Partnership • Assessment Fact Sheets: Assessor Partnerships • Assessor Guide: Validation and Moderation

Activity 2: Self Assessment § In groups of 3, review the assessment tool using the self assessment checklist from the NQC (2009) Implementation Guide (Template A. 1, p. 45). § Identify any gaps in the tool? § Discuss the pros and cons of including such additional information within the tool?

Tool Review § Has clear, documented evidence of the procedures for collecting, synthesising, judging and recording outcomes (i. e. , to help improve the consistency of assessments across assessors [inter-rater reliability]). § Has evidence of content validity (i. e. , whether the assessment task(s) as a whole, represents the full range of knowledge and skills specified within the Unit(s) of competency. § Reflect work-based contexts, specific enterprise language and job-tasks and meets industry requirements (i. e. , face validity). § Adheres to the literacy and numeracy requirements of the Unit(s) of Competency (construct validity). § Has been designed to assess a variety of evidence over time and contexts (predictive validity). § Has been designed to minimise the influence of extraneous factors (i. e. , factors that are not related to the unit of competency) on candidate performance (construct validity).

Tool Review § Has clear decision making rules to ensure consistency of judgements across assessors (inter-rater reliability) as well as consistency of judgements within an assessor (intra-rater reliability). § Has a clear instruction on how to synthesise multiple sources of evidence to make an overall judgement of performance (inter-rater reliability). § Has evidence that the principles of fairness and flexibility have been adhered to. § Has been designed to produce sufficient, current and authentic evidence. § Is appropriate in terms of the level of difficulty of the task(s) to be performed in relation to the skills and knowledge specified within the relevant unit(s) of Competency. § Has outlined appropriate reasonable adjustments that could be made to the gathering of assessment evidence for specific individuals and/or groups. § Has adhered to the relevant organisation assessment policy.

Quality Checks § Panel § Pilot § Trial Refer to Fact Sheet 4, Quality assuring assessment tools

Session 3: Assessment Quality Management NQC Products • Code of Professional Practice: Validation & Moderation • Implementation Guide: Validation and Moderation • Assessment Facts Sheets • Quality Assuring Assessment Tools • Systematic Validation • Assessor Partnerships • Assessor Guide: Validation and Moderation http: //www. nqc. tvetaustralia. com. au/nqc_publications

Validation § Validation is a quality review process. It involves checking that the assessment tool produced valid, reliable, sufficient, current and authentic evidence to enable reasonable judgements to be made as to whether the requirements of the relevant aspects of the Training Package or accredited course had been met. It includes reviewing and making recommendations for future improvements to the assessment tool, process and/or outcomes. NQC Implementation Guide: Validation and Moderation 2009

Outcomes of validation Recommendations for future improvements § § Context and conditions for the assessment Task/s to be administered to the candidates Administration instructions Criteria used for judging the quality of performance (e. g. the decision making rules, evidence requirements etc) § Guidelines for making reasonable adjustments to the way in which the evidence of performance was gathered to ensure that the expected standard of performance specified within the Unit(s) of Competency has not been altered § Recording and reporting requirements.

Moderation § Moderation is the process of bringing assessment judgements and standards into alignment. It is a process that ensures the same standards are applied to all assessment results within the same Unit(s) of Competency. It is an active process in the sense that adjustments to assessor judgements are made to overcome differences in the difficulty of the tool and/or the severity of judgements. NQC Implementation Guide: Validation and Moderation 2009

Outcomes of moderation § Recommendations for future improvement and adjustments to assessor judgements (if required) and § Recommendations for improvement to the assessment tools § Adjusting the results of a specific cohort of candidates prior to the finalisation of results and § Requesting copies of final candidate assessment results in accordance with recommended actions.

Validation vs Moderation Features Validation Moderation Assessment Quality Management Type Quality Review Quality Control Primary Purpose Continuous improvement Bring judgements and standards into alignment. Timing On-going Prior to the finalisation of candidate results Focus Assessment tools; and Candidate Evidence (including assessor judgements) (desirable only) Assessment tools, and; Candidate Evidence, including assessor judgements (mandatory) Type of Approaches Assessor Partnerships Consensus Meetings External (validators or panels) Consensus Meetings External (moderators or panels) Statistical Outcomes Recommendations for future improvements Recommendations for improvements; and Adjustments to assessor judgements (if required)

Types of Approaches - Statistical § Limited to moderation § Yet to be pursued at the national level in VET § Requires some form of common assessment task at the national level § Adjusts level and spread of RTO based assessments to match the level and spread of the same candidates scores on a common assessment task § Maintains RTO-based rank ordering but brings the distribution of scores across groups of candidates into alignment Strength § Strongest form of quality control Weakness § Lacks face validity, may have limited content validity

Types of Approaches - External § Types § Site Visit Versus § Central Agency Strengths § Offer authoritative interpretations of standards § Improve consistency of standards across locations by identifying local bias and/or misconceptions (if any) § Educative Weakness § Expensive § Less control than statistical

Types of Approaches – Assessor Partnerships § Validation only § Informal, self-managed, collegial § Small group of assessors § May involve: § Sharing, discussing and/or reviewing one another’s tools and/or judgements § Benefit § Low costs, personally empowering, non-threatening § May be easily organised § Weakness § Potential to reinforce misconceptions and mistakes Ref: Implementation Guide, Assessment Fact Sheet 5

Types of Approaches - Consensus § Typically involves reviewing their own & colleagues assessment tools and judgements as a group § Can occur within and/or across organisations § Strength § Professional development, networking, promotes collegiality and sharing § Weakness § Less quality control than external and statistical approaches as they can also be influenced by local values and expectations § Requires a culture of sharing

Systematic Validation (consensus) Indicators Yes/No Action Is there a plan for assessment validation (including validation of RPL assessment) in place? Does your plan: • Determine the sample of units of competency to be validated over a set period of time • Provide dates for proposed validation activities • Include details about who will participate in assessment validation, including the Chair of consensus panels, if relevant • Include a strategy to ensure that all relevant staff are involved • Identify what processes and materials will be used for implementing and recording the outcomes of assessment validation Does your RTO have terms of reference in place to guide the work of consensus panels? Does your RTO have validation materials (policy, procedure, forms) in place that cause participants to engage effectively in validation? Does your RTO have a process for monitoring the action taken as a result of validation? Does your RTO have a process and plan in place for reviewing the effectiveness of assessment validation? Ref: Assessor Guide

System considerations § What is the most appropriate approach to validation? Condition Suggested approach Whenever my RTO conducts internal validation few opportunities for improvement arise Consider including external representation on your validation panel Our assessors are contractors and cannot come to validation consensus meetings because my RTO can’t afford to pay for their time and some are located interstate Consider establishing assessor validation partnerships at your local level, but ensure that improvements identified are recorded and fed back to other assessors and formalised Our RTO conducts high risk units related to licensing, where the licensing authority has mandated the use of assessment tools it provides Consider consensus moderation, ideally with external representation on your panel. Our RTO is new and assessors do not have a lot of experience Consider inviting an external person with expertise in assessment tool design to validation consensus meetings

Assessment Quality Management

Quality management in diverse settings § Identified barriers: § Structural (i. e. , the organizational and resource aspects) – financial, variations of definitions across key documents § Process (i. e. , the practices and activities that take place) – rolling enrolments, partnering arrangements, workloads § Personal factors (i. e. , the attitudinal, assessment literacy and expectations of the key players). § Strategies deployed by RTOs Refer to Handout – Quality management processes in diverse settings.

Activity 3: Assessment Quality Management Approach Strategies implemented § Activity 4: Assessment Quality Man Quality assurance Quality control Quality management Ref: Implementation Guide

Andrea Bateman Director Education Consultant Quorum QA Australia Pty Ltd Email: andrea@batemangiles. com. au Phone: 0418 585 754 Chloe Dyson Director Education Consultant Quorum QA Australia Pty Ltd Email: chloed@alphalink. com. au Phone: 0408124825 Principal author Associate Professor Shelley Gillis Deputy Director Work-based Education Research Centre Victoria University Email: shelley. gillis@vu. edu. au Phone: 0432 756 638