Automated Scoring for Speaking Assessments Arizona English Language

Automated Scoring for Speaking Assessments Arizona English Language Learner Assessment Irene Hunting - Arizona Department of Education Yuan D’Antilio - Pearson Erica Baltierra - Pearson June 24, 2015

Arizona English Language Learner Assessment AZELLA 1 • AZELLA is Arizona’s own English Language Proficient Assessment. • AZELLA has been in use since school year 2006 -2007. • Arizona revised its English Language Proficiency (ELP) Standards due to the adoption of the Arizona College and Career Ready Standards in 2010. • AZELLA had to be revised to align with the new ELP Standards. • Arizona not only revised the alignment of AZELLA but also revised administration practices and procedures. • Revisions to the Speaking portion of the AZELLA are particularly notable.

AZELLA Speaking Test Administration Prior to School Year 2012 -2013 • Administered orally by test administrator • One-on-one administration • Scored by test administrator • Immediate scores • Training for test administrators 2 • Minimal • Not required

AZELLA Speaking Test Concerns Prior to School Year 2012 -2013 • Inconsistent test administration • Not able to standardize test delivery • Inconsistent scoring • 3 Not able to replicate or verify scoring

AZELLA Speaking Test Desires For School Year 2012 -2013 and beyond • Consistent test administration • Every student has the same testing experience • Consistent and quick scoring • Record student responses • Reliability statistics for scoring • Minimal burden for schools 4 • No special equipment • No special personnel requirements or trainings • Similar amount of time to administer

AZELLA Speaking Test Administration For School Year 2012 -2013 and beyond • Consistent test administration • Administered one-on-one via speaker telephone • Consistent and quick scoring • Student responses are recorded • Reliable machine scoring • Minimal burden for schools 5 • Requires a landline speaker telephone • No special personnel requirements or training • Slightly longer test administration time

Proposed Solution 6

Development of Automated Scoring Method Human raters Field testing data Testing System Human Transcribers Item Text Test Developers Recorded Items Test Spec Automated Scores Validation

Why does automated scoring of speaking work? 8 • The acoustic models used for speech recognition are optimized for various accents • Young children speech, foreign accents • The test questions have been modeled from field test data • The system anticipates the various ways that students respond

Field Tested Items The test questions have been modeled from field test data – the system anticipates the various ways that students respond e. g. “What is in the picture? ” 9

Language models a It’s I protractor don’t know protractor a 10 compass

Field Testing and Data Preparation Two Field Testing: 2011 -2012 Number of students: 31, 685 (1 st -12 th grade), 13, 141 (Kindergarten) Stage 11 Total tests I 13, 184 Used for building models 1, 200 Used for validation II 10, 646 1, 200 300 III 9, 369 1, 200 300 IV 6, 439 1, 200 300 V 5, 231 1, 200 333

Item Type for Automated Scoring Item Type 12 Score Point Domain Syllabification 0 -1 Oral Reading Wordlist 0 -1 Oral Reading Repeat 0 -6 Speaking Questions about an image 0 -4 Speaking Similarities and differences 0 -4 Speaking Give directions from a map 0 -4 Speaking Questions about a statement 0 -4 Speaking Give instructions to do something 0 -4 Speaking Open questions about a topic 0 -4 Speaking Detailed responses to a topic 0 -4 Speaking

Sample Speaking Rubric: 0 – 4 Point Item Points 4 13 Descriptors Student formulates a response in correct understandable English using two or more sentences based on given stimuli. Student responds in complete declarative or interrogative sentences. Grammar errors are not evident and do not impede communication. Student responds with clear and correct pronunciation. Student responds using correct syntax. 3 Student formulates a response in understandable English using two or more sentences based on a given stimuli. Student responds in complete declarative or interrogative sentences. Sentences have minor grammatical errors. Student responds with clear and correct pronunciation. 2 Student formulates an intelligible English response based on given stimuli. Student does not respond in two complete declarative or interrogative sentences. Student responds with errors in grammar. Student attempts to respond with clear and correct pronunciation. 1 Student formulates erroneous responses based on given stimuli. Student does not respond in complete declarative or interrogative sentences. Student responds with significant errors in grammar. Student does not respond with clear and correct pronunciation.

Sample student responses Item Next, please answer in complete sentences. Tell how to get ready for school in the morning. Include at least two steps. 14 Response Transcript first you wake up and then you put on your clothes # and eat breakfast Human Machine Score 3 3. 35

Validity evidence: Are machine scores comparable to human scores? Measures we looked at: • Reliability (internal consistency) • Candidate-level (or test-level) correlations • Item-level correlations 15

Structural reliability Stage 16 Human Cronbach α Machine Cronbach α I 0. 98 0. 99 III 0. 96 0. 94 IV 0. 95 0. 94 V 0. 96 0. 95 Average 0. 97 0. 96

Scatterplot by Stage II Stage IV 17 Stage III Stage V

Item-level performance: by item type Item Type (Stage II) Human-human correlation Machine-human correlation Questions about an image 0. 87 0. 86 Give directions from a map 0. 82 0. 84 Open questions about a topic 0. 75 0. 72 Give instructions to do something 0. 83 0. 80 Repeat 0. 95 0. 85 18

Item-level performance: by item type Item Type (Stage IV) Machine-Human correlation Questions about an image Human correlation 0. 84 Give directions from a map 0. 84 0. 90 Open questions about a topic 0. 82 Detailed response to a topic 0. 85 0. 87 Give instructions to do something 0. 87 0. 90 Repeat 0. 96 0. 89 19 0. 84

Summary of Score Comparability Machine-generated scores are comparable to human ratings • Reliability (internal consistency) • Test-level correlations • Item-type-level correlations 20

Test Administration Preparation • 21 • One-on-one practice – student and test administrator • Demonstration Video • Landline Speaker Telephone for one-on-one administration • Student Answer Document – Unique Speaking Test Code

Test Administration 22

Test Administration • Warm Up Questions • What is your first and last name? • What is your teacher’s name? • How old are you? • • • 23 Purpose of the Warm Up Questions Student becomes more familiar with prompting Sound check for student voice level, equipment Capture Demographic data to resolve future inquiries Responses are not scored

Challenges Challenge Solution Landline Speaker telephone availability • ADE purchased speaker telephones for the first year of administration Difficulty scoring young population • • Additional warm up questions Added beeps to prompt the student to respond Adjusting acceptable audio threshold Rubric Update and Scoring Engine Recalibration • • Incorrect Speaking Codes 24 • • • Captured demographics from warm up questions Speaking code key entry process updated Documentation of test administrator name and time of administration

Summary Automated delivery and scoring of speaking assessment is highly reliable solution for large-volume state assessments 25 • Standardize test delivery • Minimal test set-up and training is required • Consistent in scoring • Availability of test data for analysis and review

Questions 26