Ani Demetrashvili Head of Examinations ETS Global In
Ani Demetrashvili Head of Examinations
ETS Global • In 2017, ETS refocused its research agenda around some of the most significant, global education challenges facing societies today. Based on: • Statistical methodology; • Psychometric methodology;
Validity Evidence • Factor structure — the relationship among the test's target skills • Construct representation — how well the test tasks measure the important English‐language skills required for academic success • Criterion-related and predictive validity — whether test takers' scores reliably predict how well they will be able to use English in higher education classrooms where it is the language of instruction • Consequential validity — the effect that the test has on language learning and instruction
Research Components ETS Research Report Series, Volume: 2018, Issue: 1, Pages: 1 -12, First published: 11 September 2018, DOI: (10. 1002/ets 2. 12221)
Research Components ETS Research Report Series, Volume: 2018, Issue: 1, Pages: 1 -12, First published: 11 September 2018, DOI: (10. 1002/ets 2. 12221)
Research Challenges • Item language—differential use of idioms; • Item language—differential use of proper names; • Cultural differences (differential understanding of, and familiarity with, geocultural references); • Test materials; • Test‐taking behaviors.
Research Challenges • Levels of motivation; • Scoring of constructed‐response items; • Quantitative comparisons; • Analogies; • Sentence equivalence; • The different conceptualizations of “cheating” across cultures; • Very long explanatory essays full of colorful language versus compact essays that focus on the main points only.
Beyond & TOEFL Junior® Standard Test Textbook: • Various real‐life scenarios; • Benchmarked against CEFR and international exams ; • Easy‐to‐use structure; • Student progress monitoring tasks. TOEFL Junior ® Standard Test • Lexile ® measure to improve reading proficiency; • Well‐structured texts; • Academic reading and listening tasks; • Based on all language abilities of middle school students.
Beyond • Beyond is a 6‐level course for teenagers, taking them from A 1+ to B 2 level of the CEFR; • The combination of language skills, life skills and engaging topics prepares students for success in the 21 st century classroom and beyond. • Students Book and Workbook (Online version); • Teacher’s Book; • Premium Packs; • Presentation Software.
Skills to Tests • Linguistic progression; • Cognitive development; • Social‐emotional development; • Academic & social language skills; • General enabling skills; • Types of decisions to be made. 4 skills – includi the start on all 4 skills – including writing Focus from the start on all 4 skills – including writing
Skills to Tests • • Critical thinking; Communication; Information literacy; Flexibility; Initiative; Productivity; Decision making.
Mapping to TOEFL Junior ® Standard Test
Theory of Actions Hypothesized Actions TOEFL Test Components Test content and results • Independent measures; • Constructs based on core communication goals derived from EFL curricula and textbooks; • Test tasks focused on communication and meanings; • Linkage of scores to CEFR and Lexile levels; • Score report with meaningful feedback. • • Students Understand their strengths and weaknesses; Understand their ability according to an international standard. Parents • Understand their child’s strengths and weaknesses; • Understand their child’s ability according to an international standard; • Provide resources and support needed for their child’s English learning. Intended Effects For students • Improved attitudes, higher motivation, and greater value attached to language learning. For parents • Improvement in their child’s English ability.
Theory of Actions Hypothesized Actions TOEFL Test Components Teachers Professional Development • Professional support for teachers in the form of instructional workshops (e. g. Propell). • • • Teachers and schools Understand students’ strengths and weaknesses; Understand students; abilities according to an international standard ; Use scores to inform teaching; Identify gaps between their curricula and the test content and address gaps by modifying their curricula; Gain a better understanding of TOEFL Tests. Intended Effects For teachers and schools • Improved English instruction through better knowledge of students’ current abilities; • Increased appreciation of the value of communicative language skills.
‘. . . when analyzing a test with multiple forms administered to a heterogeneous population, the measurement invariance found in one administration may not be a valid presumption for another administration, which suggests that one sample of test takers may not be exchangeable with another in the case of heterogeneous populations that may vary across administrations, thus confining test results to specific data, and compromising the quality of the reported scores. ’ – Oliveri von Davier, Wiley Online Library
Teachers from Partner Organizations • ‘TOEFL Tests and Beyond are perfect to help students to speak, express their ideas freely and to develop English skill. ’ – Khatuna Pirtskhalava • ‘Apart from teaching basic aspects, I try to point out techniques while doing tasks in Reading or Listening skills. ’ – Ketino Mtchedlishvili • ‘The textbooks are pleasant to work with and there all useful resources to achieve our goals in mastering the language skills. ’ – Ketino Mchedlishvili
University Lecturer • TOEFL Test has had a very transformative effect for our programme. We’re now based on research. We’re now based on analysis. The TOEFL Tests set the tone for the teaching. ’ – Peter Chang, Rutgers University • My students who go overseas understand that they can take score report and it means something somewhere else, especially when it’s from ETS. ’ – Peter Chang, Rutgers University
Thank you for attention
- Slides: 18