Analyzing Standardized Tests OLC Analyzing Standardized Tests OLC

Analyzing Standardized Tests

OLC: Analyzing Standardized Tests OLC: Analyzing Presentation Standardized Tests Application Activity Supplemental Resource: Test Prep Strategies & Guidance Supplemental Resource: Responding to Test Anxiety & Guidance Supplemental Resource: Designing Selected & Constructed Response Tests

To start, reflect on the following questions: 1. What are some standardized exams that were important in your life? How did you prepare for those exams? 2. What standardized tests do you know about in your country? What are the consequences of succeeding or failing the test? 3. Can you describe the main content and the general structure of one of the exams? 4. Do you know if anyone at your school helps students developed test -taking strategies? If so, what techniques to they use to prepare students?

Here are some exam types and how they are defined • A high-stakes exam is one where doing well can lead to advance and success, whereas doing poorly can result in being cut off from (usually academic) opportunities. • This exam also has a close synonym in the term gate-keeping exam in that this kind of exam is like a “gate” that allows successful students to pass through but for others the gate is shut. • School-leaving exams are a special type of exam (like the French baccalaureate) that one takes at the end of one phase of school (and which will allow successful students to enter the next phase of school). • Entrance exams are similar to school-leaving exams except that they are usually administered by the school that students hope to enter next. • Standardized exams are ones that are (a) the same for everyone, (b) extremely formal in structure, (c) usually based on some type of research model, and (d) are revised regularly so that test takers cannot simply memorize the test. In principle, standardized tests are not always high stakes exams, however because designing and regularly revising an exam tends to become expensive, such exams tend to become important, and in practice become gate-keeping exams.

Why do Ed Volunteers need to analyze Standardized Gate-Keeping Exams? a) b) c) d) To anticipate ways students might cheat To better prepare students to pass the exams To be able to grade those exams quickly To replace current exams with better exams

The correct answer is B because- • Your country has high-stakes gate-keeping exams that students must pass to advance in school • Often the content & structure of such exams is arbitrary and not reflective of the school curriculum • If teachers don’t spend some class time preparing students for the test they won’t be contributing to an explicit measure of student success and will not therefore be seen as a good teacher.

What is a supplemental benefit of analyzing standardized tests? a) It’s a skill you can put on your resume. b) Charging for test tutoring is a way to make ends meet at the end of the month c) Showing students how to analyze their gatekeeper exams is a high-incentive way to teach them critical thinking skills d) You can complain about the test as much as you want because you have proof of how bad the test is.

The supplemental benefit is C. • Yes! It’s a very motivating way to teach students critical thinking skills: – They analyze the test to learn to think like the test, they learn strategies to determine the most likely best answer—earn higher scores—and learn techniques of logical analysis.

What are 2 characteristics of wellprepared test-takers? Well-prepared test-takers…. ____ the test structure thoroughly and know how to ____ like the test.

Well-prepared test-takers know/understand the test structure thoroughly and know how to think like the test. • Before you can help your students succeed on standardized tests you must analyze the test structure to know “how the test (or testdesigners) think(s), ” so you can then teach students the strategies to help determine correct answers & provide enough practice with test items until they learn how the test “thinks. ” Example of thinking like the test: http: //magoosh. com/gre/2013/thinking-like-the-test-makers/

Describing the Test • Look to see if there is an introductory manual for the test. If there is no manual, consult the test’s preface. • Also, if you cannot find a manual or preface for your test, you might be able to get important information verbally by asking other teachers or former test-takers. Teachers often know the test design well and have examples of previous versions of the exam. For example, see the: (a) WAEC English Language Test and/or (b) TOEIC Listening and Reading Examinee Handbook

Some Testing Terms • Norm-Referenced The test-takers results are compared to the results of others to see where their score is in relation to other test-takers. • Criteria-Referenced Test-takers are evaluated in terms of how successful they are in demonstrating mastery of predefined criteria. – (e. g. , count to 100 in English, write a story correctly using the English past tenses, start a conversion in English & continue it for at least a minute)

Are These Tests Norm or Criteria Referenced? a) The written portion of a driver’s license test: N or C? b) The SAT (US test for college readiness): c) The teacher grades a test “on the curve”: N or C? d) Invitees must score 25 out of 30 to pass the safety & security test prior to staging: N or C?

Some Common English Language Challenges on Standardized Tests The English language used is: a) academic in nature b) much higher proficiency than that of the students c) related to life in another country (e. g. , US or UK) not in the students’ country d) A different register a) (style/formality—Open the door vs. Would you be so kind as to open the door please? ) used for correct answers in different test sections? e) (not about English, but more generally) Have the students had little exposure to standardized tests? For each item suggest a possible consequence for students.

Possible answers for the previous slide a) b) c) d) e) Academic in nature – proficiency in academic language usually takes much longer to develop than language for everyday communication. If students don’t have the requisite academic language now, study sessions, even a month prior won’t help much Much higher proficiency than that of the students – Most students will not pass the test Related to life in London and not where the students are from – It will be culturally remote. If so, the students won’t have the requisite knowledge and give wrong answers A different register Students may choose a semantically correct answer that is not appropriate to the context of the test item Not about English, but more generally, the students have had little exposure to standardized tests – If so, they may do poorly simply because little experience with this kind of activity

Validity: Does the Test What It Claims to Test? • Content Validity: Does the test measure what it claims to measure? • Face Validity: does the test look like the kind of test it claims to be? • Predictive Validity: does the test (consistently) predict future performance? e. g. , the GRE is an effective predictor of a student’s future GPA. • Construct Validity: the structure of the test, content, and item types conform to theoretical constructs currently recognized as legitimate and valid within relevant fields of study.

Label the following examples as either: Content V Predictive V Construct V Face V 1. A paper & pencil test assesses professionally agreed- upon job skills which are those most commonly assessed in employee performance evaluations. Construct—key terms, “professionally agreed upon” and “most commonly assessed” 2. A final exam is careful to cover the content of the course and assesses the content in the way it was taught Content—that exam tests what and how the students were taught 3. The ACT looks a lot like the SAT, it looks good to me. Face-it meets my expectations for this kind of test 4. Students who scored more than 90% on the school’s Bio 101 final exam regularly earn B+s or better on college biology courses. Predictive—if I do well on this test I will likely do well in college bio classes

Two Strategies for Addressing the Distance between the Standardized Test & Your Course Content • Sometimes the distance between the course curriculum, & the test content can be great. This is a problem because most Volunteers and their counterparts are expected to cover the curriculum but they must also prepare students for the standardized exam to be credible as good teachers. Please see if you can think of any ways to help bridge this distance, then click the “button” for suggestions. • In terms of how you carry out the actually process of preparing students for the exam, there are probably only two genuine strategies for this: 1. Find ways to incorporate test content into existing classroom instruction. 2. Schedule independent, stand-alone test preparation sessions at times outside of class.

Cultural Bias • Cultural bias occurs in standardized tests when the testdesigners (often probably unintentionally) use situations, values, beliefs or practices which do not correspond to those in the students’ home culture. • The following examples show issues of cultural bias that can occur in tests. Then two examples of culturally biased test items are provided. You are ask to try to suggest what the bias is and the ways in which these items could be problematic for students from other cultural backgrounds, as well as suggesting some strategies for helping students work around cultural biases that could appear in standardized exams in their countries.

2 Items w/Possible Cultural Bias • The Cowboys’ offense starts at the 7 -yard line and moves 32 yards. What yard line are the Cowboys on? (Answer: 39 -yard line) • Tatu has been watching too much television because she’s melancholy. This means she’s a. curious b. bored c. sad d. interested

How to Address Cultural Bias in Gate-Keeping Exams • Discuss with students how some test items will be unknown or confusing—let them struggle to “work around” the biased content to find the answer. • Give students much practice on the types of items on the test so at least they will be familiar with the item format • Introduce explicit examples of culturally-remote questions from earlier test versions (w/the hope that new test versions will use similar contexts)

Now Analyze a Standardized Test of Your Choice