Exhibition on Testing and Measurement Measurement Experts Reactions

Exhibition on Testing and Measurement: Measurement Experts’ Reactions Thanos Patelis The College Board Symposium at the Annual Meeting of NCME Denver, CO Saturday, May 1, 2010 1

Questions we were asked to address… 1. Why is there a need for such an exhibition? 2. What are three main messages it conveys? 3. In your experience, which measurement topic is most difficult to explain to the public? 4. How can we attract the public to the exhibition? Congratulations to Avi, Diana, their institutions and the partners involved for making this a reality! 2 2

Why is there a need for such an exhibition? • Because there continues to be suspicion, misunderstanding, and myths about measurement and testing – particularly about their utility. “Tests are and will never be fair!” --8 th Grade Teacher 3 3

Why is there a need for such an exhibition? • Some myths about testing continue to persist in the public and the media ü Tests are biased. ü Tests are not useful. ü Tests are arbitrary. ü Multiple-choice questions measure only trivial pieces of information. ü Tests are easy to build. ü No one monitors the quality. ü Why do we need tests when we have data? 4 4

Why is there a need for such an exhibition? • Because the general public really does not know EVERYTHING about testing and measurement. • • • 5 The historical need for testing has a risen out of an effort to be fair and objective. Many consumers misunderstand fundamental terms: Ø Percentiles Ø Standard error or measurement Ø Reliability Ø Statistical vs. practical significance Ø Sampling (content and examinees) Ø Validity The Standards of Educational and Psychological Testing; the Code of Fair Testing Practices in Education; the Buros Institute and the Mental Measurements Yearbook. 5

Why is there a need for such an exhibition? • No one knows about the field of testing and measurement until they stumble onto it. Ø As we know there is an exciting, fulfilling career in all aspects of testing and measurement. Ø As each of us has indicated when we reflect on how we got here, we fell into this field by accident. Ø As we have heard from studies in both education and psychology, we have more jobs that the pipeline can fill. (Brennan & Plake, 1991; Patelis, Kolen & Parshall, 1997; Sireci & Khaliq, 2002; Aiken et al. 2008) 6 6

What are three main messages it conveys? 1. Testing is more than multiple-choice questions and essays. • • • 7 7 Simulators (like those shown in the exhibit) are tests that can authentically represent the symptoms of a patient to both teach and assessment knowledge and skill. First, people may not realize this is an instrument. Second, measurement issues still apply.

What are three main messages it conveys? 2. There’s a rigorous, scientific process in building tests. • The series of activities demonstrate that each aspect of the assessment process requires rigorous methodology. • The participants will come out of this experience with an appreciation of the process involved. 8 8

What are three main messages it conveys? Tests are the means for gathering information. 3. • The exhibit demonstrates that tests are used to gather information for real applications. • The exhibit demonstrates that tests are not artificial components in many areas, but inherent in the information gathering process. 9 9

In your experience, which measurement topic is most difficult to explain to the public? Why? 1. • People do not understand the value that tests provide in the specific context in which their being used. 10 10

In your experience, which measurement topic is most difficult to explain to the public? 2. What do they mean? 3. How can I use tests? • People do not understand what scores mean and then how to use them (as well as when NOT to use them). • Test publishers and the measurement community (we) are to blame. • Building informed consumers hopefully permits them to seek out the answers to these questions. 11 11

How can we attract the public to the exhibition? Interactive, experiential • Continue to add booths that offer participants a chance to interact with various aspects of the testing process. Ø For example, scoring a third grader’s essay and comparing their score to what was obtained. § Add short videos of a performance assessment. (e. g. , ABC’s of School Testing) Introduce games involving teams to see if their team can get high levels of interrater reliability by staging a set of short training experiences. Stage a game show with poor questions vs. high quality questions. Ø Ø Ø Personalized • Licensing automated scoring applications and having participants write an essay and getting feedback including a score. To extent possible (minimizing risk) give them feedback after taking self-administered assessments. Viewing an aggregate report about their state’s or nation’s test scores. Ø Ø Ø Adding a tutorial (voice or computer) to explain the meaning of the report § Prizes • Ø Provide inexpensive tangible rewards for participation or (where appropriate) performance - e. g. , for driving safely in the simulator. Advertise at the right time • • Teachers in high schools Faculty in colleges Community bulletin boards on TV, radio, newspapers 12 12