Why standardized tests Richard P Phelps University of

Why standardized tests? Richard P. Phelps University of Bucharest 4 June 2018 4 June, 2018 Why Standardized Tests? – U. Bucharest 1

“If a thing exists, it exists in some amount. If it exists in some amount, then it is capable of being measured. ” −−René Descartes, Principles of Philosophy, 1664 4 June, 2018 Why Standardized Tests? – U. Bucharest 2

Learning Curve 4 June, 2018 Why Standardized Tests? – U. Bucharest 3

Forgetting Curve (1870 s) 4 June, 2018 Why Standardized Tests? – U. Bucharest 4

4 June, 2018 Why Standardized Tests? – U. Bucharest 5

Ebbinghaus: “Learning usually requires rehearsal or repetition” 4 June, 2018 Why Standardized Tests? – U. Bucharest 6

Working Memory Capacity George Miller, 1950 s Cognitive Load Theory John Sweller, 1980 s 4 June, 2018 Why Standardized Tests? – U. Bucharest 7

Working Memory: Ability to temorarily hold and manipulate information for cognitive tasks Working Memory is challenged by: new, unfamiliar information and quantity of discrete bits of information 4 June, 2018 Why Standardized Tests? – U. Bucharest 8

Two centuries of research on learning concludes… “…repeated retrieval during learning is the key to long-term retention. ” — Henry L. “Roddy” Roediger 4 June, 2018 Why Standardized Tests? – U. Bucharest 9

10 benefits of testing and their applications to education Roediger, Putnam and Smith Benefit 1: The Testing Effect: Retrieval Aids Later Retention Benefit 2: Testing Identifies Gaps in Knowledge Benefit 3: Testing Causes Students to Learn More from the Next Study Episode Benefit 4: Testing Produces Better Organization of Knowledge Benefit 5: Testing Improves Transfer of Knowledge to New Contexts Benefit 6: Testing can Facilitate Retrieval of Material That was not Tested Benefit 7: Testing Improves Metacognitive Monitoring Benefit 8: Testing Prevents Interference from Prior Material when Learning New Material Benefit 9: Testing Provides Feedback to Instructors Benefit 10: Frequent Testing Encourages Students to Study SOURCE: Roediger, Putnam, & Smith, Ten benefits of testing and their applications to educational practice, Psychology of Learning and Motivation, 55, 2011. 4 June, 2018 Why Standardized Tests? – U. Bucharest 10

10 benefits of testing and their applications to education Roediger, Putnam and Smith Indirect effects of testing Students tested frequently study more and with more regularity. Tests permit students to discover gaps in their knowledge and adjust their study efforts to focus on difficult material. Students who study after taking a test learn more than if they had not taken a test. Students who self-test or are tested more frequently in class learn more. SOURCE: Roediger, Putnam, & Smith, Ten benefits of testing and their applications to educational practice, Psychology of Learning and Motivation, 55, 2011. 4 June, 2018 Why Standardized Tests? – U. Bucharest 11

John Hattie’s meta-analyses of meta-analyses 4 June, 2018 Why Standardized Tests? – U. Bucharest 12

John Hattie’s list of education interventions, in order of effectiveness ( those with testing ) 1. 11. Student self-assessment/self-grading Response to intervention Teacher credibility Providing formative assessments Classroom discussion Teacher clarity Feedback Reciprocal teaching Teacher-student relationships fostered Spaced vs. mass practice Acceleration Classroom behavioral techniques Vocabulary programs Repeated reading programs Creativity programs Student prior achievement Self-questioning by students Study skills Problem-solving teaching Not labeling students 4 June, 2018 21. 31. Concept mapping Cooperative vs individualistic learning Direct instruction Tactile stimulation programs Mastery learning Worked examples Visual-perception programs Peer tutoring Cooperative vs competitive learning Phonics instruction Student-centered teaching Classroom cohesion Pre-term birth weight Peer influences Classroom management techniques Outdoor-adventure programs Home environment Socio-economic status Why Standardized Tests? – U. Bucharest 13

The effect of testing on student learning • 12 -year study, read >3, 000 documents • analyzed close to 700 separate studies, and more than 1, 600 separate effects • 2, 000 other studies were reviewed and found incomplete or inappropriate • hundreds of other studies remain to be reviewed 4 June, 2018 Why Standardized Tests? – U. Bucharest 14

The effect of testing on student learning 245 Qualitative studies 813 Surveys or Polls 640 Quantitative Studies: Experiments: School- and classroom-level Multivariate studies: Large-scale testing programs 4 June, 2018 Why Standardized Tests? – U. Bucharest 15

Meta-analysis A method for summarizing a large research literature, with a single, comparable measure. ( 0. 5 effect size ≈ 1 grade level of learning ) 4 June, 2018 Why Standardized Tests? – U. Bucharest 16

Findings from Phelps (2012): • Survey study effect sizes average >1. 0 • Over 90% of qualitative studies positive • For quantitative studies, effect sizes vary between 0. 55 and 0. 88: + testing more frequently + testing with stakes + testing with feedback 4 June, 2018 Why Standardized Tests? – U. Bucharest 17

Cognitive Scientists’ 6 Strategies for Effective Learning Retrieval Practice Spaced Practice Dual Coding Interleaving Concrete Examples Elaboration 4 June, 2018 Why Standardized Tests? – U. Bucharest 18

Retrieval Practice 4 June, 2018 Why Standardized Tests? – U. Bucharest 19

10 benefits of testing and their applications to education Roediger, Putnam and Smith Most teachers should be testing much more frequently, …with smaller, shorter tests. Students learn more when they test. But learn best when the tests are “spaced”. What is the optimal lapse of time between tests? 4 June, 2018 Why Standardized Tests? – U. Bucharest 20

4 June, 2018 Why Standardized Tests? – U. Bucharest 21

4 June, 2018 Why Standardized Tests? – U. Bucharest 22

Implications for Teachers 1 Most teachers should test more frequently, …with smaller, shorter, low-stakes tests Understand that useful assessment can be short and simple. 4 June, 2018 Why Standardized Tests? – U. Bucharest 23

Implications for Teachers 2 Does the test format matter? • multiple-choice? • essay? • short answer? • oral? • demonstration? • …etc. ? Not so much. 4 June, 2018 Why Standardized Tests? – U. Bucharest 24

Implications for Teachers 3 Tests provide feedback to teachers about what works and what does not Just like students can learn by testing each other; teachers can help each other by reviewing each others’ tests. 4 June, 2018 Why Standardized Tests? – U. Bucharest 25

Why Standardized tests? In some places, the only objective measure available to the public (i. e. , not under the control of insiders). 4 June, 2018 Why Standardized Tests? – U. Bucharest 26

Studies of the reliability of teacher grading, 1890 s to 1920 s e. g. , Starch & Elliot, 1912 • Two actual English examination papers • Sent to 142 teachers to grade • Grades ranged from 50 to 98% • One paper: 14 grades < 80% & 14 > 94% 4 June, 2018 Why Standardized Tests? – U. Bucharest 27

Studies of the reliability of teacher grading, 1890 s to 1920 s Starch & Elliot, 1912 • Two actual Geometry examination papers • Sent to 116 teachers to grade • Grades ranged from 28 to 92% • One paper: 20 grades < 60% & 9 > 85% 4 June, 2018 Why Standardized Tests? – U. Bucharest 28

How can those outside a school or classroom judge the quality of a school, its instruction, or its students? Schools vary in quality Courses vary in quality Grade comparisons are not reliable Standardized tests’ most important feature is standardization. 4 June, 2018 Why Standardized Tests? – U. Bucharest 29

Why high-stakes tests? Most respond to both intrinsic and extrinsic motivators and the proportion varies from individual to individual. High-stakes tests provide both forms of inducement. High-stakes tests tend to be taken more seriously and administered with tighter security. 4 June, 2018 Why Standardized Tests? – U. Bucharest 30

Findings from Phelps & Silva Meta-Regression (2018) To raise achievement: – Test more often + – Test more often with more feedback ++ (formative testing) – Test more often with more stakes ++ (summative testing) – Test more often with feedback AND stakes? – NO (they are less effective together than separate) Formative & summative differ; serve different purposes. 4 June, 2018 Why Standardized Tests? – U. Bucharest 31

Cognitive Psychology experiments were conducted with “formative” tests in schools and classrooms 4 June, 2018 Why Standardized Tests? – U. Bucharest 32

The more systemwide decision points, the better ? SOURCE: Phelps, Benchmarking to the best in mathematics, Evaluation Review, 2001 4 June, 2018 Why Standardized Tests? – U. Bucharest 33

Quality control has proportionally greater effect in poorer countries SOURCE: Phelps, Benchmarking to the best in mathematics, Evaluation Review, 2001 4 June, 2018 Why Standardized Tests? – U. Bucharest 34

What about systemwide, large-scale tests? First priority: do no harm to the formative testing programs in schools and classrooms 4 June, 2018 Why Standardized Tests? – U. Bucharest 35

Besides, systemwide tests are needed for other purposes, such as… …selection to programs with limited number of places …monitoring and system diagnosis …workforce planning …accountability …credentialing That’s enough! 4 June, 2018 Why Standardized Tests? – U. Bucharest 36

Some large-scale test advantages On per-student basis, inexpensive Cognitive laboratory pre-testing possible Standardization offers comparisons across schools and regions. May produce high-quality items that schools and teachers can use. MOST IMPORTANT: provides reliable, comparative information to all those not involved in a particular school 4 June, 2018 Why Standardized Tests? – U. Bucharest 37

Effect of scale on testing achievement benefits from Phelps & Silva (2018) • size of study population • small +0. 34 over large • scale of test administration • small-scale +0. 14 over large-scale • responsible level of government • local tests +0. 29 over state tests 4 June, 2018 Why Standardized Tests? – U. Bucharest 38

Large-scale test, tight security 4 June, 2018 Why Standardized Tests? – U. Bucharest 39

Large-scale test, lax security 4 June, 2018 Why Standardized Tests? – U. Bucharest 40

4 June, 2018 Why Standardized Tests? – U. Bucharest 41

“Teaching to the Test” Teachers will teach only material that will appear on a standardized test. Rebuttal If high-stakes tests are kept behind lock and key until the day of test administration, teachers will not know what material will be on the test, except in the most general terms. In the absence of common standards and tests, the curriculum becomes arbitrary and of uncertain origin. 4 June, 2018 Why Standardized Tests? – U. Bucharest 42

“Teaching to the Test” Teachers will teach only material that will appear on a standardized test. Counter-argument In the absence of common standards and tests, the curriculum becomes arbitrary and of uncertain origin. Why is that better than teaching to a required curriculum? 4 June, 2018 Why Standardized Tests? – U. Bucharest 43

“Narrowing the Curriculum” A common curriculum prescribed by standards has less content than a teacher-made curriculum. Rebuttal A school year’s fixed number of hours and days render it unlikely that a common curriculum will have less content than a teacher-arbitrary curriculum. I. e. , if a teacher drops one topic when standards are introduced, she necessarily has added another. 4 June, 2018 Why Standardized Tests? – U. Bucharest 44

“Narrowing the Curriculum” A common curriculum prescribed by standards has less content than a teacher-made curriculum. Counter-argument What teachers and schools do in the classroom without common standards is not necessarily “broader. ” In fact, it can often be “narrower”— governed in the absence of other criteria by personal preferences. 4 June, 2018 Why Standardized Tests? – U. Bucharest 45

Cheating by Students High-stakes standardized testing increases students’ incentive to cheat. Rebuttal Cheating is far easier to prevent and detect with standardized tests. Different forms used in the same classroom can make copying unrewarding. Computer programs run after the fact can look for telltale patterns. 4 June, 2018 Why Standardized Tests? – U. Bucharest 46

Cheating by Students High-stakes standardized testing increases students’ incentive to cheat. Counter-argument Cheating in regular classroom work is common. The overwhelming majority of US students admit to cheating in polls. Teachers and schools are illequipped to monitor or detect most cheating. Meanwhile, the Internet makes cheating far easier than in the past. 4 June, 2018 Why Standardized Tests? – U. Bucharest 47

Cheating by Teachers Many teachers have been caught cheating on high-stakes standardized tests. Rebuttal The fact of detection may be evidence of how easily such cheating can be detected. 4 June, 2018 Why Standardized Tests? – U. Bucharest 48

Cheating by Teachers Many teachers have been caught cheating on highstakes standardized tests. Counter-argument Social promotion and grade inflation provide evidence that nonstandardized testing and grading are far from infallible. And consider that in surveys, the majority of teachers surveyed invariably claim overwhelming pressure to award high grades to undeserving students. 4 June, 2018 Why Standardized Tests? – U. Bucharest 49

Preferred Instructional Methods Classrooms governed by standards are barren, dreary places where only factoids are learned. Rebuttal A curriculum will always rely on some sort of standard or criteria for inclusion. The question is, Do we want formal, open standards, openly arrived at, or should their origins be more obscure or idiosynchratic to each teacher? 4 June, 2018 Why Standardized Tests? – U. Bucharest 50

Preferred Instructional Methods Classrooms governed by standards and testing are barren, dreary places where only factoids are learned. Counter-argument Many teachers, especially inexperienced or quick hires in underperforming schools, will rely on the teacher’s versions of textbooks for course content or, worse, make it up. Is the classroom shorn of standardized testing wonderful (rich, innovative curriculum; the joy and magic of learning, and so on)? 4 June, 2018 Why Standardized Tests? – U. Bucharest 51

Opposition to Norm-Referenced Tests Norm-referenced standardized tests are unfair. (I. e. , it is unfair to simply rank kids, rather than measure them against standards. ) Rebuttal The alternative, grade-point-averages, are normreferenced measures, normed at the school level. 4 June, 2018 Why Standardized Tests? – U. Bucharest 52

Opposition to Norm-Referenced Tests Norm-referenced standardized tests are unfair. (I. e. , it is unfair to simply rank kids, rather than measure them against standards. ) Counter-argument The alternative, grade-point-averages, are normreferenced measures, normed at the school level. 4 June, 2018 Why Standardized Tests? – U. Bucharest 53

Preference for Teacher-Made Classroom Testing Standardized tests are imposed from outside by persons and committees unfamiliar with and perhaps insensitive to the local students and community. Rebuttal Standardized tests are developed by testing and measurement Ph. D. ’s. The most capable measurement experts in the world work in developing standardized tests. 4 June, 2018 Why Standardized Tests? – U. Bucharest 54

Preference for Teacher-Made Classroom Testing Standardized tests are imposed from outside by persons and committees unfamiliar with and perhaps insensitive to the local students and community. Counter-argument Teacher education programs provide little training in measurement, yet the absence of standardized testing would have us rely exclusively on their measurement decisions. 4 June, 2018 Why Standardized Tests? – U. Bucharest 55

4 June, 2018 Why Standardized Tests? – U. Bucharest 56

4 June, 2018 Why Standardized Tests? – U. Bucharest 57

https: //standardizedtests. procon. org/ 4 June, 2018 Why Standardized Tests? – U. Bucharest 58

Why Standardized Tests? https: //nonpartisaneducation. org richard {at} nonpartisaneducation {dot} org 4 June, 2018 Why Standardized Tests? – U. Bucharest 59