Highlights from PIRLS and TIMSS 2011 Jack Buckley
Highlights from PIRLS and TIMSS 2011 Jack Buckley National Center for Education Statistics Washington, DC December 11, 2012
Overview v. What are PIRLS and TIMSS and which education systems participate in them? v. How are U. S. students performing compared to their international peers in: v 4 th-grade reading? v 4 th- and 8 th-grade mathematics? v 4 th- and 8 th-grade science? v. How are students performing in participating U. S. states? 2
PIRLS What are…? TIMSS • Progress in International Reading Literacy Study (PIRLS) • Trends in International Mathematics and Science Study (TIMSS) • 4 th-grade reading assessment • 4 th- and 8 th-grade mathematics and science assessment • Developed by the TIMSS & PIRLS International Study Center at Boston College, under contract to the International Association for the Evaluation of Educational Achievement (IEA) 2001 1995 2011 1999* 2003 2011 2006 2007 * In 1999, no grade 4 assessment. 3
PIRLS participating education systems (in 4 th grade) 32 26 53 educationsystems 2011 4
TIMSS participating education systems (in either/both grades) 37 74 educationsystems 57 44 47 2011 5
Participating states • 9 states participated as separate entities to obtain scores in PIRLS or TIMSS or both PIRLS TIMSS grade 4 TIMSS grade 8 Florida North Carolina Alabama California Colorado Connecticut Indiana Massachusetts Minnesota • • Each participated as part of the nation and on its own NCES funded participation at grade 8 of all states except Florida as part of a study to statistically link NAEP (National Assessment of Educational Progress) and TIMSS (Florida received other Education Department funding) 6
U. S. national sample size PIRLS TIMSS 4 th grade Schools: 370 Students: 12, 726 Schools: 369 Students: 12, 569 8 th grade Schools: 501 Students: 10, 477 7
U. S. state sample sizes (public schools only) PIRLS TIMSS 4 th grade (FL) (FL, NC) Schools: 77 Students: 2, 598 Schools: 46 – 77 Students: 1, 792 – 2, 661 8 th grade (AL, CA, CO, CT, FL, IN, MA, MN, NC) Schools: 53 – 82 Students: 1, 712 – 2, 614 8
READING
PIRLS 2011 framework PIRLS Definition The ability to understand use those written language forms required by society and/or valued by the individual. Young readers can construct meaning from a variety of texts. They read to learn, to participate in communities of readers in school and everyday life, and for enjoyment. Content Literary and informational texts Cognitive dimensions Focus on and retrieve Make straightforward inferences Interpret and integrate Examine and evaluate 10
What is on the PIRLS assessment? Percentage of 4 th-graders answering correctly: U. S. : 90% Int’l Avg. : 89% 11
What is on the PIRLS assessment? Percentage of 4 thgraders answering correctly: U. S. : 42% Int’l Avg. : 29% 12
U. S. average score (556) higher than the PIRLS scale average (500) Higher than U. S. Not measurably different than U. S. Lower than U. S. 13
U. S. average score (556) lower than in 5 education systems Higher than U. S. Not measurably different than U. S. Lower than U. S. 14
U. S. average (556) not measurably different than in 7 education systems Higher than U. S. Not measurably different than U. S. Lower than U. S. 15
U. S. average score (556) higher than in 40 education systems Higher than U. S. Not measurably different than U. S. Lower than U. S. 16
Average reading scores of 4 th-grade students increased from 2006 to 2011 in 13 education systems, including the U. S. 7* Hong Kong-CHN 9* Singapore 16* UNITED STATES 8* Denmark 18* Chinese Taipei-CHN 12* England-GBR 9* Slovenia 6* Poland 9* Norway 17* Georgia 35* Trinidad and Tobago 36* Iran, Islamic Rep. of 24* Indonesia -20 -10 0 10 20 30 40 Change in average score *p <. 05. Change in average scores is significant. NOTE: Education systems ordered according to average reading score in 2011. 17
Average reading scores of 4 th-grade students decreased from 2006 to 2011 in 8 education systems -12* Alberta-CAN -8* Sweden -10* Italy -7* Germany -12* Hungary -15* Bulgaria Austria -9* Lithuania -9* -30 -20 -10 0 10 20 30 Change in average score *p <. 05. Change in average scores is significant. NOTE: Education systems ordered according to average reading score in 2011. 18
PIRLS international reading benchmarks Grade 4 Advanced (625) Students can interpret figurative language, distinguish and interpret complex information from different parts of text, and integrate ideas across text to provide interpretations about characters’ feelings and behaviors. High (550) Students can recognize some textual features, make inferences on the basis of abstract or embedded information, and integrate information to recognize main ideas and provide explanations. Intermediate (475) Students can identify central events, make straightforward inferences from the text, and begin to make connections across parts of the text. Low (400) Students can retrieve explicitly stated details from literary and informational texts. 19
Percentages of U. S. 4 th-grade students reaching PIRLS reading benchmarks were higher than international medians in 2011 Percent 100 90 80 70 60 50 40 30 20 10 0 98 United States International median 95 86 80 56 44 17 8 Low Intermediate High Advanced Benchmark NOTE: All U. S. percentages are significantly higher than the corresponding PIRLS international median at the. 05 level of statistical significance. 20
Two systems had higher percentages of 4 th-grade students reaching Advanced than the U. S. 24 Singapore 22 Florida-USA 19 Russian Federation Northern Ireland-GBR 19 Finland 18 Higher than U. S. (p <. 05) England-GBR 18 Not measurably different than U. S. (p <. 05) Hong Kong-CHN 18 UNITED STATES 17 16 Ireland 15 Ontario-CAN 0 10 20 30 40 Percentage reaching Advanced benchmark 50 60 NOTE: Education systems with lower percentages of students reaching the Advanced benchmark than the percentage of U. S. students reaching the Advanced benchmark are not included in figure. 21
MATHEMATICS
TIMSS 2011 mathematics framework TIMSS mathematics Content dimensions Grade 4 Number Geometric Shapes and Measures Data Display Grade 8 Number Algebra Geometry Data and Chance Cognitive dimensions Knowing Applying Reasoning 23
What is on the TIMSS mathematics assessment? G R A D E Percentage of 4 thgraders answering correctly: U. S. : 80% Int’l Avg. : 54% 4 24
What is on the TIMSS mathematics assessment? G R A D E 4 Percentage of 4 thgraders answering correctly: U. S. : 76% Int’l Avg. : 65% 25
What is on the TIMSS mathematics assessment? G G R R A A D D E E Percentage of 8 thgraders answering correctly: U. S. : 87% Int’l Avg. : 70% 8 8 26
What is on the TIMSS mathematics assessment? G R A D E Percentage of 8 thgraders answering correctly: U. S. : 66% Int’l Avg. : 45% 8 27
U. S. average score (541) higher than the TIMSS scale average (500) Higher than U. S. Not measurably different than U. S. Lower than U. S. 28
U. S. average score (541) lower than in 8 education systems Higher than U. S. Not measurably different than U. S. Lower than U. S. 29
U. S. average (541) not measurably different than in 6 education systems Higher than U. S. Not measurably different than U. S. Lower than U. S. 30
U. S. average score (541) higher than in 42 education systems Higher than U. S. Not measurably different than U. S. Lower than U. S. 31
U. S. average score (509) higher than the TIMSS scale average (500) Higher than U. S. Not measurably different than U. S. Lower than U. S. 32
U. S. average score (509) lower than in 11 education systems Higher than U. S. Not measurably different than U. S. Lower than U. S. 33
U. S. average (509) not measurably different than in 12 education systems Higher than U. S. Not measurably different than U. S. Lower than U. S. 34
U. S. average score (509) higher than in 32 education systems Higher than U. S. Not measurably different than U. S. Lower than U. S. 35
Average mathematics scores of 4 thgrade students increased from 2007 to 2011 in 12 education systems, including the U. S. 15* Chinese Taipei-CHN 17* Japan 12* UNITED STATES Denmark 14* Quebec-CAN 14* 11* Slovenia 24* Czech Republic 22* Norway 24* Dubai-UAE 12* Georgia 28* Iran, Islamic Rep. of 32* Tunisia -20 -10 0 10 20 30 40 Change in average score *p <. 05. Change in average scores is significant. NOTE: Education systems ordered according to average mathematics score in 2011. 36
Average mathematics scores of 8 thgrade students increased from 2007 to 2011 in 10 education systems 16* Korea, Rep. of 18* Singapore 11* Chinese Taipei-CHN 27* Russian Federation 1 UNITED STATES 19* Italy Ukraine 17* Dubai-UAE 17* 22* Georgia 11* Bahrain 37* Palestinian Nat'l Auth -20 -10 0 10 20 30 40 Change in average score *p <. 05. Change in average scores is significant. NOTE: Education systems ordered according to average mathematics score in 2011. 37
Average mathematics scores of 8 thgrade students decreased from 2007 to 2011 in 6 education systems -12* Hungary -7* Sweden Malaysia -34* -14* Thailand -21* Jordan -15* Syrian Arab Republic -40 -30 -20 -10 Change in average score 0 10 20 *p <. 05. Change in average scores is significant. NOTE: Education systems ordered according to average mathematics score in 2011. 38
TIMSS international mathematics benchmarks Grade 4 Grade 8 Advanced (625) Students can apply their understanding and knowledge in a variety of relatively complex situations and explain their reasoning. Students can reason with information, draw conclusions, make generalizations, and solve linear equations. High (550) Students can apply their knowledge and understanding to solve problems. Students can apply their understanding and knowledge in a variety of relatively complex situations. Intermediate (475) Students can apply basic mathematical knowledge in straightforward situations. Low (400) Students have some basic mathematical knowledge. Students have some knowledge of whole numbers and decimals, operations, and basic graphs. 39
Percentages of U. S. 4 th-graders reaching TIMSS mathematics benchmarks were higher than international medians in 2011 Percent at or above 100 96 90 90 United States International median 81 80 69 70 60 47 50 40 28 30 20 13 4 10 0 Low Intermediate High Advanced Benchmark NOTE: All U. S. percentages are significantly higher than the corresponding TIMSS international median at the. 05 level of statistical significance. 40
Seven systems had higher percentages of 4 th-grade students reaching Advanced than the U. S. 43 Singapore 39 Korea, Rep. of 37 Hong Kong-CHN 34 Chinese Taipei-CHN 30 Japan 24 Northern Ireland-GBR 18 England-GBR Higher than U. S. (p <. 05) Not measurably different than U. S. (p <. 05) 16 North Carolina-USA 14 Florida-USA Russian Federation 13 UNITED STATES 13 12 Finland 0 10 20 30 40 Percentage reaching Advanced benchmark 50 60 NOTE: Education systems with lower percentages of students reaching the Advanced benchmark than the percentage of U. S. students reaching the Advanced benchmark are not included in figure. 41
Percentages of U. S. 8 th-graders reaching TIMSS mathematics benchmarks were higher than international medians in 2011 Percent at or above 100 92 90 United States International median 75 80 68 70 60 46 50 40 30 30 17 20 7 10 3 0 Low Intermediate High Advanced Benchmark NOTE: All U. S. percentages are significantly higher than the corresponding TIMSS international median at the. 05 level of statistical significance. 42
Eleven systems had higher percentages of 8 th-graders reaching Advanced than the U. S. Chinese Taipei-CHN Singapore Korea, Rep. of Hong Kong-CHN Japan Massachusetts-USA Russian Federation North Carolina-USA Minnesota-USA Israel Connecticut-USA Australia England-GBR Florida-USA Colorado-USA Hungary Turkey Indiana-USA UNITED STATES Quebec-CAN Romania Lithuania New Zealand Dubai-UAE California-USA 49 48 47 34 27 19 14 14 13 12 10 9 8 8 7 7 7 6 Higher than U. S. (p <. 05) 5 5 5 0 5 Not measurably different than U. S. (p <. 05) 10 15 20 25 30 35 40 Percentage reaching Advanced benchmark 45 50 NOTE: Education systems with lower percentages of students reaching the Advanced benchmark than the percentage of U. S. students reaching the Advanced benchmark are not included in figure. 55 43
SCIENCE
TIMSS 2011 science framework TIMSS science Content dimensions Grade 4 Earth science Life science Physical science Grade 8 Biology Chemistry Earth science Physics Cognitive dimensions Knowing Applying Reasoning 45
What is on the TIMSS science assessment? G R A D E 4 Percentage of 4 thgraders answering correctly: U. S. : 96% Int’l Avg. : 83% 46
What is on the TIMSS science assessment? G R A D E 4 Percentage of 4 thgraders answering correctly: U. S. : 90% Int’l Avg. : 73% 47
What is on the TIMSS science assessment? G R A D E Percentage of 8 thgraders answering correctly: U. S. : 28% Int’l Avg. : 27% 8 48
What is on the TIMSS science assessment? G R A D E Percentage of 8 thgraders answering correctly: U. S. : 76% Int’l Avg. : 61% 8 49
U. S. average score (544) higher than the TIMSS scale average (500) Higher than U. S. Not measurably different than U. S. Lower than U. S. 50
U. S. average score (544) lower than in 6 education systems Higher than U. S. Not measurably different than U. S. Lower than U. S. 51
U. S. average (544) not measurably different than in 3 education systems Higher than U. S. Not measurably different than U. S. Lower than U. S. 52
U. S. average score (544) higher than in 47 education systems Higher than U. S. Not measurably different than U. S. Lower than U. S. 53
U. S. average score (525) higher than the TIMSS scale average (500) Higher than U. S. Not measurably different than U. S. Lower than U. S. 54
U. S. average score (525) lower than in 12 education systems Higher than U. S. Not measurably different than U. S. Lower than U. S. 55
U. S. average (525) not measurably different than in 10 education systems Higher than U. S. Not measurably different than U. S. Lower than U. S. 56
U. S. average score (525) higher than in 33 education systems Higher than U. S. Not measurably different than U. S. Lower than U. S. 57
Average science scores of 4 th-grade students increased from 2007 to 2011 in 9 education systems 11* Japan 5 UNITED STATES 21* Czech Republic 9* Sweden 8* Netherlands 11* Denmark 17* Norway 37* Georgia 17* Iran, Islamic Rep. of 27* Tunisia -20 -10 0 10 20 30 40 Change in average score *p <. 05. Change in average scores is significant. NOTE: Education systems ordered according to average science score in 2011. 58
Average science scores of 4 th-grade students decreased from 2007 to 2011 in 5 education systems -19* Hong Kong-CHN -13* England-GBR -11* Italy -12* Australia -7* New Zealand -30 -20 -10 0 Change in average score 10 20 30 *p <. 05. Change in average scores is significant. NOTE: Education systems ordered according to average science score in 2011. 59
Average science scores of 8 th-grade students increased from 2007 to 2011 in 9 education systems 23* Singapore 7* Korea, Rep. of 15* Minnesota-USA 13* Russian Federation 5 UNITED STATES 13* Quebec-CAN 16* Ukraine 8* Norway 15* Iran, Islamic Rep. of 16* Palestinian Nat'l Auth -20 -10 0 10 Change in average score 20 30 40 *p <. 05. Change in average scores is significant. NOTE: Education systems ordered according to average science score in 2011. 60
Average science scores of 8 th-grade students decreased from 2007 to 2011 in 7 education systems -17* Hungary -15* Bahrain -20* Thailand -33* Jordan -44* Malaysia -26* Syrian Arab Republic -21* Indonesia -50 -40 -30 -20 Change in average score -10 0 10 *p <. 05. Change in average scores is significant. NOTE: Education systems ordered according to average science score in 2011. 61
TIMSS international science benchmarks Grade 4 Grade 8 Advanced (625) Students can apply knowledge and understanding of scientific processes and relationships and show some knowledge of the process of scientific inquiry. Students communicate an understanding of complex and abstract concepts in biology, chemistry, physics, and earth science. High (550) Students apply their knowledge and understanding of the sciences to explain phenomena in everyday and abstract contexts. Students demonstrate understanding of concepts related to science cycles, systems, and principles. Intermediate (475) Students have basic knowledge and understanding of practical situations in the sciences. Students recognize and apply their understanding of basic scientific knowledge in various contexts. Low (400) Students have some elementary knowledge of life science and physical science. Students can recognize some basic facts from the life and physical sciences. 62
Percentages of U. S. 4 th-graders reaching TIMSS science benchmarks were higher than international medians in 2011 Percent at or above 100 96 United States International median 92 90 81 80 72 70 60 49 50 40 32 30 15 20 5 10 0 Low Intermediate High Advanced Benchmark NOTE: All U. S. percentages are significantly higher than the corresponding TIMSS international median at the. 05 level of statistical significance. 63
Three systems had higher percentages of 4 th-grade students reaching Advanced than the U. S. Singapore 33 Korea, Rep. of 29 Finland 20 Russian Federation 16 Chinese Taipei-CHN 15 UNITED STATES Higher than U. S. (p <. 05) Not measurably different than U. S. (p <. 05) 15 Japan 14 Florida-USA 14 Hungary 13 North Carolina-USA 12 0 10 20 30 40 Percentage reaching Advanced benchmark 50 60 NOTE: Education systems with lower percentages of students reaching the Advanced benchmark than the percentage of U. S. students reaching the Advanced benchmark are not included in figure. 64
Percentages of U. S. 8 th-graders reaching TIMSS science benchmarks were higher than international medians in 2011 Percent at or above 100 93 90 79 80 United States International median 73 70 60 52 50 40 40 30 21 20 10 10 4 0 Low Intermediate High Advanced Benchmark NOTE: All U. S. percentages are significantly higher than the corresponding TIMSS international median at the. 05 level of statistical significance. 65
Twelve systems had higher percentages of 8 th-graders reaching Advanced than the U. S. 40 Singapore Massachusetts-USA Chinese Taipei-CHN Korea, Rep. of Japan Minnesota-USA Colorado-USA Connecticut-USA Russian Federation England-GBR Slovenia Florida-USA Finland North Carolina-USA Alberta-CAN Israel Australia Indiana-USA UNITED STATES Hong Kong-GBR New Zealand Hungary Turkey 24 24 20 18 16 14 14 13 13 13 12 12 11 11 10 10 Higher than U. S. (p <. 05) Not measurably different than U. S. (p <. 05) 9 9 9 8 0 10 20 30 40 Percentage reaching Advanced benchmark 50 60 NOTE: Education systems with lower percentages of students reaching the Advanced benchmark than the percentage of U. S. students reaching the Advanced benchmark are not included in figure. 66
Summary of change in average U. S. scores over time Grade 4 PIRLS Reading 2006 -2011 Grade 8 TIMSS Mathematics 2007 -2011 TIMSS Science 2007 -2011 Change over time in U. S. average scores was statistically significant. Change over time in U. S. average scores was not measurably different. 67
For more information PIRLS TIMSS PIRLS at NCES: http: //nces. ed. gov/surveys/pirls/ International Data Explorer: http: //nces. ed. gov/surveys/pirls/idepirls/ TIMSS at NCES: http: //nces. ed. gov/timss/ International Data Explorer: http: //nces. ed. gov/timss/idetimss/ Contact: Sheila Thompson NCES Sheila. Thompson@ed. gov 202 -502 -7425 Contact: Stephen Provasnik NCES Stephen. Provasnik@ed. gov 202 -502 -7480 68
- Slides: 68