Assessing Higher Order Thinking Skills Vickie Mohnacky April

Assessing Higher Order Thinking Skills Vickie Mohnacky April 19, 2011

Assessing Higher Order Thinking Skills Today’s Objectives • To identify personal, social and academic student outcomes in the Pre. K-Grade 12 Programming Standards • To align student outcomes with different types of assessments

What do you want your students to learn as a result of gifted education? • http: //www. twiddla. com/528716 What do I value the most? (As a teacher – as a human being)

What do we value most? • Following directions? • Timeliness? • Speed? • Accuracy? • Knowledge of content?

What do we value most? Complex thinking Originality Research Understanding systems Seeing connections Self Knowledge Perspective Abstract Concepts Leadership Civic Responsibility Empathy

How will we use the information? Teachers’ responses:

How will we use the information? Grade? Class rank? Determine entrance to program? Embarrass the student? Punishment? Predict success? Determine coursework? Measure growth/progress in learning? Plan instruction?

Assessment Uses • Determines what the student Pre-assessment already knows; • Determines readiness Formative Assessments Benchmark – Summative Assessments • A process during learning • As the learning is forming • An event after learning • Sums up learning

Formative Assessment An assessment FOR learning. Occurs while learning is forming. Not high stakes/Not for accountability. Used to adjust instruction/ improve student learning. Not for report card grades. Examples: • Teacher informal questioning • Warm-ups • Homework • K-W-L Chart

Benchmark/Interim Assessment An assessment OF learning – Interim points Not high stakes. Examples • End of chapter tests § Acuity http: //wvde. state. wv. us/oaa/acuity. php § DIBELS § Benchmark testing

Benchmark/Interim Assessment An assessment OF learning – Interim points Not high stakes. Used to adjust overall curriculum, programming More Examples: • Attitude scales • Interest Inventories • Established rubrics for class product and performances • Critical thinking tests/checklists (Cornell, Watson. Glaser, Test of Critical Thinking) • AP/IB course taking and performance

Summative Assessment An assessment OF learning; Sums up; Occurs at set points High stakes. Reporting and accountability Used to evaluate the overall effects of programming Examples • State Standardized Test (WESTEST 2) • NAEP (National Assessment of Educational Perf. ) • PISA (Program of International Student Assessment • Act Explore/ACT/SAT

Difference in Formative and Summative Assessments? How they are used Level of student involvement

What are the ultimate student outcomes? Critical Consumers Responsible Producers Brian Housand

What is the ultimate outcome? Solve

Houston, we have a problem • http: //www. youtube. com/watch? v=p b. BS 8 s. Hr. Dg. A&NR=1

The point is that we have to be clear in our objectives before we can differentiate instruction and properly assess our students’ attainment of those objectives. ” Rick Wormeli 2006 Fair Isn’t Always Equal

Look at Your Data • What does it measure? • Will it give you the information you need?

2010 Gifted Education Programming Standards Focus on Student Outcomes Students identified with gifts and talents demonstrate important learning progress as a result of programming and services (Assessment Criterion 2. 5)

Learning and Development Student Outcomes 1. 1 – 1. 3 Self-Understanding • Demonstrate self-knowledge of interests, strengths, identities and needs. • Possess developmentally appropriate understanding of how they learn and grow and recognize the influences of their beliefs, traditions and values on their learning and behavior • Demonstrate an understanding and respect for similarities and differences between themselves and their peer group and others in the general population.

Learning and Development Student Outcomes 1. 4 – 1. 5 Awareness of Needs • Access resources from community to support cognitive and affective needs. • Receive support from family and communities who understand similarities and differences with respect to the development and characteristics of advanced and typical learners.

Learning and Development Student Outcomes 1. 6 – 1. 8 Cognitive and Affective Growth • Benefit from meaningful and challenging learning activities. • Recognize preferred approaches to learning and expand their repertoire. • Identify future career goals that match their talents and abilities and resources needed to meet goals.

Curriculum Planning and Instruction Student Outcomes 2. 4 -2. 5 Assessment • Demonstrate advanced and complex learning as a result of using multiple, appropriate and ongoing assessments • Demonstrate important learning progress as a result of programming and services.

Curriculum Planning and Instruction Student Outcomes 3. 1 Curriculum Planning • Demonstrate growth commensurate with aptitude.

Curriculum Planning and Instruction Student Outcomes 3. 2 – 3. 3 Talent Development • Become more competent in multiple talent areas and across dimensions of learning. • Develop their abilities in their domain of talent and/or area of interest.

Curriculum Planning and Instruction Student Outcomes 3. 4 - 3. 5 Instructional Strategies • Become independent investigators. • Develop knowledge and skills for living and being productive in a multicultural, diverse and global society.

Curriculum Planning and Instruction Student Outcomes 3. 6 Resources • Benefit from programming that provides a variety of high quality resources and materials.

Learning Environments Student Outcomes 4. 1 Personal Competence • Demonstrate growth in personal competence and dispositions for academic and creative productivity (selfawareness, self-advocacy, self-efficacy, confidence, motivation, resilience, independence, curiosity, risk taking).

Learning Environments Student Outcomes 4. 2 – 4. 3 Social Competence - Leadership • Develop social competence manifested in positive peer relationships and social interactions. • Demonstrate personal and social responsibility and leadership skills.

Learning Environments Student Outcomes 4. 4 Cultural Competence • Value their own and other’s language, heritage, and circumstance. • Possess skills in communicating, teaming and collaborating with diverse individuals and across diverse groups. • Use positive strategies to address social issues, including discrimination and stereotyping.

Learning Environments Student Outcomes 4. 5 Communication Competence • Develop competence in interpersonal and technical communication skills. • Demonstrate advanced oral and written skills, balanced biliteracy or multiliteracy, and creative expression. • Display fluency and technologies that support effective communication.

• Specify clearly and exactly what it is you want to assess • Design questions and tasks that require students to demonstrate this skill or knowledge

Understanding By Design (UBD) 1. Identify desired results (what the student will know) 2. Determine acceptable evidence (how will the student demonstrate knowledge) 3. Plan instruction; Implement the design

“I began by seeing assessment as judging performance, then as informing teaching, and finally as informing learning. ” • Carol Ann Tomlinson

Types of Assessment • Standard • Selected Response (Multiple Choice) • Constructed Response (Short Answer) • Fill in the Blank • True/False • Performance-based or Product • Rubrics • Rating scales • Checklists

High School Entrance Exam – 1928 http: //wvde. state. wv. us/1928/

Standardized – Large Scale • Mass produced • Mass graded

Balanced Assessment System • Content – Knowledge • Reasoning • Performance Skill • Product

Standard/Objective: Type: Knowledge Reasoning Performance Skill Product Learning Targets What are the knowledge, reasoning, skill or product targets underpinning the standard/objective? Knowledge Targets Reasoning Targets Performance Skill Targets What knowledge would students need to master this skill? What reasoning proficiencies (if any) would students need to master this skill? What performance skills (if any) would students need to practice to master this skill? Product Targets What products (if any) would students need to practice creating to master this skill?

Example from real life: Drive with skill. Type: Knowledge Reasoning Skill Product Learning Targets What are the knowledge, reasoning, skill or product targets underpinning the standard/objective? Knowledge Targets • Know the law • Understand informal rules of the road • Understand what different parts of the car do • Read signs and understand what they mean • Understand what “creating a danger” means • Understand what “creating a hazard” means • Other? Reasoning Targets Performance Skill Targets • Analyze road conditions, vehicle performance, and other driver’s actions • Driving actions such as: steering, shifting, parallel parking, looking, signaling, backing up, braking, accelerating, etc. • Compare/contrast this information with knowledge and past experience • Synthesize information and evaluate options to make decisions on what to do next • Evaluate “Am I safe? ” and synthesize information to take action if needed. • Fluidity/automaticity in performance driving actions. • Other? Product Targets None Since the ultimate type of target is a performance skill, there are no embedded product targets

Standard/Objective: SS. O. 06. 05. 04: Interpret quotes of famous Americans from various periods of history Type: Knowledge Reasoning Performance Skill Product Learning Targets What are the knowledge, reasoning, skill or product targets underpinning the standard/objective? Knowledge Targets What knowledge would students need to master this standard? Reasoning Targets What reasoning proficiencies would students need to master this standard? Performance Skill Targets None? Product Targets None?

Standard/Objective: SC. O. 7. 1. 11: Construct. . . charts, graphs and tables for various purposes. . . Type: Knowledge Reasoning Performance Skill Product Learning Targets What are the knowledge, reasoning, skill or product targets underpinning the standard or benchmark? Knowledge Targets What knowledge would students need to master this standard? Reasoning Targets What reasoning proficiencies would students need to master this standard? Performance Skill Targets Product Targets What performance skills would students need to practice to master this standard? What products would students need to practice creating to master this standard?

Informal Questioning Concept Questions - • Why do we care about this? Why are we studying this? • Open ended • Broad, over-arching, outcome oriented questions • Will motivate students and target higher-order thinking. What is the big concept you are trying to uncover? Do the standards have any “big concept” words? Content Questions – • What are the facts? What are the procedures? • Closed ended. • Will help strengthen and develop students’ understanding of the larger questions.

Informal Questioning Concepts: Conflict – How can conflict be resolved? (How could this particular conflict have been resolved? ) Change - How can we cope with change? Friendship – What does it mean to be a friend? Freedom (of speech) – Is pure freedom of speech desirable in today’s world? Why or why not? http: //www. esldiscussions. com/index. html Questions Resource

Teachers who do not specifically plan classroom discussion questions ahead of time to tap particular higher-order thinking skills, but rather ask extemporaneous questions “on their feet, ” are likely to ask recall questions. Susan M. Brookhart, 2010

Checklists Checklist of items - the least complex form of assessment. Yes or No – Is it present or not? There is no value attached to the performance. All elements weighted the same. Skill ü On-task throughout time period Participates in class discussions Collaborates with other students Score To determine a score, the total number of checked items or the percentage of total possible. No quality is attached unless specified in the item. Example, “three paragraphs required” or “neatly” or “ 500 words. ” Meaning is then attached to the score. For example, what is the minimum score that would be considered proficient?

Checklists Example of basic speech skills: The student: ü Maintains eye contact with the audience Speaks loudly enough to be heard in all parts of the room Enunciates clearly Stands up straight (does not shift from foot to foot) Does not go over the allotted time Has notes Uses notes sparingly Score

Rating Scales Graphic rating scale on a continuum – example : Never Seldom Sometimes • Turns in lessons on time • Uses correct capitalization • Completes projects Usually Always

Rating Scales Numerical rating scale – example : 1 to 10 with 10 being the highest Performance Rating Time on task Participation in class discussion Developing a plan/Setting goals Perseverance during project Total:

Rating Scales Numerical rating scale with descriptions example: 1 = typical for grade/age 2 = above average for grade/age 3. Quite advanced for grade/age Rating 4 = Remarkable for grade/age (1 in 50) Learns quickly Shows power of concentration Enjoys “adult” conversations Has many and/or intense interests Asks many questions Invents, creates Total:

Rubrics A quality is attached to each skill. Start with the skill you want to assess. Example: Does the student reason inductively from the examples to arrive at a clear, accurate description of physical and chemical changes (example from book)? Completely and clearly – Response give clear evidence of reasoning from the examples. Partially – Response is accurate, but reasoning from examples isn’t clear or is only partial. No – Response does not demonstrate reasonable conclusions from the examples.

Rubrics A quality is attached to each skill. Very generalized example of SCALE: Scale refers to numerical or word ratings. 5 4 • Exceptional • Above or Mastery Distinguished 3 2 1 • Proficient or Mastery • Below Mastery or Proficient • Limited or Novice Scoring Performance Assessments: Scoring is not always simple and straightforward. Students generate their own responses.

Rubrics Another example. Very generalized example of Portfolio (Paulson & Paulson, 1996) Developing Competent Proficient • Preparation • Organization of the portfolio Rubrics • may have holistic areas or specific dimensions. • may be used for self-monitoring and self-assessment • may be used in authentic contexts • may be used formatively, summatively or as interim assessments

Rubrics A quality is attached to each skill. More descriptive example of Portfolio (Paulson & Paulson, 1996) Developing Competent Proficient • Reflection • Does not accurately self assess strengths and weaknesses Accurately self assesses strengths and weaknesses Focuses on what and how the knowledge or skill was learned • Goal setting • Goals vague; takes no responsibility for planning work Accepts responsibility/ describes progress toward goals Identifies future goals that link to strengths and weaknesses

Rubrics Define the highest performance level first. A three-level rubric is common, but five levels allow for some “gray areas. ” Skill Criteria • Accuracy of Information 100% accurate information • Craftsmanship Well organized, logical/ clear (strong word choices, good sentence variety, powerful images) • Reasoning Skills • Reasons are relevant and to the point the writer is making. Justifications Criteria Inaccurate information

4 3 2 1 Fluency I can think of many ideas. I can think of some ideas If I get some I have a hard help, I can think time thinking of ideas Flexibility I notice what is surprising and unusual I notice unusual things around me When someone I hardly ever reminds me, notice unusual things Evaluation I know several ways of deciding I can tell which ideas are worth working on With help, I cannot tell which ideas worthwhile are worthwhile Risk-taking I like to try new ideas I try new ideas Sometimes I try I do not try new ideas Seeking Challenges Goal setting (etc. ) I do not set goals Elaboration When I have I can usually good idea, I add details to make better great Sometimes, I can think of way to make better I do not know how to make better Score

Criteria Exemplary (4 -5) Good (2 -3) Needs Improvement (0 -1) Initial Questions are probing and help clarify facts All questions may not be relevant Few or no questions formulated Understanding the Clearly defines the problem Statement has some vagueness or missing information Problem defined incorrectly Seeking information Identifies several sources of information Relies on few sources Not clear as to what is needed Risk-taking I try new ideas Sometimes I try new ideas I do not try new ideas Integration of knowledge Effectively applies previous knowledge Applies limited amount of prior knowledge Unable to connect previous knowledge

General Rubric for Critical Thinking Involving Judgment Criteria 2 1 0 Thesis (judgment of credibility, identification of assumption or persuasive tactic, etc. ) Thesis is clear, is complete, and answers the question posed by the problem or task. Thesis is clear and at least partially answers the question posed by the problem or task. Thesis is not clear or does not answer the question posed by the problem or task. Evidence is accurate, relevant and complete. Evidence is mostly clear, relevant and complete. Evidence is not clear, relevant or complete. Reasoning and clarity The way in which the evidence supports thesis is clear, logical and well explained. The way in which the evidence supports thesis is mostly clear and logical. Some explanation is given. The way in which the evidence supports thesis is not clear, is illogical or is not explained.

The Crow and the Pitcher A crow, dying of thirst, came upon a pitcher which had once been full of water. When the crow put his beak into the mouth of the pitcher, he found that only very little water was left in it, and he could not reach far enough down to get at it. He tried and tried, but at last had to give up in despair. Then a thought came to him. He took a pebble and dropped it into the pitcher. Then he took another pebble and dropped it into the pitcher. At last he saw the water rising toward him, and after casting a few more pebbles into the pitcher, he was able to drink and save his life.

Consequences and Implications A 3 What would have happened if the crow had done the following: a) b) c) Cause and Effect A 2 What caused the water to reach the crow? What overall effect did it have on the crow? Sequencing What steps did the crow use to get water? Outline them below in A 1 1. 2. 3. 4. order: _________________ The Crow and the Pitcher d) Kept putting his beak in the bottle? Flown away? Broken the bottle? Waited for rain?

Main Idea/Theme What main idea(s) did you get from this story? Inference C 2 What made the crow successful in getting a drink of water? Why did his plan work? Characterization C 1 What are the crow’s most important qualities? What other characters have you read about that show similar life qualities? How were their situations similar or different from the crow’s situation? The Crow and the Pitcher C 3

Assessment/Response Form Main Idea Crows get thirsty. Crows are smart. The crow kept putting pebbles in the water until it could drink. I didn’t even know that. Inference The crow was successful because when you put the pebbles in the water, the water came up. That helped him to get water. He had to get pebbles and keep doing that until water came up. Characterization There’s not a book I read about that was similar to this one. The crow’s most important qualities are water and food. Teacher Comments:

Learning – Thinking Skills Rubric

Assess How?

Many people may be surprised that higherorder thinking can be assessed with “wellwritten” multiple choice test items. Susan M. Brookhart (2010)

NAEP (reasoning with data) The table below shows information about the weather in four cities on the same day. City 1 City 2 City 3 City 4 High Temperature 65 ◦ 80 ◦ 48 ◦ 25 ◦ Low Temperature Precipitation – Rain or Snow (inches) 56 ◦ 2 inches 66 ◦ 38 ◦ 0 inches 1 inch 10 ◦ 1 inch In which city did snow most likely fall at some time during the day? A. City 1 B. City 2 C. City 3 D. City 4

WESTEST 2 Do you use your phone to access your e-mail? Do you make a lot of calls on your cell phone? Do you spend a lot of time on your cell phone? Do you IM with your friends? Have you ever thought about reading a book on your phone? It’s all the rage in Japan! Cell phone novels, called keitai shosetsu, are becoming increasingly trendy in Japan and are starting to gain popularity in China and South Korea. The spread of cell phone novels is mainly due to their appeal to young adults, the same audience with whom blogging, IMing, text messaging, and cell phone use are all the rage. Since cell phone screens are so small, only about a … What is the effect of the series of rhetorical questions at the beginning of this article? A. They draw readers into the article by making them think about their answers to the questions. B. They illustrate the point that so many people now have cell phones at their disposal. C. They point out that the article’s main purpose is to talk about the many uses of cell phones. D. They show American readers can become successful novel writers using their cell phones.

“How to Assess Higher Order Thinking Skills” (example from book) Which of the following scenarios describes behavior that is legal because of the First Amendment? A. Mr. Jones threw a rock through the front window of Mr. Smith’s house. Around the rock was tied a paper that called Mr. Smith nasty names. B. Mr. Jones waited until Mr. Smith left for work one morning, then got in his car and followed him, honking and yelling. C. Mr. Jones doesn’t trust his neighbor, Mr. Smith. Jones believes Smith is a dangerous person and a threat to the peace of the neighborhood. Therefore, Mr. Jones buys a gun. D. Mr. Jones wrote a letter to the editor of the local paper. Mr. Smith heads a local environmental committee, and Mr. Jones called his position “disastrous. ”

“How to Assess Higher Order Thinking Skills” (example from book) Essay questions assessing the ability to make deductive conclusions: 1. Select one of the amendments in the Bill of Rights of the U. S. Constitution. Describe a specific example of one of the rights in the Bill of Rights. The example can be from a real event or something you make up yourself, but it must be a clear illustration of one of the rights in the Bill of Rights. Tell the story of your example. Then explain which right your story exemplifies, from which amendment, and tell why. Criteria for feedback or rubric: • • Appropriate identification of a particular right and amendment. Appropriateness of example. Appropriateness of evidence. Soundness of reasoning and clarity of explanation.

“How to Assess Higher Order Thinking Skills” (example from book) Questions assessing critical thinking involving judgment: 12. The poster shown above was made during the First World War. What was the poster designed to do? A. Make people feel that it would be easy to win the war. B. Make people feel guilty for thinking that war is harmful. C. Get people to join the army by making them feel responsible for starting the war. D. Get people to join the army by appealing to patriotic feelings.

“How to Assess Higher Order Thinking Skills” (example from book) Questions assessing critical thinking involving judgment: 12. The poster shown above was made during the First World War. What was the poster designed to do? Explain how you came to this conclusion. Criteria for feedback or rubric: • Clear, appropriate statement of the main point. • Appropriateness of evidence. • Soundness of reasoning and clarity of explanation.

Multiple Choice Examples from PISA – Program for International Student Assessment http: //nces. ed. gov/surveys/pisa/items. asp? sub=yes

Test Items from ACT http: //www. actstudent. org/sampletest/index. html

In Review: • What is the primary aim of assessment? • Who will use the information? • How will the information be used?

What other measures are there? How do we measure the football program? How do we measure the music/dance program?

What other measures are there? Competitions/Games Exhibits Bulletin Boards Websites

What other measures are there? Portfolio – of successes/best work. Knowing that the “failures” along the way were lessons learned on the way to success.

Resources • Julia Roberts NAGC’s WOW series • Susan K. Johnson NAGC’s WOW series • Brookhart, Susan M. (2010) Assess Higher-Order Thinking Skills • Dweck, Carol (2010) Even Geniuses Work Hard, Educational Leadership, September 2010, Vol. 68 No. 1 www. ascd. org • Tomlinson, Carol Ann, & Doubet, Kristina (2006) SMART in the Middle Grades, Westerville, OH, National Middle School Association • Wormeli, Rick (2006) Fair Isn’t Always Equal; Assessing & Grading in the Differentiated Classroom, p. 14, Stenhouse Publishers, Portland, Maine.

Resources New Online IEP Form: http: //wvde. state. wv. us/osp/forms. html Policy 2419 at http: //wvde. state. wv. us/policies. html Resources for teachers at http: //wvde. state. wv. us/osp/gifted. html Dweck, Carol (2010) Even Geniuses Work Hard, Educational Leadership September 2010, Vol. 68 No. 1 www. ascd. org Pink, Daniel (2006) A Whole New Mind, Riverhead Books Published by the Penguin Group. New York, NY. Reis, Sally & Renzullli, Joseph Curriculum Compacting: A Systematic Procedure for Modifying the Curriculum for Above Average Ability Students The National Research Center on the Gifted and Talented - University of Connecticut. Tomlinson, Carol Ann, & Doubet, Kristina (2006) SMART in the Middle Grades, Westerville, OH, National Middle School Association Van-Tassel-Baska, Joyce (2003) Content-Based Curriculum for High-Ability Learners, Waco, TX, Prufrock Press, Inc.