Teaching and Assessing Critical Thinking Helping College Students

Teaching and Assessing Critical Thinking: Helping College Students Become Thinkers Diane F. Halpern, Ph. D Claremont Mc. Kenna College, Claremont, CA September 2012 With special thanks to a lifetime of coauthors, especially Milt Hakel, Art Graesser, Keith Millis, my NCUEP Colleagues, and the real thinkers—our wonderful students

Thinking About Critical Thinking o o o Six percent of Americans say the moon landing of 40 years ago was staged, according to Gallup. In Egypt, 43 percent of people think Israel was behind the 9/11 attacks in America, a poll by World. Public. Opinion. org found last year. And then, there is the flat earth society. Egan, T. (2009, August 9). Palin’s poison. The New York Times, Retrieved from http: //egan. blogs. nytimes. com/2009/08/09/palins-poison/

Examples over the past year o Power Balance Wristbands n n Improve energy, flexibility, and balance! Over 2. 5 million sold o o n n o Retails $29. 95 USD $35 million USD in 2010 Endorsed by several athletes Sports Product of the Year in 2010 by CNBC Proof?

Testimonial: Power Band Really Works

Examples over the past year o Power Balance Wristbands n n Improve energy, flexibility, and balance! Over 2. 5 million sold o o n n o Retails $29. 95 USD $35 million USD in 2010 Endorsed by several athletes Sports Product of the Year in 2010 by CNBC Proof?

What is Critical Thinking? Critical thinking is the use of those cognitive skills and abilities that increase the probability of a desirable outcome. It is purposeful, reasoned, and goal directed. It is the kind of thinking involved in solving problems, formulating inferences, calculating likelihoods, and making decisions. Critical thinkers use these skills appropriately, without prompting, and usually with conscious intent, in a variety of settings. That is, they are predisposed to think critically. When we think critically, we are evaluating the outcomes of our thought processes--how good a decision is or how well a problem is solved. Critical thinking also involves evaluating the thinking process--the reasoning that went into the conclusion we've arrived at or the kinds of factors considered in making a decision.

BEYOND ARGUMENT ANALYSIS

What is Critical Thinking? o Numerous definitions of Critical Thinking Effortful, careful, consciously controlled processing that maximizes the use of all available evidence and cognitive strategies, and purposefully strives to overcome individual biases (Riggio & Halpern, 2006) (for reviews see Ennis, 1993; Halpern, 2003; Moseley et al. 2005; Sternberg, Roediger, & Halpern, 2007) o Researchers generally agree that critical thinking is attempting to achieve a desired outcome by thinking rationally in a goal-oriented fashion o Critical Thinking is not the same construct as Intelligence (Stanovich, 2008)

Hart Research Associates Employer Survey-Intellectual and practical skills o o o o o The ability to communicate effectively, orally and in writing (89%) Critical thinking and analytical reasoning skills (81%) The ability to analyze and solve complex problems (75%) Teamwork skills and the ability to collaborate with others in diverse group settings (71%) The ability to innovate and be creative (70%) The ability to locate, organize, and evaluate information from multiple sources (68%) The ability to work with numbers and understand statistics (63%) Hart Research Associates. 2009. Raising the bar: Employers’ views on college learning in the wake of the economic downturn. Washington, DC: Hart Research Associates.

Hart Survey continued: Employers’ expectations of employees have increased Page 5

Management and Business Societies Agree: o o “Proficiency in reading, writing, and arithmetic has traditionally been the entry-level threshold to the job market, but the new workplace requires more from its employees. Employees need to think critically, solve problems, innovate, collaborate, and communicate more effectively” American Management Association, 2010, p. 1

Hunt (1995) asked: “Will We be Smart Enough? ” The answer to this question will determine the quality of life and the future of the planet. As Hunt persuasively argued, the workforce is one critical place where the dizzying pace of change can be witnessed. The number of jobs available in manufacturing is shrinking; those workers with poor cognitive skills can expect more competition for fewer jobs that pay poorly, while at the same time, there is an increased demand for a new type of worker--this new job category has been dubbed the "knowledge worker" 'or the "symbol analyst" to describe someone who can carry out multistep operations, manipulate abstract and complex symbols and ideas, efficiently acquire new information, and remain flexible enough to recognize the need for continuing change and new paradigms for lifelong learning.

Concern about critical thinking is not new—Paul Meehl stopped attending case conferences because of sloppy patterns of thought and inability to distinguish weak from strong evidence. “A casual anecdote about one’s (demented) uncle as remembered from childhood is given the same group interest and intellectual respect that is accorded to the citation of a high-quality (research) study. ” (Meehl, 1973, p. 228). o Cited by Tom Oltmanns and E. David Klonsky (2010)

Critical Thinking is What Intelligence Tests Miss “IQ tests measure only a small set of the thinking abilities that people need. ” o IQ tests are good measures of how well a person can hold beliefs in short-term memory and manipulate those beliefs, but they do not assess at all whether a person has the tendency to form beliefs rationally when presented with evidence. ” What we really want for our politicians, lawyers, doctors, and everyone else is to gauge their ability to think critically, which is largely absent from intelligence tests. o “Rational thinking can be surprisingly dissociated from intelligence. ” o --Keith Stanovich (2009, p. 3 and 39)

Our Students—the Millennials o o They are the children of late Baby Boomers (1945 -64) or Gen Xers (1965 -81) They spend 6. 5 hours a day on various media They are 36% of US population They are more diverse than any prior group

o o Multitaskers--Attention is limited, and when it is divided among too many tasks or the tasks are difficult, performance suffers. Of course, people have always been able to multitask (that is, attend to more than one thing): Parents monitor their children while cooking dinner, we carry on conversations while walking, students do homework while listening to music, and so on. But those who overload their systems by, for example, studying while checking e-mail, instant messaging, and watching television will do worse at all these tasks than they would if they focused on one at a time. The more they use the web, the more difficult it becomes to stay focused on long pieces of writing.

Rethinking the Purpose of Education The best education for life in the 21 st century must be built on the twin pillars of "learning how to learn" and how to think critically about the vast array of information that confronts us.

So, if you are thinking critically, you are wondering if it is possible to help students improve how they think? Why not? We teach writing, oral communication, math with the belief that these skills will transfer to appropriate situations.

Evidence That Better Thinking Can Be Learned with Appropriate Instruction o o o o "Blind" evaluations of programs designed to enhance thinking skills (e. g. , the Venezuela project) Student self reports (weak evidence, but students believe that they have improved) Gains in cognitive growth and development (e. g. , Piagetian tasks that measure cognitive stages) More expert-like mental representations following instruction (relative to control groups) Decision makers trained to “reorganize” existing knowledge in naturalistic settings showed more “expert-like” performance Tests of cognitive skills (e. g. , standardized tests for critical thinking) Spontaneous and uncued transfer (e. g. , call students at home months after the class is completed under the guise of a survey) Inductive reasoning tasks were taught to college students using realistic scenarios form many different domains. The authors conclude that critical thinking is “a skill” and that “it is transferable”

Most important robust finding: o “The majority of studies report positive impact on pupil’s attainment across a range of noncurriculum measures (such as reasoning or problemsolving). No studies reported negative impact on such measures (Thinking Skills Review Group, October 2003).

Thinking About Critical Thinking Instruction

Effective critical thinking instruction is predicated on two assumptions: (1) There are clearly identifiable and definable thinking skills that students can be taught to recognize and apply, and (2) If recognized and applied, the students will be more effective thinkers.

Teaching & Learning to Think Critically: A Four-Part Model 1. Explicitly teach/learn the skills of critical thinking 2. Encourage/develop the disposition of effortful thinking and learning 3. Direct learning activities in ways that increase the probability of transfer 4. Make metacognitive monitoring explicit and overt (Halpern, 1998, 2004)

Here is a generic list of thinking skills that would be applicable in many situations: o o o o o recognizing that a problem exists developing an orderly, planful approach so that tasks are prioritized and problems are recognized as differing with regard to how serious and urgent they are understanding how cause is determined, recognizing and criticizing assumptions, analyzing means-goals relationships, giving reasons to support a conclusion, assessing degrees of likelihood and uncertainty, incorporating isolated data into a wider framework using analogies to solve problems.

Generic Thinking Skills, continued o o o Relating new knowledge to information that was previously learned Using numerical information, including the ability to think probabilistically and express thoughts numerically Using matrices and other diagrams to communicate Synthesizing information from a variety of sources Determining credibility and using this information in formulating and communicating decisions Selecting among alternatives with the use of a reasoned method

Build on critical thinking skills throughout the curriculum o o o Identify the skills you want students to learn in your General Education critical thinking courses, then be sure they are used in other classes Deliberately add new skills throughout the curriculum—may be some differences by discipline, but ensure overlap among GE courses Some critical thinking skills that are firmly rooted in psychology n n n n Understanding that small samples yield more extreme results Recognizing and avoiding hindsight bias Knowing why we need control groups Avoiding either-or thinking (e. g. , is it nature or nurture) Being aware of the fallibility of memory Self-serving attributes for success and failure Halo effects

Dispositions for effortful thinking and learning Model and provide explicit instruction in the o o willingness to engage in and persist at a complex task conscious use of plans and suppression of impulsivity flexibility and open-mindedness willingness to abandon nonproductive strategies and self-correct

HCTA Facets of Critical Thinking Verbal Reasoning The skills needed to comprehend and defend against persuasive techniques embedded in everyday language. Argument Analysis The skills of identifying conclusions, rating the quality of reasons, and determining the overall strength of an argument. Hypothesis Testing The skills used in thinking as hypothesis testing are the same ones that are used in scientific inquiry. Likelihood and Uncertainty Correct use of probability and likelihood in everyday decision making. Decision Making and Problem Solving Ability to use multiple problem statements to define the problem, identify possible goals, generate and evaluate solutions.

: 1. Verbal Reasoning Skills The skills listed under this rubric include those that are needed to comprehend and defend against the persuasive techniques that are embedded in everyday language (also known as natural language). Thinking and language are closely tied constructs, and the skills included in this section recognize the reciprocal relationship between language and thought in which an individual's thoughts determine the language used to express them, and the language that is used shapes the thoughts.

A Voucher By Any Other Name “Do you favor or oppose allowing students and parents to choose a private school to attend at public expense? favor 41% “Do you favor or oppose allowing students and parents to choose any school, public or private, to attend using public funds? favor 63% Report from 36 th Annual Phi Beta Kappa (Wirthlin) Gallup Poll of Attitudes Toward Public Education Reported in Wall Street Journal. (2004, August 27), p. A 12.

Suppose you want to privatize social security (Bush) or oppose it (Gore) Use polls to find words voters like (if you are Bush) or don’t like (if you are Gore) For Bush: o Instead of social security say “retirement security” “compound interest, ” opportunity, and savings are good words For Gore: “Wall Street Roulette” “bankruptcy, ” and “break the contract” Green, Joshua. (2002, April). The Other War Room. Washington Monthly. http: //www. washingtonmonthly. com/features/2001/0204. green. html Accessed September 9, 2004.

Now rewrite one question As written: Should taxpayers subsidize tuition for illegal aliens at state colleges and universities? Rewrite it to slant toward an answer that would be supportive of paying tuition.

Think how you might respond to “Should the defendant serve life in jail without the possibility of parole? ” compared to “Should the defendant be sentenced to three years on probation and community service? ” Without knowing anything else, like the nature of the crime, can you see that most people would use these examples of sentences as anchors and tend to suggest much more severe sentences if they are asked the first sentence than if they are asked the second?

For each of the following categories, give the first example that comes to mind: o o o o o a bird a color a triangle (a picture is okay) a motor vehicle a sentence o a heroic action a game a philosopher a writer o o o o o Robin or sparrow Red or blue Drew or named equilateral Car or truck Short, declarative sentence (e. g. , The girl ran. ) Superman, Batman, fireman (all male) Single act (e. g. , saved from fire) Monopoly or other board game Socrates or Aristotle (dead, white male) Stephan King or other white male Best examples vary by culture—important implications Who is a hero, who is a philosopher, who should be promoted

2. Argument Analysis Skills An argument is a set of statements with at least one conclusion and one reason that supports the conclusion. In real life settings, arguments are complex with reasons that run counter to the conclusion, stated and unstated assumptions, irrelevant information, and intermediate steps. Arguments are found in commercials, political speeches, textbooks, and anywhere else where reasons are presented in an attempt to get the reader or listener to believe that the conclusion is true. The skills of identifying conclusions, rating the quality of reasons, and determining the overall strength of an argument should be sharpened in college course work.

The following example was taken from a catalogue that sells copper bracelets. I have altered it only slightly for this context. For hundreds of years people have worn copper bracelets to relieve pain from arthritis. This folklore belief has persisted and copper bracelets continue to be popular. These bracelets promote close contact between the copper and your wrist. What is the conclusion? What is the evidence or reasoning to support the conclusion? What is implied?

Making Arguments Worksheet Example 1 Does violence on television really have a negative influence on children's behavior? 1. State your conclusion. (although you may begin your formal writing here, but sure that the conclusion follows from your reasons). As you work, this is the last part this is filled in, not the first. 2. Give three reasons (or some other number) that support your conclusion. 3. Rate each reason as weak, moderate, strong, or very strong.

4. Give three counterarguments (or some other number) that weaken your conclusion. Rate how much each counterargument weakens the conclusion: little, moderate, much, or very much. 5. List any qualifiers (limitations on the reasons for or against—for example some evidence may be restricted to early childhood) 6. List any assumptions. 7. Are your reasons and counterarguments directly related to your conclusion? 8. What is the overall strength of your argument: weak, moderate, strong, or very strong? Now that you have completed this worksheet, rate the overall strength of your argument.

3. Skills in Thinking as Hypothesis Testing The rationale for this category is that much of our day-today thinking is like the scientific method of hypothesis testing. In many of our everyday interactions, people function like intuitive scientists in order to explain, predict, and control the events in their life. The skills used in thinking as hypothesis testing are the same ones that are used in scientific reasoning--the accumulation of observations, formulation of beliefs or hypotheses, and then using the information collected to decide if it confirms or disconfirms the hypotheses.

Clinical Example: o Design a study to compare a new mood enhancing drug, participation in an exercise program and group therapy for depressed teenagers. Be sure to include combination treatments.

Applying Research to Practice Consider this hypothetical example: o Researchers at Snooty University have studied the causes of divorce. They found that 53% of recently divorced couples reported that they had serious disagreements over money during the two-year period that preceded the divorce. The researchers concluded that disagreements over money are a major reason why couples divorce. They go on to suggest that couples should learn to handle money disagreements as a way of reducing the divorce rate.

Applying Research to Practice Consider this hypothetical example: Your local newspaper reported that children who attended preschool were better readers than those who did not attend preschool and that this advantage persisted throughout elementary school. Based on these data, a local politician is arguing that preschool should be available for all preschool children. What do you think?

o Suppose you are have a student in one of your classes and you are beginning to suspect that she is being abused by her boyfriend. How do you go about thinking about this possibility? (I am not considering reporting laws or treatment issues here, but thinking about whether she is being abused. )

Reasons For Reasons Against Thinking A. She is being 1. abused Strength: 2. Strength: 1. B. She is NOT being abused Strength: 2. Strength: 1. Strength: 2. Strength:

Everyday Thinking Like a Scientist There are probably 100 home remedies for the common cold. How should we decide which, if any, actually relieve cold symptoms? Probably, somewhere, sometime, someone gave you chicken soup when you had a cold. Undoubtedly, you got better. Almost everyone who gets a cold gets better. The question is, "Did the chicken soup make you better? " o o o o Self-fulfilling prophesy Placebo—time alone cures colds Need inert chicken-tasting warm liquid without the essential chicken Sample size Will turkey do? Operational definitions Replicability Random assignment to soup conditions

4. Using Likelihood and Uncertainty Because very few events in life can be known with certainty, the correct use of probability and likelihood plays a critical role in almost every decision. Huff's (1954) tiny, popular book How To Lie With Statistics is still widely quoted because it explains how easy it is to mislead someone who does not understand basic concepts in probability. The critical thinking skills that are subsumed under this heading are an important dimension of a college-level critical thinking taxonomy.

Using Liklihood and Uncertainty o o o A health survey was conducted in a sample of adult males in British Columbia, of all ages and occupations. Please give your best estimate of the following values: What percentage of the men surveyed have had one or more heart attacks? _____ What percentage of the men surveyed both are over 55 years old and have had one or more heart attacks? ____ Stop now and fill in the blanks above with your best estimate of these values. Over 65% of the respondents believed that a higher percentage of the men would be both over 55 and have had a heart attack than the percentage of men who reported that they had a heart attack. Do you recognize this as an example of a conjunction error? The probability of two uncertain events both occurring cannot be greater than the probability of just one of them occurring.

The Problem of False Positives o o Researchers designed a test that can correctly identify 99% of all ___ (people who will be terrorists or people with a rare disease) Someone you know was identified with this test? What is the probability that the diagnosis is correct?

Anchoring Demonstration o o o o Is the population of China more or less than 1. 5 Billion people? Estimate the population of China ___ +++++++++++ Is the population of China more or less than 40 Million people? Estimate the population of China___ +++++++++++ Weeks later—When interviewing for a job is it a better strategy to name a starting salary or wait for the employer to name one (assuming you have no special knowledge about this situation)

5. Decision Making and Problem Solving Skills In some sense, all of the critical thinking skills are used to make decisions and solve problems, but the ones that are included here involve the generation and selection of alternatives and judging among them. Many of these skills are especially useful in quantitative reasoning problems.

o Suppose that you walked into a voting booth on a primary election day and were faced with the following choice: County Solicitory (Choose One) Myron Jones - John Adams - Victor Light Unfortunately, you have not kept up with local politics, and are unfamiliar with the record of any of these candidates. Which candidate would you vote for? Studies indicate that you probably chose John Adams. In an election in New Hampshire about 15 years ago, John Adams, an unemployed cab driver who did not campaign, won the Republican nomination for the State's First Congressional District. Why did John Adams, a man who spent no money on his campaign and who never gave a speech, win his party's nomination? Psychologists believe that when voters were confronted with three names they didn't recognize, they picked the one linked in history with a political figure. Hence, prior exposure creates a sense of familiarity, which in turn can enhance your liking for the stimulus, a phenomenon known as the mere exposure effect.

Decision Making and Problem Solving—Lots of Possibilities o You and a friend just spent $7 each to see a movie. About a half hour into the movie, you both realize that it is "two thumbs down"--a really bad movie. What do you do? List some good reasons for staying until the end of the movie, then list some good reasons for leaving after a half hour. o Stop now and try this demonstration.

Are you an organ donor? If you live in Austria, Belgium, France or Poland, you almost certainly responded “yes. ” But if you live in Denmark, Netherlands, United Kingdom or Germany, you probably are not

We need to direct learning activities that make transfer more likely o o o o Draw a diagram/graphic that organizes information. What additional information would you want before answering the question? Explain why you selected (a particular) multiple choice question. State the problem in at least two ways. Which information is most important? Why? Which information is least important? Why? Categorize the information in a meaningful way.

Use elaboration to create interconnected knowledge structures (Dansereau)

Let’s take a few minutes and come up with some ways to help students think about o o Evidence and moon landings Biases in evaluating evidence and why so many Egyptians believe that Israel was behind the 911 attacks

What Makes Great Teachers Great? “Create a natural critical learning environment. "Natural" because what matters most is for students to tackle questions and tasks that they naturally find of interest, make decisions, defend their choices, sometimes come up short, receive feedback on their efforts, and try again. "Critical" because by thinking critically, students learn to reason from evidence and to examine the quality of their reasoning, to make improvements while thinking, and to ask probing and insightful questions. This is, by far, the most important principle -- the on which all others are based and which commands the greatest explanation. ” Ken Bain Chronicle of Higher Education, April 9, 2004; Volume 50, Issue 31, Page B 7

Measuring Critical Thinking

Halpern’s Taxonomy of Critical Thinking Skills A Guide for Instruction & Assessment Argument Analysis Likelihood/Uncertainty Verbal Reasoning CT measured In terms of Hypothesis Testing Decisions/Problem Solving

Halpern’s Critical Thinking Assessment o Verbal reasoning: ability to comprehend & defend against persuasive techniques embedded in everyday language; o Argument Analysis: ability to judge how well evidence supports a conclusion; o Hypothesis testing: accumulating observations, forming beliefs/hypotheses/seeking convergent evidence;

HCTA, continued o Likelihood & Uncertainty: recognizing regression to the mean, understanding the limits of extrapolation; probability & chance. o Decision making/problem solving: Seeking information to reduce uncertainty; listing alternatives; recognizing bias.

Assessing Critical Thinking – the Halpern Critical Thinking Assessment (HCTA) • Available on the Internet (2 versions) (With a commercial test publisher in Europe) • Scenario-based • Each competency is evaluated with forcedchoice & constructed responses • Forced-choice: recognition memory • Constructed response: free recall

Halpern Critical Thinking Assessment: Sample Question o o o A recent report in a magazine for parents showed that adolescents who smoke cigarettes also tend to get low grades in school. As the number of cigarettes smoked each day increase, GPA decreased. One suggestion made in this report was that we could improve school achievement by preventing adolescents from smoking. Based on this information, would you support this idea as a way of improving the school achievement of adolescents who smoke? Type “yes” or “no” and explain why or why not.

Based on this information, which is the best answer? 1. School grades probably will improve if we prevent adolescents from smoking. 2. School grades might increase, but we cannot be certain because we only know that grades go down when smoking increases. 3. There is no way to know because we only know that smoking and grades are related, not whether smoking causes grades to change. 4. There will probably be no effect on grades if we prevent adolescents from smoking because the magazine is written for parents, so it is probably biased against teen smoking.

Based on this information, which is the best answer? 1. School grades probably will improve if we prevent adolescents from smoking. 2. School grades might increase, but we cannot be certain because we only know that grades go down when smoking increases. 3. There is no way to know because we only know that smoking and grades are related, not whether smoking causes grades to change. 4. There will probably be no effect on grades if we prevent adolescents from smoking because the magazine is written for parents, so it is probably biased against teen smoking.

“Most people would sooner die than think, In fact, they do. ” --Bertrand Russell

Contact Information Dr. Diane F. Halpern Claremont Mc. Kenna College 850 Columbia Ave. Claremont, CA 91711 (909) 607 -9647 diane. halpern@cmc. edu