Teaching Learning and Testing Finding Congruence BILC Professional

Teaching, Learning, and Testing: Finding Congruence BILC Professional Seminar Zagreb, Croatia 16 October 2018 Ray Clifford

An old proverb says, Time is the best teacher. 2

But it has also been noted that: Time may be the best teacher, but it eventually kills all of its students. 3

A seminal work in Education is Bloom’s Taxonomy of Educational Objectives Note: the author called this book, “The most-quoted, least-read book ever published. ” There is a revised version titled, A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives, Abridged Edition. Editors, Lorin W. Anderson, et al. Longman, New York. 2001 www. celt. iastate. edu/teaching/Revised. Blooms 1. html 4

The revised version divides the original taxonomy into two dimensions: 1. Ways of thinking and 2. Types of knowledge. This revision is summarized in the chart on the next slide. • Thinking. The columns represent each of the six levels of the Cognitive Process Dimension—ranging from lower-order thinking skills at the left to higher-order thinking skills at the right. • Knowing. The rows represent the Knowledge Dimension—ranging from concrete knowledge at the bottom through abstract knowledge at the top. 5

Bloom’s Revised Taxonomy (With added highlighting and annotation) Common Combinations Cognitive Dimension Remember Understand Apply Analyze Evaluate Create Knowledge Dimension Metacognitive Procedural Conceptual Factual 6

Revised Bloom’s Taxonomy: Greater Accuracy, but Greater Complexity • 4 levels of knowledge. • 6 levels of cognition. • 4 x 6 equals 24 interactions (or combinations). Is this increased accuracy useful? How much complexity can humans handle? 7

“How Many Variables Can Humans Process? ” Graeme S. Halford, et al. Psychology Science, Vol 16, No. 1, pp. 70 -76, American Psychology Society, 2005. Page 70. n=30 / Three quotes from the research findings: • “As the order [the number] of the interaction[s] increases, the number of variables increases. ” • “Results showed a significant decline in accuracy and speed of [problem] resolution from three-way to four-way interactions. ” • “Furthermore, performance on a five-way interaction was at [the] chance level. ” 8

“How Many Variables Can Humans Process? ” Graeme S. Halford, et al. Psychology Science, Vol 16, No. 1, pp. 70 -76, American Psychology Society, 2005. n=30 (Trained individuals) 2 Problems Right 1 Problem Right 0 Problems Right My interpretation: Problem Complexity 2 -way 3 -way 4 -way 5 -way* 30 26 13 NA 0 4 13 12 0 0 4 10 Good OK Poor Terrible * One problem, n = 22. Chance responses, consistent with results extrapolated from the other study. 9

Can Bloom’s revised taxonomy be simplified to make it more useable? The Knowledge and Cognitive Dimensions can be (re)combined into three levels: 1. Direct Application of factual knowledge 2. Near transfer of conceptual knowledge. 3. Far transfer of abstract knowledge. These three levels can be applied to learning and to testing activities. 10

The expected level of learning: (3 Levels of Learning Outcomes) 1. Direct Application 2. Near Transfer 3. Far Transfer 11

The 1 st Level of Learning Outcomes • With direct application learning, students… • Memorize and practice specific responses. • Focus on the content of a specific course, textbook, or curriculum. • Learn only what is taught. 12

The 2 nd Level of Learning Outcomes • With near transfer learning, students… • Go beyond rote responses and use rehearsed and semi-rehearsed responses. • Focus on a specific set of tasks and communicative settings and respond within those domains. • Apply what they learn within a range of familiar, predictable settings. 13

The 3 rd Level of Learning Outcomes • With far transfer learning, students… • Develop the ability to transfer what is learned from one context to another. • Acquire the knowledge and skills needed to respond spontaneously to new, unknown, or unpredictable situations. • Learn how to continue learning and to become independent learners. 14

The testing method used: (3 Types of Tests) 1. Achievement 2. Performance 3. Proficiency 15

The 1 st Type of Test • Achievement tests measure: • Practiced, memorized responses. • Acquisition of what was taught. • The content of a specific textbook or curriculum. 16

The 2 nd Type of Test • Performance tests measure: • Rehearsed and semi-rehearsed responses. • One’s ability to respond in constrained, familiar, and predictable settings. • The transfer of learning to similar situations. 17

The 3 rd Type of Test • Proficiency tests measure: • Whether skills are transferable to new tasks. • One’s spontaneous, unrehearsed abilities. • The general ability to accomplish tasks across a wide variety of realworld settings. 18

Advanced Proficiency Requires Far Transfer Learning A By-Level Proficiency Summary with Text Types (Green = Far Transfer, Blue = Near Transfer, Red = Direct Application) NATO LEVEL FUNCTION/TASKS CONTEXT/TOPICS ACCURACY 5 All expected of an educated NS [Books] 4 Tailor language, counsel, motivate, persuade, negotiate [Chapters] Wide range of professional needs Extensive, precise, and appropriate 3 Support opinions, hypothesize, explain, deal with unfamiliar topics [Multiple pages] Practical, abstract, special interests Errors never interfere with communication & rarely disturb Narrate, describe, give directions [Multiple paragraphs] Q & A, create with the language [Multiple sentences] Concrete, realworld, factual Intelligible even if not used to dealing with non-NS Everyday survival Intelligible with effort or practice 2 1 0 Memorized [Words and Phrases] All subjects Random Accepted as a well-educated NS Unintelligible 19

A Personal Observation

Tests Define Instructional Expectations • Students (especially adults learners) don’t want to waste their time studying what is not going to be “needed”. • For students (and often commanders, teachers, and administrators); the tests used and not a course’s stated learning objectives define what is “needed”. • Therefore, tests can have a negative or a positive impact on learning. 21

“Washback” Effects of Testing has a negative impact when: • Educational goals are reduced to those that are most easily measured. • Testing procedures do not reflect course goals, for instance… üGiving multiple choice tests in speaking classes. üUsing grammar tests as a measure of general proficiency. 22

“Washback” Effects of Testing has a positive impact when: • Tests reinforce course objectives. • Tests act as change agents for improving teaching and learning. • The test results are useful for students, teachers, and administrators. 23

Aligning Teaching and Testing Promotes Learning • Direct Application <=> Achievement • Memorized responses using the content of a specific textbook or curriculum. • Near Transfer <=> Performance • Rehearsed ability to communicate in specific, familiar settings. • Far Transfer <=> Proficiency • Unrehearsed general ability to accomplish real-world communication tasks across a wide range of topics and settings. 24

When teaching and testing are not aligned, learning suffers. • Direct Application Teaching + Proficiency Testing => Learners will fail the tests. • Learners won’t be prepared for the tests. • Motivation will be reduced. • Far Transfer Teaching + Achievement Testing => You’ll get Direct Application learning. • Students will adjust their learning down to the level of the tests. • Motivation will be reduced. 25

When teaching and testing are not aligned, learning suffers. • Direct Application Teaching + Proficiency Testing => Learners will fail the tests. • Far Transfer Teaching + Achievement Testing => You’ll get Direct Application learning. Note # 1: The type of test used can limit the level of students’ learning. 26

Curriculum, Instructional, and Testing Constraints Can Also Reduce the Scope of Learning… 1. High academic goals are set and learner outcomes are defined. Instructional Goals and Learning Outcomes 2. Developers include only examples of the most frequently occurring or important goals in a textbook. Textbook 3. Teachers present as much of the textbook as time allows. Teaching 4. Students are tested on a sample of items drawn from the textbook. Test 27

Curriculum, Instructional, and Testing Constraints Can Reduce the Scope of Learning… 1. High academic goals are set and learner outcomes are defined. Instructional Goals and Learning Outcomes 2. Developers include only examples of the most frequently occurring or important goals in a textbook. Textbook 3. Teachers present as much of the textbook as time allows. Teaching 4. Students are tested on a sample of items drawn from the textbook. Test Note # 2: The reduced content of the tests used can limit the breadth of students’ learning. 28

An Alternative Teaching and Testing Model 1. Set instructional goals and define expected learner outcomes. Textbook Real-world Instructional Domains: cognitive understanding, psychomotor skills, and affective insights. 2 b. Test developers use an independent sample of the real-world domain areas to create proficiency tests that are not based on the textbook. Test Teacher Students 2 a. Course developers sample from the real-world domain areas to create a textbook. 3. Teachers adapt text materials to learners’ abilities, diagnose learning difficulties, adjust activities and add supplemental materials to help students apply new knowledge and skills in constrained achievement and performance areas, and then in real -world proficiency settings. 4. Students practice, expand, and then demonstrate their unrehearsed extemporaneous proficiency across a broad range of real-world settings that are not in the textbook. 29

Simply stated, “good” teaching (reinforced by “good” testing) can expand the scope of learning. 1. High academic goals are set and learner outcomes are defined. Instructional Goals and Learning Outcomes (STANAG 6001? ) 2. Developers include only examples of the most frequently occurring or important goals in a textbook. Textbook 3. Teachers present as much of the textbook as time allows. Teaching 4. Students are tested on a sample of items drawn from the textbook. Test 30

So, what constitutes “good” teaching and testing? 1. Making your teaching/learning activities match the level of your desired learner outcomes. 2. Using the kind of the tests that match the level of those learning outcomes. • Achievement tests for direct application objectives. • Performance tests for near transfer objectives. • Proficiency tests for far transfer objectives. 31

If those suggestions for “good” teaching are followed, a teaching and testing model will emerge that: • Will not be based on successively derived, reduced subsets of the original objectives. • Will maintain the students’ focus on the course’s true learning objectives. • Will reinforce the teacher’s role as a “facilitator” of learning, rather than as a “presenter” of information. 32

! ! ! 27 x 3 = The e 3 3 x. dl rk han e. o 3 esn’t wn only at a tim Do d ca ions n mi eract int Desired Learning Outcomes a. Direct application b. Near transfer c. Far transfer Keep it simple! Teaching Activities Work with a a. Presentation b. Application c. Problem solving maximum of three variables. Reduce complexity by aligning levels; i. e. use all “a”, all “b”, or all “c” levels. Testing a. Achievement b. Performance c. Proficiency 33

And once again… If things align, everything’s fine! Thank you. 34