Learning Objectives Upon completion of this lesson participants
Learning Objectives Upon completion of this lesson, participants will be able to: • Distinguish between assessment and evaluation. • Recognize examples and non-examples of appropriate multiple-choice test item construction. • Summarize the guidelines for constructing multiple-choice test items. • Define principles of evaluation. • Identify the criteria of a good evaluation. • Identify appropriate evaluation methods for specific outcomes. • • Identify different types of course evaluation and evaluation data. Describe methods for evaluation of a training module. Prepare an end of course report. Identify and plan self development needs as an instructor.
Definitions Educational Assessment: is the process of documenting, usually in measurable terms, knowledge, skills, attitudes, and beliefs. Educational Evaluation: is the evaluation process of characterizing and appraising some aspect/s of an educational process. Course Evaluation: is a questionnaire, which requires a written or selected response answer to a series of questions in order to evaluate the instruction of a given course. The term may also refer to the completed survey form or a summary of responses to questionnaires.
TEST ITEM CONSTRUCTION
Why Multiple-Choice? MULTIPLE-CHOICE VERSATILITY RELATION OF ITEM TYPE TO TEST OBJECTIVES FACTOR ESSAY or ORAL SHORT ANSWER MULTIPLE-CHOICE Ability to organize ++ + - Potential diagnostic value -- + ++ Easily and reliably scored -- + ++ Takes little time to prepare + + - Measures higher mental processes ++ - ++ Broad content sampling -- + ++ Measures application ++ + ++ Adequate sampling of objectives -- + ++ Thorndike and Hagen (1969) have summarized the advantages of various essay and objective test formats. Positive attributes have been designated by plus signs in the table above, while distinct disadvantages have minus signs.
Overview MULTIPLE-CHOICE ITEMS • Anatomy • Construction guidelines • Examples TEST
Anatomy of a Test Item Stem + Alternatives = Test Item What is primarily responsible for the increase in the average length of life in the USA during the last fifty years? Distractor a. Compulsory health and physical education courses in public schools. Answer b. The reduced death rate among infants and young children. * Distractor c. The safety movement. Which has greatly reduced the number of deaths from accidents. Distractor d. The substitution of machines for human labor. Stem Alternatives
Advantages and Limitations Advantages of multiple-choice test items • Good for many subject areas • Takes less time to answer • Less guessing • Easy to score
Advantages and Limitations of multiple-choice items • Not good for displaying thought • Still subject to some guessing
Construction Guidelines 1. Constructed using a single written objective 2. Stem presents definite problem 3. Use simple and clear wording 4. Avoid clue words 5. Avoid grammatical giveaways 6. Alternatives of equal length 7. Highlight key words 8. Positively stated stems 9. Use as much wording as possible in stem 10. One clear best answer 11. Distractors are plausible and related
Examples Construct each item to assess a single written objective. Items that are not written with a specific objective in mind often end up measuring lower-level objectives exclusively or covering trivial material that is of little educational worth.
Examples POOR EXAMPLE California: a. b. c. d. Contains the tallest mountain in the United States. Has an eagle on its state flag. Is the second largest state in terms of area. Was the location of the Gold Rush of 1849. * BETTER EXAMPLE What is the main reason so many people moved to California in 1849? a. b. c. d. California land was fertile, plentiful, and inexpensive. Gold was discovered in central California. * The east was preparing for a civil war. They wanted to establish religious settlements.
Examples POOR EXAMPLE The term hypothesis, as used in research, is defined as: a. A conception or proposition formed by speculation or deduction or by abstraction and generalization from facts, explaining or relating an observed set of facts, given probability by experimental evidence or by factual or conceptual analysis but not conclusively established or accepted. b. A statement of an order or relation of phenomena that so far as is known is invariable under the given conditions, formulated on the basis of conclusive evidence or tests and universally accepted, that has been tested and proven to conform to facts. c. A proposition tentatively assumed in order to draw out its logical or empirical consequences and so test its accord with facts that are known or may be determined, of such a nature as to be either proved or disproved by comparison with observed facts. * BETTER EXAMPLE The term hypothesis, as used in research, is defined as: a. An assertion explaining an observed set of facts that has not been conclusively established. b. A universally accepted assertion explaining an observed set of facts. c. A tentative assertion that is either proved or disproved by comparison with an observed set of facts. * The alternatives in the poor example above are rather wordy, and may require more than one reading before the student understands them clearly. In the better example, the alternatives have been streamlined to increase clarity without losing accuracy.
Examples POOR EXAMPLE A word used to describe a noun is called an: a. b. c. d. Adjective. * Conjunction. Pronoun. Verb. BETTER EXAMPLE A word used to describe a noun is called: a. b. c. d. An adjective. * A conjunction. A pronoun. A verb.
Examples POOR EXAMPLE Which of the following is the best indication of high morale in a supervisor’s unit? a. The employees are rarely required to work overtime. b. The employees are willing to give first priority to attaining group objectives, subordinating any personal desires they may have. * c. The supervisor enjoys staying late to plan the next day. d. The unit gives expensive birthday presents to each other. BETTER EXAMPLE Which of the following is the best indication of high morale in a supervisor’s unit? a. b. c. d. The employees are rarely required to work overtime. The employees willingly give first priority to attaining group objectives. * The supervisor enjoys staying late to plan for the next day. The unit members give expensive birthday presents to each other.
Examples EXAMPLE Which of the following is the BEST example of treating an open wound? a. Stop the bleeding and apply a dressing. b. Stop the bleeding, apply a dressing and see a doctor. c. Clean the wound, stop the bleeding and apply a dressing. *
Examples Positively stated stems Negatively-worded items are those in which the student is instructed to identify the exception, the incorrect answer, or the least correct answer.
Examples POOR EXAMPLE If the pressure of a certain amount of gas is held constant, what will happen if its volume is increased? a. The temperature of the gas will decrease. b. The temperature of the gas will increase. * c. The temperature of the gas will remain the same. BETTER EXAMPLE If you increase the volume of a certain amount of gas while holding its pressure constant, its temperature will: a. Decrease. b. Increase. * c. Remain the same.
Examples POOR EXAMPLE The United States should adopt a foreign policy based on: a. b. c. d. A strong army and control of the North American continent. Achieving the best interest of all nations. Isolation from international affairs. Naval supremacy and undisputed control of the world’s sea lanes. * BETTER EXAMPLE According to Alfred T. Mahan, the United States should adopt a foreign policy based on: a. b. c. d. A strong army and control of the North American continent. Achieving the best interest of all nations. Isolation from international affairs. Naval supremacy and undisputed control of the world’s sea lanes. *
Examples POOR EXAMPLE Which of the following artists is known for painting the ceiling of the Sistine Chapel? a. b. c. d. Warhol. Flintstone. Michelangelo. * Santa Claus. BETTER EXAMPLE Which of the following artists is known for painting the ceiling of the Sistine Chapel? a. b. c. d. Botticelli. da Vinci. Michelangelo. * Raphael
Worksheet Write your objectives in this box. Keeps test item tied to a single objective. Write your test items in this box. Guidelines on the left will help you stay away from pitfalls that plague test writers.
Interim Summary MULTIPLE-CHOICE ITEMS • Anatomy • Construction guidelines • Examples
Critical Elements • • Data Judgment Educational objective Predetermined standard
Types of Evaluation • • Placement Diagnostic Formative Summative
Criteria for Evaluation • • • Validity Reliability Objectivity Comprehensiveness Differentiation
Organization of the Test • • Arrangement of test items Arrangement of alternatives Review test for accuracy Change test periodically
Interim Summary • Types of evaluation • Criteria of evaluation • Organization of the test
Effective Course Evaluation
Effective Course Evaluation Lesson Outline: • • What is evaluation? Portions of course to be evaluated Types of evaluation Collecting evaluation data Responding to significant feedback End of course report Instructor development
What is Evaluation? • Identifies and illuminates what is actually occurring in the area or activity that is the subject of evaluation • Produces sufficient information to enable the evaluator, using agreed upon criteria, to make valid and useful judgments about what is being evaluated
Portions to Evaluate • Appropriateness of objectives for the learning group • Organization and sequencing of content • Effectiveness of the instructional methods • Quality of teaching and learning aids used • Validity of assessment methods used • Performance of students on the course
Types of Evaluations
Data Collection Surveys Interviews Observation
Learner Feedback As learners are reaching the end of teaching or training, their experiences are particularly useful in terms of course evaluation.
Learner Performance The following approach is recommended in evaluating test scores. • Accurately record trainee and class (average) scores. • Compare class results against the standards. • Check effectiveness of objectives. • Make notes for later analysis. • Identify students that bring down class average. • If the class is achieving or surpassing standard, continue to evaluate. 4 t h 3 r d 1 st 2 nd 5 t h
Self Evaluation In evaluating lessons, the following questions are a good starting point. • What worked very well in terms of student learning, and why was it so effective? • What am I least satisfied in the lesson and why? • How would I change it for next time?
Responding to Feedback The following list shows some typical issues that may come up during evaluation. • Students find certain objectives too difficult to meet • Some objectives are outdated • Course materials are not sufficiently comprehensive • Certain parts of the course materials are confusing • Instructional methods may be uninteresting • Assessments are not fully testing all learning objectives • Assessment items are not correctly constructed
Preparing End of Course Report This feedback is critical to the development of the course. • Content of Report • Format of Report
Instructor Development Address the following questions. • What are the specific competencies that need to be developed? • What options are available to develop these competencies? • How will I monitor and evaluate progress? • What indicators will I use to determine success?
Summary • • What is evaluation? Portions of course to be evaluated Types of evaluation Collecting evaluation data Responding to significant feedback End of course report Instructor development
Close Create a program you can be proud of!!
- Slides: 40