Training Instructional Design Assessment This material Comp 20Unit

Training & Instructional Design Assessment This material (Comp 20_Unit 6) was developed by Columbia University, funded by the Department of Health and Human Services, Office of the National Coordinator for Health Information Technology under Award Number 1 U 24 OC 000003.

Assessment Learning Objectives By the end of this unit learners will be able to: • Identify an instructional problem • Plan and implement an instructional needs assessment • Analyze learner, task, and situational characteristics Health IT Workforce Curriculum Version 3. 0/Spring 2012 Training & Instructional Design 2 Assessment

Evaluation & Learner Assessment in Instructional Design Evaluate Analysis Implement Design Develop Health IT Workforce Curriculum Version 3. 0/Spring 2012 Training & Instructional Design 3 Assessment

Purpose of Program Evaluation Assessment determines the extent to which a program achieves its stated goals and objectives. The results can be used to inform: • • Decision-making to improve the program Revisions to program goals and objectives Budget requests and resource allocation Progress reports to management Health IT Workforce Curriculum Version 3. 0/Spring 2012 Training & Instructional Design 4 Assessment

The Evaluation Plan How well are the stated training goals being met? How will the information be collected? How will the data be used to improve the program? Health IT Workforce Curriculum Version 3. 0/Spring 2012 Training & Instructional Design 5 Assessment

The Evaluation Plan (Cont. ) The assessment plan should outline: • • Program goals Intended outcomes Data gathering & interpreting methods Plan for using data as evidence to inform program improvement • When and how the evaluation will be conducted Health IT Workforce Curriculum Version 3. 0/Spring 2012 Training & Instructional Design 6 Assessment

Develop a Logic Model Inputs • Resources • Contributions • Investments Outputs • • Activities Services Events Products Health IT Workforce Curriculum Version 3. 0/Spring 2012 Outcomes • Results or changes for participants Training & Instructional Design 7 Assessment

Simple Logic Model Get Pills Take Pills • INPUTS Health IT Workforce Curriculum Version 3. 0/Spring 2012 • OUTPUTS Feel Better • OUTCOMES Training & Instructional Design 8 Assessment

Logic Model: EHR Training Workshop OUTCOMES INPUTS OUTPUTS What we invest What we do Who we reach Short-term to long- term Participants Increase knowledge of software Patients Increase office efficiency Management Increase heath outcomes Trainers Lecture Funds Computers, software Training curriculum Individual & group work Program Evaluation Health IT Workforce Curriculum Version 3. 0/Spring 2012 Training & Instructional Design 9 Assessment

Purpose of the Logic Model The logic model helps us: • Match evaluation to the program • Determine what and when to measure • Decide on the type of evaluation, process and/or outcomes • Determine the testing instruments and procedures • Prioritize where to spend evaluation resources Health IT Workforce Curriculum Version 3. 0/Spring 2012 Training & Instructional Design 10 Assessment

Assessment & Evaluation Definitions Process evaluation: determines if activities are delivered to the target population as intended Outcome evaluation: determines if the program successfully produced the desired changes or outcomes Formative evaluation: gathers data to inform program improvement Summative evaluation: makes summary judgments about a program’s performance Health IT Workforce Curriculum Version 3. 0/Spring 2012 Training & Instructional Design 11 Assessment

Writing Assessment Plans For each evaluation objective, we need to identify: • Evaluation questions • Collection methods • Instruments used for data collection Health IT Workforce Curriculum Version 3. 0/Spring 2012 Training & Instructional Design 12 Assessment

Logic Model: EHR Training Workshop OUTCOMES INPUTS OUTPUTS What we invest What we do Who we reach Short-term to long- term Participants Increase knowledge of software Patients Increase office efficiency Management Increase heath outcomes Trainers Lecture Funds Computers, software Training curriculum Individual & group work Program Evaluation Health IT Workforce Curriculum Version 3. 0/Spring 2012 Training & Instructional Design 13 Assessment

Process & Formative Evaluation Questions • • What amount of money and time were invested? Were all the sessions delivered? How many persons were trained? Was the intended population trained? How well were the sessions delivered? What was the level of participation? Were participants satisfied? Health IT Workforce Curriculum Version 3. 0/Spring 2012 Training & Instructional Design 14 Assessment

Process Evaluation Plan Questions Collection Methods Amount of money Receipts & time Instruments Proposed budget Timesheets Number of Training sessions delivered curriculum Program requirements Number of Attendance persons trained records Population trained Attendance records Job descriptions Table 1. 1 Health IT Workforce Curriculum Version 3. 0/Spring 2012 Training & Instructional Design 15 Assessment

Process Evaluation Plan (Cont. ) Questions Collection Methods Instruments How well were the End of workshop sessions evaluation delivered? Self-report pencil and paper of online survey What was the level Number & quality of participation? questions asked Computer activity Observation rubric Were participants End of workshop satisfied? evaluation Self-report pencil and paper or online survey Table 1. 2 Health IT Workforce Curriculum Version 3. 0/Spring 2012 Training & Instructional Design 16 Assessment

Outcome Evaluation Questions • To what extent did skills improved? • To what extent did behaviors improve? • Which participants demonstrated a change in skills and behaviors? Health IT Workforce Curriculum Version 3. 0/Spring 2012 Training & Instructional Design 17 Assessment

Outcome/Learner Assessment Learner assessments start with good learning objectives. Create learning objectives that: • Focus on student performance • Outline measurable & observable tasks • State how you will know that the objective has been reached Health IT Workforce Curriculum Version 3. 0/Spring 2012 Training & Instructional Design 18 Assessment

Outcome/Learner Assessment (Cont. ) By end of the workshop, participants will be able to recall the steps of the intake process. • Can assess this objective with an oral test, or • Observe the participants’ performance as they actually use the computer to complete the task • But, multiple assessment methods are best to authentically measure student learning Health IT Workforce Curriculum Version 3. 0/Spring 2012 Training & Instructional Design 19 Assessment

Types of Learner Assessments Tests or Quizzes Specified increase in the number of correct responses as compared to a pretest or a specified number of students who are able to attain a targeted level of correct responses. Observations Specific student behaviors, e. g. to accomplish a specific task Student Writing Specific content knowledge or structural characteristics such as organization, development, or other specified features. Oral Level of participation, specific content knowledge, specific thinking skills, or other indications of student progress as indicated by student comments. Self Assessment Student comments, written statements, visual indicators, or other indications from students regarding their progress towards the learning goals. Table 1. 3 Health IT Workforce Curriculum Version 3. 0/Spring 2012 Training & Instructional Design 20 Assessment

Using Rubrics A rubric is a set of criteria used for assessing a piece of work or performance • Includes levels of achievement for each criterion • There will often be numerical scores for each level of achievement, and a summary score for the may be produced by adding the scores • May also include space to describe the reasons for each score Health IT Workforce Curriculum Version 3. 0/Spring 2012 Training & Instructional Design 21 Assessment

Sample Rubric The Performance Task requires students to collect and process information, using it for an authentic purpose. (Scarborough, 2006 [Based upon White & Scarborough, 2004]) • The task incorporates a variety of information gathering techniques and information resources. Students are required to interpret and synthesize information and accurately assess the value of information gathered. They are required to collect the right information for an authentic purpose, e. g. solve a problem, apply or use in a complex project, etc. • The task requires students to gather and synthesize information, but the value of the information gathered is not assessed. Information may not be used for a purpose. • The task requires the students to gather information, but not to interpret it. • The task requires no gathering or processing of information. Health IT Workforce Curriculum Version 3. 0/Spring 2012 Training & Instructional Design 22 Assessment

Outcome Evaluation Questions • To what extent did skills improved? • To what extent did behaviors improve? • Which participants demonstrated a change in skills and behaviors? Health IT Workforce Curriculum Version 3. 0/Spring 2012 Training & Instructional Design 23 Assessment

Outcome Evaluation/Learner Assessment Plan Questions Collection Methods Instruments To what extent did knowledge of the system improved? Written quizzes Participation in discussions To what extent did skills & behaviors improve? Performance task assessed in rubric workshop, and later on the job Which participants demonstrated a change in skills and behaviors? Compile Personnel records performance tasks Performance task results and group by rubric job description and department Table 1. 4 Health IT Workforce Curriculum Version 3. 0/Spring 2012 Test rubrics Discussion protocol Training & Instructional Design 24 Assessment

How to Use Evaluation Results can open up a discussion about a program’s strengths and weaknesses and inform changes to improve its effectiveness Compare program objectives with the outcome data, and determine the degree to which each objective was met or not met Understanding the reasons why a program succeeded or failed can lead to recommendations Health IT Workforce Curriculum Version 3. 0/Spring 2012 Training & Instructional Design 25 Assessment

Communicating Evaluation Results Evaluation report should outline: • The objectives and outcomes • How data was collected and identify sources (e. g. participants) • Barriers • Recommendations Health IT Workforce Curriculum Version 3. 0/Spring 2012 Training & Instructional Design 26 Assessment

Assessment Summary Program Evaluation & Learner Assessment • • Decision-making to improve the program Revisions to program goals and objectives Budget requests and resource allocation Progress reports to management Health IT Workforce Curriculum Version 3. 0/Spring 2012 Training & Instructional Design 27 Assessment

Assessment References 1. Clark, D. R. (2004). Bloom’s Taxonomy [monograph on the Internet]. Big Dog & Little Dog’s Performance Juxtaposition. Retrieved on June 21 st, 2010 from: http: //www. nwlink. com/~donclark/hrd/sat. html. 2. Clark, D. R. Instructional System Design (ISD). (2004). Big Dog & Little Dog’s Performance Juxtaposition. Retrieved on June 21 st, 2010 from: http: //nwlink. com/~donclark/hrd/ahold/isd. html. 3. Carkhuff, R. R. , & Fisher, S. G. (1984). Instructional systems design: volumes I & II. Amherst, MA: Human Resource Development Press. 4. Carliner, S. Training Design. (2003). Danvers, MA: American Society for Training and Development. 5. Chatterji, M. (2003). Designing and Using Tools for Educational Assessment. Massachusetts: Allyn & Bacon 6. Gagne, R. M. , Wager, W. W. , & Golas, K. (2004). Principles of Instructional Design (5 th ed. ). California: Wadsworth Publishing. 7. Reigeluth, C. M. Instructional-design Theories and Models: A new paradigm of instructional theory. Mahwah, New Jersey: Lawrence Erlbaum Associates, Inc. Charts, Tables & Figures 1. 1 & 1. 2 Table: Zimmerman, J. (2010). Process evaluation. Department of Biomedical Informatics, Columbia University Medical Center, New York, NY. 1. 3 Table: Zimmerman, J. (2010). Types of learner assessments. Department of Biomedical Informatics, Columbia University Medical Center, New York, NY. 1. 4 Table: Zimmerman, J. (2010). Outcome evaluation/learner assessment plan. Department of Biomedical Informatics, Columbia University Medical Center, New York, NY. Health IT Workforce Curriculum Version 3. 0/Spring 2012 Training & Instructional Design 28 Assessment