How to Effectively and Efficiently Conduct and Use

How to Effectively and Efficiently Conduct and Use Annual Assessment Director, Office of Academic Program Assessment (OAPA) Dr. Amy Liu, Professor of Sociology Assessment Consultants, OAPA Dr. Jacqueline Brooks, Professor of Sociology Dr. Chia-Jung Chung, Professor of Education Dr. Milica Markovic, Professor of Engineering http: //www. csus. edu/programassessment/ California State University, Sacramento Spring 2019 1

Outline 1. The Annual Assessment Process, OAPA, and Your Program 2. 2017 -2018 Feedback Report and Appendices as Resources How to Use the Feedback Reports to Improve Learning & Success 3. 2018 -2019 Assessment Template: Simple, Clear, and Useful Report One Outcome in Detail: How to Answer the Open-Ended Assessment Questions 2

Office of Academic Program Assessment (OAPA) • Run by faculty to provide guidance and feedback on program assessment • Help Academic Affairs – Program Review – WASC – Academic Strategic Planning & Resource Allocation • Help Programs - To make Annual Assessment: simple, clear, and useful (high quality) - To connect to the university strategic plan and assess the program strategic plan 3

How OAPA uses annual assessment? • Provide feedback to programs/departments to improve: – student learning and success – annual assessment – program review – strategic planning and resource allocation • Successfully complete University and Department accreditations – WASC, ABET, AACSB, or CCTC • Develop professional development workshops and FLCs • Assess strategic plan and use action plan for improvement 4

Outline 1. The Annual Assessment Process, OAPA, and Your Program 2. 2017 -2018 Feedback Report and Appendices as Resources How to Use the Feedback Reports to Improve Learning & Success 3. 2018 -2019 Assessment Template: Simple, Clear, and Useful Report One Outcome in Detail: How to Answer the Open-Ended Assessment Questions 5

Assessment Principles Assessment—an ongoing, interactive process using backward design and five basic principles: “Backward Design” for Assessment (Outcomes & Definitions) Results -Data Tables -Findings -Conclusions Updated Assessment Action Plan: -Using Assessment Data -Closing the loop Report step 2 -Curriculum map Methods and Measures -Assignments -Tests -Projects - Design Instruction Help Students Achieve Outcomes Report step 1 -Select -Define Expectations 1: -Develop or adopt Rubrics _ Define Standards of performance Decide on Assessments - Evidence of Learning Outcomes Plan step 3 PLOs: Plan step 2 Plan step 1 Define Learning Goals - Outcomes & Definitions - Curriculum Map Q 2. 2 6

Appendices 8 -9 Baccalaureate Learning Goals, Graduate Learning Goals, and AAC&U VALUE Rubrics (Simple) Refer to the table below to know the relationship between the BLGs and GLGs and how they both relate to the AAC&U VALUE rubrics. 1. Baccalaureate Learning Goals (BLGs) (Appendix 8 in 2017 -2018 Feedback) 2. Graduate Learning Goals (GLGs) (Appendix 9 in 2017 -2018 Feedback) 3. AAC&U (Associations of American Colleges and Universities) VALUE Rubrics 1. 1 Competence in the Disciplines 2. 1 Disciplinary Knowledge 3. 1 Competence in the Disciplines 1. 2 Knowledge of Human Cultures and the Physical and Natural World (GE) 3. 2 Knowledge of Human Cultures and the Physical and Natural World 1. 3 Intellectual and Practical Skills 2. 3 Intellectual and Practical Skills 3. 3 Intellectual and Practical Skills 1. 4 Personal and Social Responsibility (Values) 2. 4 Personal and Social Responsibility (Values) 3. 4 Personal and Social Responsibility (Values) 1. 5 Integrative Learning 3. 5 Integrative Learning 2. 6 Research 7

Baccalaureate Learning Goals, Graduate Learning Goals, AAC&U VALUE Rubrics, DQP, and Bloom’s Taxonomy (Detailed) 1. Baccalaureate Learning Goals (BLGs) (Appendix 8) 2. Graduate Learning Goals (GLGs) (Appendix 9) 3. AAC&U (Associations of American Colleges and Universities) VALUE Rubrics 4. DQP (Degree Qualification Profile: Learning Goals for Associate, Bachelor, and Master Degrees) 5. Bloom’s Revised Taxonomy 1. 1 Competence in the Disciplines 2. 1 Disciplinary Knowledge 3. 1 Competence in the Disciplines 4. 1 Specialized Knowledge 5. 1 Bloom’s Revised Taxonomy 3. 2 Knowledge of Human Cultures and the Physical and Natural World 4. 2 Broad and Integrative Knowledge 5. 2 Bloom’s Revised Taxonomy 4. 3 Intellectual Skills 5. 3 AAC&U VALUE Rubrics (Valid Assessment of Learning in Undergraduate Education) 1. 2 Knowledge of Human Cultures and the Physical and Natural World (GE) 1. 3 Intellectual and Practical Skills 1. 3. 1 Communication 1. 3. 1 a Written 1. 3. 1 b Oral 1. 3. 2 Critical Thinking 1. 3. 3 Literacy 1. 3. 3 a Information 1. 3. 3 b Quantitative 1. 3. 4 Inquiry and Analysis 1. 3. 5 Creative Thinking 1. 3. 6 Reading 1. 3. 7 Teamwork 1. 3. 8 Problem Solving 2. 3 Intellectual and Practical Skills 2. 3. 1 Communication 2. 3. 2 Critical thinking/analysis 2. 3. 3 Information literacy 3. 3 Intellectual and Practical Skills 3. 3. 1 Communication 3. 3. 1 a Written 3. 3. 1 b Oral 3. 3. 2 Critical Thinking 3. 3. 3 Literacy 3. 3. 3 a Information 3. 3. 3 b Quantitative 3. 3. 4 Inquiry and Analysis 3. 3. 5 Creative Thinking 3. 3. 6 Reading 3. 3. 7 Teamwork 3. 3. 8 Problem Solving 1. 4 Personal and Social Responsibility (Values) 1. 4. 1 Global Learning 1. 4. 2 Intercultural Knowledge and Competence 1. 4. 3 Civic Knowledge and Engagement 1. 4. 4 Ethical Reasoning 1. 4. 5 Foundations and Skills for Lifelong Learning 2. 4 Personal and Social Responsibility (Values) 2. 4. 1 Intercultural/global perspectives 3. 4 Personal and Social Responsibility (Values) 3. 4. 1 Global Learning 3. 4. 2 Intercultural Knowledge and Competence 3. 4. 3 Civic Knowledge and Engagement 3. 4. 4 Ethical Reasoning 3. 4. 5 Foundations and Skills for Lifelong Learning 5. 4 AAC&U VALUE Rubrics 2. 4. 6 Professionalism 1. 5 Integrative Learning 3. 5 Integrative Learning 2. 6 Research 4. 5 Applied and Collaborative Learning 5. 5 AAC&U VALUE Rubrics 4. 6 Civic and Global Learning 5. 6 Civic and Global Learning 8

Appendix 3: WSCUC “Rubric for Assessing the Quality of Academic Program Learning Outcomes” http: //www. wascsenior. org/search/site/Rubrics%20 combined Criterion Initial Emerging The list of outcomes is problematic: e. g. , very The list includes reasonable 1. Comprehensive incomplete, overly detailed, inappropriate, outcomes but does not specify and disorganized. It may include only expectations for the program as a List Developed The list is a well-organized set of reasonable outcomes that focus on the key knowledge, skills, and values students learn in the program. discipline-specific learning, ignoring relevant whole. Relevant institution-wide It includes relevant institution-wide outcomes institution-wide learning. The list may learning outcomes and/or national (e. g. , communication or critical thinking skills). confuse learning processes (e. g. , doing an disciplinary standards may be Outcomes are appropriate for the level internship) with learning outcomes (e. g. , ignored. Distinctions between (undergraduate vs. graduate); national application of theory to real-world problems). expectations for undergraduate and disciplinary standards have been considered. graduate programs may be unclear. Outcomes statements do not identify what Most of the outcomes indicate how Each outcome describes how students can do to demonstrate learning. students can demonstrate their demonstrate learning, e. g. , “Graduates can write “Statements understand scientific method” do learning. reports in APA style” or “Graduate can make not specify how understanding can be original contributions to biological knowledge. ” demonstrated and assessed. There is no clear relationship between the Students appear to be given The curriculum is designed to provide outcomes and the curriculum that students reasonable opportunities to develop opportunities for students to learn and to develop experience. the outcomes in the required increasing sophistication with respect to each curriculum. outcome. This design may be summarized in a curriculum map. There is no formal plan for assessing each The program relies on short-term The program has a reasonable, multi-year outcome. planning, such as selecting which assessment plan that identifies when each outcome(s) to assess in current year. outcome will be assessed. The plan may explicitly include analysis and implementation of improvements. 9 Highly Developed The list is reasonable, appropriate, and comprehensive, with clear distinctions between undergraduate and graduate expectations, if applicable. National disciplinary standards have been considered. Faculty has agreed on explicit criteria for assessing students’ level of mastery of each outcome. Outcomes describe how students can demonstrate their learning. Faculty has agreed on explicit criteria statements such as rubrics, and have identified examples of student performance at varying levels of each outcome. Pedagogy, grading, the curriculum, relevant 3. Alignment student support services, and co- curriculum are explicitly and intentionally aligned with each outcome. Curriculum map indicates increasing levels of proficiency. The program has a fully-articulated, sustainable, 4. Assessment multi-year assessment plan that describes when Planning and how each outcome will be assessed and how improvements based on findings will be implemented. The plan is routinely examined and revised, as needed. Students have some knowledge of Students have a good grasp of program outcomes. Students are well-acquainted with program 5. The Students know little or nothing about the overall outcomes of the program outcomes. Communication is They may use them to guide their own learning. outcomes and may participate in creation and use Experience Communication of outcomes to students, e. g. , occasional and informal, left to Outcomes are included in most syllabi and are of rubrics. They are skilled at self-assessing in in syllabi or catalog, is spotty or nonexistent. individual faculty or advisors. readily available in the catalog, on the web page, relation to the outcome levels of performance. and elsewhere. Program policy calls for inclusion of outcomes in all course syllabi, and they are readily available in other program documents. 2. Assessable Outcomes

Outline 1. The Annual Assessment Process, OAPA, and Your Program 2. 2017 -2018 Feedback Report and Appendices as Resources How to Use the Feedback Reports to Improve Learning & Success 3. 2018 -2019 Assessment Template: Simple, Clear, and Useful Report One Outcome in Detail: How to Answer the Open-Ended Assessment Questions 10

Make 2018 -2019 Assessment Simple, Clear, and Useful Section 1 – Report all outcomes assessed in 2018 -2019 • Q 1. 1, Q 1. 3 a Section 2 – Report one learning outcome in detail • Q 2. 1 -Q 5. 3. 1 Section 3 – Report other assessment activities • Q 6 -Q 9. 1 Section 4 – Background information about the program • Q 10 -Q 24 11

Section 1 – Report all outcomes assessed in 2018 -2019 What do you want your students to know, to do, and to value? 12

13

Baccalaureate Learning Goals, Graduate Learning Goals, AAC&U VALUE Rubrics, DQP, and Bloom’s Taxonomy (Detailed) 1. Baccalaureate Learning Goals (BLGs) (Appendix 8) 2. Graduate Learning Goals (GLGs) (Appendix 9) 3. AAC&U (Associations of American Colleges and Universities) VALUE Rubrics 4. DQP (Degree Qualification Profile: Learning Goals for Associate, Bachelor, and Master Degrees) 5. Bloom’s Revised Taxonomy 1. 1 Competence in the Disciplines 2. 1 Disciplinary Knowledge 3. 1 Competence in the Disciplines 4. 1 Specialized Knowledge 5. 1 Bloom’s Revised Taxonomy 3. 2 Knowledge of Human Cultures and the Physical and Natural World 4. 2 Broad and Integrative Knowledge 5. 2 Bloom’s Revised Taxonomy 4. 3 Intellectual Skills 5. 3 AAC&U VALUE Rubrics (Valid Assessment of Learning in Undergraduate Education) 1. 2 Knowledge of Human Cultures and the Physical and Natural World (GE) 1. 3 Intellectual and Practical Skills 1. 3. 1 Communication 1. 3. 1 a Written 1. 3. 1 b Oral 1. 3. 2 Critical Thinking 1. 3. 3 Literacy 1. 3. 3 a Information 1. 3. 3 b Quantitative 1. 3. 4 Inquiry and Analysis 1. 3. 5 Creative Thinking 1. 3. 6 Reading 1. 3. 7 Teamwork 1. 3. 8 Problem Solving 2. 3 Intellectual and Practical Skills 2. 3. 1 Communication 2. 3. 2 Critical thinking/analysis 2. 3. 3 Information literacy 3. 3 Intellectual and Practical Skills 3. 3. 1 Communication 3. 3. 1 a Written 3. 3. 1 b Oral 3. 3. 2 Critical Thinking 3. 3. 3 Literacy 3. 3. 3 a Information 3. 3. 3 b Quantitative 3. 3. 4 Inquiry and Analysis 3. 3. 5 Creative Thinking 3. 3. 6 Reading 3. 3. 7 Teamwork 3. 3. 8 Problem Solving 1. 4 Personal and Social Responsibility (Values) 1. 4. 1 Global Learning 1. 4. 2 Intercultural Knowledge and Competence 1. 4. 3 Civic Knowledge and Engagement 1. 4. 4 Ethical Reasoning 1. 4. 5 Foundations and Skills for Lifelong Learning 2. 4 Personal and Social Responsibility (Values) 2. 4. 1 Intercultural/global perspectives 3. 4 Personal and Social Responsibility (Values) 3. 4. 1 Global Learning 3. 4. 2 Intercultural Knowledge and Competence 3. 4. 3 Civic Knowledge and Engagement 3. 4. 4 Ethical Reasoning 3. 4. 5 Foundations and Skills for Lifelong Learning 5. 4 AAC&U VALUE Rubrics 2. 4. 6 Professionalism 1. 5 Integrative Learning 3. 5 Integrative Learning 2. 6 Research 4. 5 Applied and Collaborative Learning 5. 5 AAC&U VALUE Rubrics 4. 6 Civic and Global Learning 5. 6 Civic and Global Learning

Think-Pair-Share • Circle all the program learning outcomes assessed in your program. • Where does each PLO belong in the BLG and GLG? • Where does each PLO belong in the detailed curriculum map? 15

Think-Pair-Share • How is that outcome practiced in your detailed curriculum map? • Are your students struggling with thesis or critical thinking? If yes, what do you plan to do? 16

Table 5: Sociology Evidence Map at the Program Level: Other than GPA, What Data Are Used to Measure PLOs Outcomes Year 2012 – 2013 (program review) 2013 - 2014 - 2015 - 2016 Outcome 1: Advanced Sociological knowledge Outcome 2: Sociological contributions and applications Outcome 3: Challenges in sociology Outcome 4: Oral Communication Outcome 5: Written Communication Outcome 6: Critical Thinking Outcome 7: Intercultural Knowledge & Competence Outcome 8: Integration: Inquiry and Analysis Thesis Research paper Oral exam (thesis hearing)? Exit survey? Thesis defense 2016 - 2017 - 2018 Self-study 17

An Example: The Curriculum Map for the Sociology Graduate Program: Aligning (Linking) Graduate Program Learning Outcomes to Each Course in the Curriculum 1 “I” stands for “Introduced”, “D” for “Developed”, and “M” for “Mastered”. Outcome 1: Outcome 2: Outco Advanced Sociological mes Sociological Contributions Knowledge and Classes Applications Required Classes SOC 200 A I 1 I (Introduction) SOC 200 B (Thesis D, M Prep. ) SOC 214 (Methods) D, M SOC 215 (Analysis) D, M D SOC 235 (Soc. D, M D Psych. ) SOC 240 (Theory) D, M D SOC 500/599 (Thesis M M Work) Elective Classes SOC 210 (Urban D, M Sociology) SOC 220 (Change) D, M SOC 225 D, M (Stratification) SOC 226 (Gender) D, M SOC 230 (Social D, M Org. ) Outcome 3: Outcome 4: Outcome 5: Challenges in Oral Written Sociology Communicati on on Outcome 6: Critical Thinking Outcome 7: Outcome 8: Intercultural Integration: Knowledge & Inquiry and Competence Analysis I I I D, M D, M D I, D I, D I, D D M I, D I, D M I, D I, D I, D I, D I, D I, D D D, M D SOC 238 (Environmental) SOC 225 (Crime) SOC 260 (Contem. Issues. ) SOC 265 (Race) D, M D D I, D D, M D I, D I, D D, M I, D I, D SOC 266 (Family) D, M I, D D I, D

The Detailed Curriculum/Evidence Map for the Sociology Graduate Program: Aligning (Linking) Graduate Program Learning Outcomes to Key Assignments in Each Course in the Curriculum Outcomes Classes Outcome 1: Advanced Sociological knowledge Outcome 2: Sociological contributions and applications Outcome 3: Challenges in sociology Outcome 4: Oral Communication Required Classes SOC 200 A (Introduction) Reflection papers SOC 200 B (Thesis Prep. ) Thesis proposal SOC 214 (Methods) Research proposal Thesis proposal SOC 215 (Analysis) Data analysis SOC 235 (Soc. Psych. ) SOC 240 (Theory) Annotation/literature Annotation review Research paper Annotation/literature review Research paper Class presentation SOC 500/599 (Theses) Thesis Elective Classes SOC 210 (Urban Soc. ) Projects SOC 220 (Change) SOC 225 (Stratification) Projects SOC 226 (Gender) Projects SOC 230 (Social Org. ) Research papers SOC 238 (Environmental) Projects SOC 225 (Crime) Research papers Class projects Projects Outcome 6: Critical Thinking Outcome 7: Intercultural Knowledge & Competence Outcome 8: Integration: Inquiry and Analysis Reflection papers 1. Thesis proposal 2. Q 5 in student survey 1. Thesis proposal 2. Q 6 in student survey Research proposal Data analysis assignments Literature review Data analysis assignments Annotation Data analysis assignments 3 papers/exam Weekly paper Global perspectives Thesis 4 papers 2 essay exams Literature review Research paper/weekly paper 2 essay exams Essay exams Literature review Group/class projects Group/class project 20 page paper Research papers Global perspectives Research papers 20 page papers Cultural comparison 20 page paper Papers Class presentation Research paper/ 6 -8 critical review Weekly short paper Research paper/ Critical review Weekly short paper Content Research paper Class presentation 2 -3 class presentations Class presentation SOC 260 (Contem. Issues. Projects In Middle East) SOC 265 (Race) Projects SOC 266 (Family) Discussion and presentation 1. Thesis proposal 2. Oral presentation Outcome 5: Written Communication Class presentation Reflection papers Thesis proposal Reflection papers 1. Thesis proposal 2. Q 7 in student survey Research proposal 2 data analysis papers Literature review 20 page papers Thesis Short weekly paper Research paper 19 Literature review

An Example: The Curriculum Map for a Graduate Program: Aligning (Linking) Graduate Program Learning Outcomes to Each Course in the Curriculum 1 “I” stands for “Introduced”, “D” for “Developed”, and “M” for “Mastered”. Outcome 1 Outco mes Classes Required Classes CLASS 1 I, D CLASS 2 CLASS 3 CLASS 4 CLASS 5 CLASS 6 CLASS 7 Outcome 2 Outcome 3 D D Outcome 4 D Outcome 5 Outcome 7 D D D, M Outcome 8 I, D M D, M M I M CLASS 8 D M D CLASS 9 CLASS 10 CLASS 11 Outcome 6 D I, D D CLASS 12 CLASS 13 CLASS 14 D M D M I D D I D CLASS 15 CLASS 16 D M CLASS 17 CLASS 18 M D D M D M M 20

Section 2 – Report one learning outcome in detail 21

Outline for one PLO: 1. The PLO and Its Definition (Q 2. 1 and Q 2. 1. 1) 2. The Rubric(s) and Standard(s) of Performance/Expectations (Q 2. 2 a) 3. The Direct Measure(s) (Q 3. 3. 2) 4. The Data Table(s) (Q 4. 1) 5. The Assessment Plan = Action Plan (Q 5. 1. 1 -Q 5. 1. 2) 22

PLO 6: Critical Thinking in the Sociology Graduate Program • 6: Sociology graduate students will demonstrate a habit of systematically exploring social issues, ideas, artifacts, and events before accepting or formulating an opinion or conclusion” (Learning Goal/Outcomes): they will (PLO 6: Critical thinking adopted from the VALUE rubric): • 6. 1: Clearly identify and state the social issue/problem that needs to be considered critically, comprehensively describe the social issue/problem, and deliver all relevant information so it is necessary for a full understanding of the issue/problem (Explanation of issues); • 6. 2: Thoroughly interpret and evaluate sociological perspectives, theories, social methods/statistics, and any other current, credible, and relevant information (research, knowledge, tools, and/or views) to complete a thorough review of relevant literature and the problems to develop a comprehensive analysis or synthesis (Evidence); • 6. 3: Thoroughly analyze their own and others' assumptions and carefully evaluate the relevance of contexts when presenting a position (Influence of context and assumptions); • 6. 4: Consider the complexities (all sides) of a social issue. Limits of position and others' points of view are acknowledged and synthesized within position (Student's position); • 6. 5: Form conclusions, consequences and implications that are logical and reflect student’s informed evaluation and ability to place evidence and perspectives discussed in priority order (Conclusions and related outcomes). 23

Outline for one PLO: 1. The PLO and Its Definition (Q 2. 1 and Q 2. 1. 1) 2. The Rubric(s) and Standard(s) of Performance/Expectations (Q 2. 2 a) 3. The Direct Measure(s) (Q 3. 3. 2) 4. The Data Table(s) (Q 4. 1) 5. The Assessment Plan = Action Plan (Q 5. 1. 1 -Q 5. 1. 2) 24

The VALUE Rubric for the Critical Thinking Skill Exceed Standards 4 Criterion Meet Standards 3 Meet Minimum Standards 2 Don’t Meet Standards 1 6. 1: Explanation of issues Issue/problem to be considered critically is stated clearly and described comprehensively, delivering all relevant information necessary for full understanding. Issue/problem to be considered critically is stated, described, and clarified so that understanding is not seriously impeded by omissions. Issue/problem to be considered critically is stated but description leaves some terms undefined, ambiguities unexplored, boundaries undetermined, and/or backgrounds unknown. Issue/problem to be considered critically is stated without clarification or description. 6. 2: Evidence Selecting and using information to investigate a point of view or conclusion Information is taken from source(s) with enough interpretation/evaluation to develop a comprehensive analysis or synthesis. Information is taken from source(s) with enough interpretation/evaluation to develop a coherent analysis or synthesis. Information is taken from source(s) with some interpretation/evaluation, but not enough to develop a coherent analysis or synthesis. Information is taken from source(s) without any interpretation/evaluation. Viewpoints of experts are taken as fact, without question. 6. 3: Influence of context and assumptions Thoroughly (systematically and Identifies own and others' methodically) analyzes own and assumptions and several relevant others' assumptions and carefully contexts when presenting a evaluates the relevance of position. contexts when presenting a position. Questions some assumptions. Identifies several relevant contexts when presenting a position. May be more aware of others' assumptions than one's own (or vice versa). Shows an emerging awareness of present assumptions (sometimes labels assertions as assumptions). Continue onto next slide 25

The VALUE Rubric for the Critical Thinking Skill Criterion 6. 4: Student's position (perspective, thesis/hypothesis) 6. 5: Conclusions and related outcomes (implications and consequences) Exceed Standards 4 Specific position (perspective, thesis/hypothesis) is imaginative, taking into account the complexities of an issue. Limits of position (perspective, thesis/hypothesis) are acknowledged. Others' points of view are synthesized within position. Conclusions and related outcomes (consequences and implications) are logical and reflect student’s informed evaluation and ability to place evidence and perspectives discussed in priority order. Meet Standards 3 Meet Minimum Standards 2 Don’t Meet Standards 1 Specific position (perspective, thesis/hypothesis) takes into account the complexities of an issue. Others' points of view are acknowledged within position (perspective, thesis/hypothesis). Specific position (perspective, thesis/hypothesis) acknowledges different sides of an issue. Specific position (perspective, thesis/hypothesis) is stated, but is simplistic and obvious. Conclusion is logically tied to a range of information, including opposing viewpoints; related outcomes (consequences and implications) are identified clearly. Conclusion is logically tied to information (because information is chosen to fit the desired conclusion); some related outcomes (consequences and implications) are identified clearly. Conclusion is inconsistently tied to some of the information discussed; related outcomes (consequences and implications) are oversimplified. An example of the Program Standard of Performance for the Critical Thinking PLO: Eighty percent (80%) of our graduate students should achieve a score of at least 3 in all dimensions of the above rubric (eg. Soc. 200 B) and 3. 5 by the time of graduation (Soc. 500). The program standard of performance helps programs identify how well students perform within and across the program learning outcome (PLO). 26

Plan step 3 Define Standard of Performance Answers to Q 2. 2. a: Q 2: Standards of Performance/Expectations: Eighty percent (80%) of our students will score 3. 0 or above in all five dimensions of the VALUE rubric (by the time they graduate). 27 Expectations 2: -Define Standard of Performance

Outline for one PLO: 1. The PLO and Its Definition (Q 2. 1 and Q 2. 1. 1) 2. The Rubric(s) and Standard(s) of Performance/Expectations (Q 2. 2 a) 3. The Direct Measure(s) (Q 3. 3. 2) 4. The Data Table(s) (Q 4. 1) 5. The Assessment Plan = Action Plan (Q 5. 1. 1 -Q 5. 1. 2) 28

Attach assignment instructions that students received Questions from 2018 -19 Annual Assessment Report Template: Q 3. 3. 2. Please attach the assignment instructions that the students received to complete the assignment (Appendix I): Plan step 4 Q 3. 3. 1. Which of the following direct measures (key assignments, projects, portfolios, course work, student tests, etc. ) were used? [Check all that apply] 1. Capstone projects (including theses, senior theses), courses, or experiences Methods and 2. Key assignments from required classes in the program Measures 3. Key assignments from elective classes -Assignments 4. Classroom based performance assessments such as simulations, comprehensive exams, critiques -Tests 5. External performance assessments such as internships or other community based projects -Projects 6. E-Portfolios 7. Other portfolios 8. Other measure. Specify: Back to Guide 29

Example assignment description Answers to Q 3. 3. 2: The key assignment for the i. MET program assessment is the Action Research Report. i. MET used this Action Research Report (Master’s Thesis) included in an e. Portfolio as its direct measure to assess its critical thinking program learning outcome. This culminating experience report (the master thesis) includes the following tasks: 1. Designing and implementing a study using data collection tools that will allow the students to "show" the reader what happened during and as a result of the intervention. 2. Sorting through the findings after collecting the data, looking for data that reveal some information pertinent to the study. 3. Looking for relationships (patterns) between the data. These patterns emerge from a variety of sources such as things that have happened, things that students have observed, things that people have said, and things that students have measured. These are the findings (conclusions) of the study. 30

Is the annual assessment useful to you? What do you want your students to know, to do, and to value? Are your students struggling with thesis or critical thinking? If yes, what do you plan to do? 31

Outline for one PLO: 1. The PLO and Its Definition (Q 2. 1 and Q 2. 1. 1) 2. The Rubric(s) and Standard(s) of Performance/Expectations (Q 2. 2 a) 3. The Direct Measure(s) (Q 3. 3. 2) 4. The Data Table(s) (Q 4. 1) 5. The Assessment Plan = Action Plan (Q 5. 1. 1 -Q 5. 1. 2) 32

Think-Pair-Share: Data Presentation • What does the data table look like? • What should be included in the data table? 33

Describe results Report step 1 Questions from the 2018 -19 Annual Assessment Report Template: Q 4. 1. Please provide tables and/or graphs to summarize the assessment data, findings, and conclusions for the selected PLO in Q 2. 1 (see Appendix 12 in our Feedback Packet Example). Please do NOT include student names and other confidential information. This is going to be a PUBLIC document. Q 4. 3. For the selected PLO, the student performance: 1. Exceeded expectations/standards 2. Met expectations/standards 3. Partially met expectations/standards 4. Did not meet expectations/standards 5. No expectations or standards have been specified 6. Don’t know Back to Guide 34 Results -Data Tables -Findings -Conclusions

Five Criteria/outcomes 6. 1: Explanation of issues 6. 2: Evidence (Selecting and using information to investigate a point of view or conclusion) 6. 3: Influence of context and assumptions 6. 4: Student's position (perspective, thesis/ hypothesis) Exceed Standards Meet Standards (4) 38% (n=5) (3) 54% (n=7) 15% (n=2) 40% (n=6) 15% (n=2) Meet Minimum Standards (2) 0% (n=0) Don’t Meet Standards (1) Report step 1 Example Data Table Results -Data Tables -Findings -Conclusions Total (N=13) 8% (n=1) (100%, N=13) 21% (n=3) 15% (n=2) (100%, N=13) 41% (n=6) 20% (n=3) 15% (n=2) (100%, N=13) 23% (n=3) 54% (n=7) 8% (n=1) 15% (n=2) (100%, N=13) 15% (n=2) 55% (n=7) 15% (n=2) (100%, N=13) 6. 5: Conclusions and related outcomes (Implications and consequences) 35

Example Data Table Five Criteria/Outcomes 6. 1: Explanation of issues 6. 2: Evidence (Selecting and using information to investigate a point of view or conclusion) 6. 3: Influence of context and assumptions 6. 4: Student's position (perspective, thesis/ hypothesis) Exceed Standards (4) Meet Standards (3) Meet Minimum Standards (2) Don’t Meet Standards (1) Total (N=13) 38% (n=5) 54% (n=7) 0% (n=0) 8% (n=1) (100%, N=13) 6. 1: 38 + 54 = 92% achieving 3. 0 or higher. 15% (n=2) 40% (n=6) 21% (n=3) 15% (n=2) (100%, N=13) 6. 2: 15 + 40 = 55% not achieving 3. 0 or higher. 15% (n=2) 41% (n=6) 20% (n=3) 15% (n=2) (100%, N=13) 6. 3: 15 + 41 = 56% not achieving 3. 0 or higher. 23% (n=3) 54% (n=7) 8% (n=1) 15% (n=2) (100%, N=13) 6. 4: 23 + 54 = 77% achieving 3. 0 or higher. 15% (n=2) 55% (n=7) 15% (n=2) (100%, N=13) 6. 5: 15 + 55 = 70% achieving 3. 0 or higher. 6. 5: Conclusions and related outcomes (Implications and consequences) 36

Exceed Standards 4 Issue or problem to be considered critically is stated clearly and described comprehensively, delivering all relevant information necessary for full understanding. Criterion 6. 1: Explanation of issues Meet Standards 3 Issue or problem to be considered critically is stated, described, and clarified so that understanding is not seriously impeded by omissions. Meet Minimum Standards 2 Issue or problem to be considered critically is stated but description leaves some terms undefined, ambiguities unexplored, boundaries undetermined, and/or backgrounds unknown. Don’t Meet Standards 1 Issue or problem to be considered critically is stated without clarification or description. 6. 2: Evidence (Selecting and using information to investigate a point of view or conclusion) Information is taken from source(s) with enough interpretation or evaluation to develop a comprehensive analysis or synthesis. Information is taken from source(s) with enough interpretation or evaluation to develop a coherent analysis or synthesis. Information is taken from source(s) with some interpretation or evaluation, but not enough to develop a coherent analysis or synthesis. Information is taken from source(s) without any interpretation or evaluation. Viewpoints of experts are taken as fact, without question. 6. 3: Influence of context and assumptions Thoroughly (systematically and methodically) Identifies own and others' assumptions and analyzes own and others' assumptions and carefully several relevant contexts when presenting a evaluates the relevance of contexts when presenting position. a position. 6. 4: Student's position (perspective, thesis/ hypothesis) Specific position (perspective, thesis or hypothesis) is imaginative, taking into account the complexities of an issue. Limits of position (perspective, thesis/hypothesis) are acknowledged. Others' points of view are synthesized within position. Questions some assumptions. Identifies several Shows an emerging awareness of relevant contexts when presenting a position. present assumptions (sometimes labels May be more aware of others' assumptions assertions as assumptions). than one's own (or vice versa). Specific position (perspective, thesis or hypothesis) takes into account the complexities hypothesis) acknowledges different sides of of an issue. Others' points of view are an issue. acknowledged within position (perspective, thesis/hypothesis). Specific position (perspective, thesis or hypothesis) is stated, but is simplistic and obvious. 6. 5: Conclusions and related outcomes (Implications and consequences) Conclusions and related outcomes (consequences and implications) are logical and reflect students’ informed evaluation and ability to place evidence and perspectives discussed in priority order. Conclusion is logically tied to a range of information, including opposing viewpoints; related outcomes (consequences and implications) are identified clearly. Conclusion is logically tied to information (because information is chosen to fit the desired conclusion); some related outcomes (consequences and implications) are identified clearly. Conclusion is inconsistently tied to some of the information discussed; related outcomes (consequences and implications) are oversimplified. Q 2: Standards of Performance/Expectations: Seventy (70%) of our students will score 3. 0 or above in all the five dimensions of the VALUE rubric (by the time they graduate from the four semester program. ) 37

Results: Numerical Conclusions Report step 1 Q 2: Standards of Performance/Expectations: Seventy percent (70%) of our students will score 3. 0 or above in all the five dimensions of the VALUE rubric (by the time they graduate from the four semester program. ) Summary conclusion for Q 4. 2 Students meet the following standards 6. 1 (92%), 6. 4 (77%) and 6. 5 (70%). Students do not meet the following standards 6. 2 (61%) and 6. 3 (61%). Results -Data Tables -Findings -Conclusions The two areas needing improvement: 6. 2: Evidence (61%) 6. 3: Influence of context and assumptions (61%). Student Performance (Q 4. 3): 3. Partially meet the standards 38

Data Collection Sheet for Each Student Exceed Standards 4 Issue or problem to be considered critically is stated clearly and described comprehensively, delivering all relevant information necessary for full understanding. Criterion 6. 1: Explanation of issues Meet Standards 3 Issue or problem to be considered critically is stated, described, and clarified so that understanding is not seriously impeded by omissions. Meet Minimum Standards 2 Issue or problem to be considered critically is stated but description leaves some terms undefined, ambiguities unexplored, boundaries undetermined, and/or backgrounds unknown. Don’t Meet Standards 1 Issue or problem to be considered critically is stated without clarification or description. 6. 2: Evidence (Selecting and using information to investigate a point of view or conclusion) Information is taken from source(s) with enough interpretation or evaluation to develop a comprehensive analysis or synthesis. Information is taken from source(s) with enough interpretation or evaluation to develop a coherent analysis or synthesis. Information is taken from source(s) with some interpretation or evaluation, but not enough to develop a coherent analysis or synthesis. Information is taken from source(s) without any interpretation or evaluation. Viewpoints of experts are taken as fact, without question. 6. 3: Influence of context and assumptions Thoroughly (systematically and methodically) Identifies own and others' assumptions and analyzes own and others' assumptions and carefully several relevant contexts when presenting a evaluates the relevance of contexts when presenting position. a position. 6. 4: Student's position (perspective, thesis/ hypothesis) Specific position (perspective, thesis or hypothesis) is imaginative, taking into account the complexities of an issue. Limits of position (perspective, thesis/hypothesis) are acknowledged. Others' points of view are synthesized within position. Questions some assumptions. Identifies several Shows an emerging awareness of relevant contexts when presenting a position. present assumptions (sometimes labels May be more aware of others' assumptions assertions as assumptions). than one's own (or vice versa). Specific position (perspective, thesis or hypothesis) takes into account the complexities hypothesis) acknowledges different sides of of an issue. Others' points of view are an issue. acknowledged within position (perspective, thesis/hypothesis). Specific position (perspective, thesis or hypothesis) is stated, but is simplistic and obvious. 6. 5: Conclusions and related outcomes (Implications and consequences) Conclusions and related outcomes (consequences and implications) are logical and reflect students’ informed evaluation and ability to place evidence and perspectives discussed in priority order. Conclusion is logically tied to a range of information, including opposing viewpoints; related outcomes (consequences and implications) are identified clearly. Conclusion is logically tied to information (because information is chosen to fit the desired conclusion); some related outcomes (consequences and implications) are identified clearly. Conclusion is inconsistently tied to some of the information discussed; related outcomes (consequences and implications) are oversimplified. Q 2: Standards of Performance/Expectations: Seventy percent (70%) of our students will score 3. 0 or above in all the five dimensions of the VALUE rubric (by the time they graduate from the four semester program. ) 39

Question 4 (Q 4): An example of a Data Collection Sheet for Each Student/Assignment Reference: Your data tables are based on the rubric and the data collection sheet. The moment your rubric is developed, you also have a data collection sheet! The following table is an example of a data collection sheet for a student: 40

Report step 1 Results: Conclusions Answer for Q 4. 2: Results -Data Tables -Findings -Conclusions We can see (using the above table) that students meet the criteria for 6. 1 (92%), 6. 4 (77%), and 6. 5 (70%) based on the assessment of our selected Critical Thinking PLO and our identified program standards of performance (70% of students should achieve a score of 3 or higher in all dimensions of the Critical Thinking Rubric). Students do not meet the criteria of 6. 2 (61%) and 6. 3 (61%). Students meet some of our program standards for the Critical Thinking Skill, thus they “Partially Met Program Standards. ” Two areas need improvement: 1) Criterion 6. 2: Evidence (61%), and 2) Criterion 6. 3: Influence of context and assumptions (61%). 41

Outline for one PLO: 1. The PLO and Its Definition (Q 2. 1 and Q 2. 1. 1) 2. The Rubric(s) and Standard(s) of Performance/Expectations (Q 2. 2 a) 3. The Direct Measure(s) (Q 3. 3. 2) 4. The Data Table(s) (Q 4. 1) 5. The Assessment Plan = Action Plan (Q 5. 1. 1 -Q 5. 1. 2) 42

Think-Pair-Share: Data Analysis and Presentation • How useful is the data? • Can you use the data to improve student learning and success? • What issues are you struggling with? – Are you collecting data to address those issues? – Have you used the data to promote your program? 43

Question 5. 2: Use of Assessment Data Report step 2 Q 5. 2. To what extent did you apply previous assessment results collected through your program in the following areas? [Check all that apply] (1) (2) (3) Very Much Quite a Bit Some Updated 1. Improved specific courses X Assessment 2. Modified curriculum X Action Plan: 3. Improved advising and mentoring X 4. Revised learning outcomes/goals X -Using 5. Revised rubrics and/or expectations Assessment 6. Developed/updated assessment plan X Data 7. Annual assessment reports X 8. Program review X -Closing the 9. Prospective student and family information loop 10. Alumni communication 11. WASC accreditation (regional accreditation) X 12. Program accreditation X 13. External accountability reporting requirement X 14. Trustee/Governing Board deliberations 15. Strategic planning X 16. Institutional benchmarking X 17. Academic policy development or modification X 18. Institutional Improvement X 19. Resource allocation and budgeting 20. New faculty hiring X 21. Professional development for faculty and staff 22. Recruitment of new students 23. Other Specify: 44 (4) Not at all (8) N/A X X X X

Question 5. 1. 1: Use of Assessment Data Questions from 2018 -19 Annual Assessment Report Template: Q 5. 1. 1. Please describe what changes you plan to make in your program as a result of your assessment of this PLO. Include a description of how you plan to assess the impact of these changes. Report step 2 Answer to Q 5. 1. 1: In order to help students in our program successfully become critical thinking researchers, we will design more classroom activities and assignments related to: 1) Re-examination of evidence (6. 2) and context and assumptions (6. 3) in the research, and 2) Require students to apply these skills as they compose comprehensive Updated Assessment responses for all their assignments. Action Plan: Back to Guide 45 -Using Assessment Data -Closing the loop

Thank you! 46

A Simple Example 1 Educational Technology (i. MET), MA (Example of a graduate-level Intellectual skills PLO with multiple dimensions) 47

A Simple Example 2 Chemistry, BS/BA (Example of an Undergraduate-level Disciplinary Competence PLO) 48

Table 5: Sociology Evidence Map at the Program Level: Other than GPA, What Data Are Used to Measure PLOs Outcomes Year 2012 – 2013 (program review) 2013 - 2014 - 2015 - 2016 Outcome 1: Advanced Sociological knowledge Outcome 2: Sociological contributions and applications Outcome 3: Challenges in sociology Outcome 4: Oral Communication Outcome 5: Written Communication Outcome 6: Critical Thinking Outcome 7: Intercultural Knowledge & Competence Outcome 8: Integration: Inquiry and Analysis Thesis Research paper Oral exam (thesis hearing)? Exit survey? Thesis defense 2016 - 2017 - 2018 Self-study 49

How programs have used and/or can use assessment data Q 5. 1. Q 1. 1. 36% 32% 26% 19% 18% 17% 13% 10% 8% 8% 6% 6% 5% 5% 4% 4% 3% 1% Written communication Inquiry and analysis Critical thinking Overall competencies in the major/discipline Information literacy Oral communication Integrative and applied learning Ethical reasoning Problem solving Intercultural knowledge and competency Others Quantitative literacy Team work Creative thinking Foundations and skills for lifelong learning Reading Overall competencies for GE Knowledge Global learning Civic knowledge and engagement The Office of Academic Program Assessment Preliminary Data, 2014 Annual assessment reports Developing/updating assessment plan Revising rubrics and/or expectations Improving specific courses Program review Modifying curriculum Improving advising and mentoring Revising learning outcomes/goals Professional development for faculty and staff New faculty hiring Academic policy development or modification External accountability reporting requirement Strategic planning Alumni communication Program accreditation Prospective student and family information Institutional Improvement WASC accreditation (regional accreditation) Recruitment of new students Resource allocation and budgeting Institutional benchmarking Trustee/Governing Board deliberations Very much Quite a bit 30% 12% 21% 14% 21% 9% 10% 14% 8% 9% 10% 6% 13% 6% 12% 5% 13% 5% 6% 5% 4% 4% 9% 4% 1% 3% 9% 3% 5% 1% 3% 0% 10% 0% 4% 0% 3% 0% 0% Some 14% 21% 30% 25% 15% 35% 26% 21% 24% 12% 4% 17% 10% 4% 6% 13% 13% 6% 5% Not at all 8% 9% 4% 15% 9% 13% 11% 23% 13% 27% 22% 24% 33% 19% 33% 21% 32% 27% 18% No answer 37% 36% 37% 31% 54% 33% 44% 39% 51% 53% 65% 46% 51% 65% 53% 60% 56% 51% 53% 67% 77%

University Learning Outcomes • • • Baccalaureate Learning Goals Graduate Learning Goals AAC&U VALUE Rubrics DQP Bloom’s Revised Taxonomy 51

Review 2017 -2018 Assessment Report Feedback Activity Results -Data Tables -Findings -Conclusions Report step 2 Methods and Measures -Assignments -Tests -Projects Report step 1 Expectations 2: -Define Standard of Performance Plan step 4 Expectations 1: -Develop or adopt Rubrics 2. Definne Plan step 3 PLOs: -Select -Define -Curriculum map Plan step 2 Plan step 1 1. Please review 2017 -2018 Assessment Report Feedback from this link. 2. Write the key commendation or recommendation for each component of the Assessment Report on the table we provide. Updated Assessment Action Plan: -Using Assessment Data -Closing the loop 10

Sacramento State Baccalaureate Learning Goals 1. Competency in the Disciplines (to know) 2. Knowledge of Human Cultures and the Physical and Natural World (GE) (to know) 3. Intellectual and Practical Skills (to know and to do) 4. Personal and Social Responsibility (to value) 5. Integrative Learning (to do) 54

Graduate Learning Goals Competency in the Discipline: Disciplinary Knowledge: Master, integrate, and apply disciplinary knowledge and skills to current, practical, and important contexts and situations. Intellectual and Practical Skills: Communication: Communicate key knowledge with clarity and purpose both within the discipline and in broader contexts. Critical Thinking/Analysis: Demonstrate the ability to be creative, analytical, and critical thinkers. Information Literacy: Demonstrate the ability to obtain, assess, and analyze information from a myriad of sources. Research (Doctoral Programs) Personal and Social Responsibilities: Professionalism: Demonstrate an understanding of professional integrity. Intercultural/Global Perspectives: Demonstrate relevant knowledge and application of intercultural and/or global perspectives. 55

AAC&U and VALUE Rubrics 3. 1 Critical Thinking (WSCUC core competency) 3. 2 Information Literacy (WSCUC core competency) 3. 3 Written Communication (WSCUC core competency) 3. 4 Oral Communication (WSCUC core competency) 3. 5 Quantitative Literacy (WSCUC core competency) 3. 6 Inquiry and Analysis (Sixth VALUE rubric) 3. 7 Creative Thinking (Seventh VALUE rubric) 3. 8 Reading (Eighth VALUE rubric) 3. 9 Teamwork (Ninth VALUE rubric) 3. 10 Problem Solving (Tenth VALUE rubric) 56

Appendices 8 -9 Baccalaureate Learning Goals, Graduate Learning Goals, and AAC&U VALUE Rubrics (Detailed) 1. Baccalaureate Learning Goals (BLGs) (Appendix 8) 2. Graduate Learning Goals (GLGs) (Appendix 9) 3. AAC&U (Associations of American Colleges and Universities) VALUE Rubrics 1. 1 Competence in the Disciplines 2. 1 Disciplinary Knowledge 3. 1 Competence in the Disciplines 1. 2 Knowledge of Human Cultures and the Physical and Natural World (GE) 1. 3 Intellectual and Practical Skills 1. 3. 1 Communication 3. 2 Knowledge of Human Cultures and the Physical and Natural World 2. 3 Intellectual and Practical Skills 2. 3. 1 Communication 1. 3. 1 a Written 1. 3. 1 b Oral 1. 3. 2 Critical Thinking 1. 3. 3 Literacy 1. 3. 3 a Information 1. 3. 3 b Quantitative 1. 3. 4 Inquiry and Analysis 1. 3. 5 Creative Thinking 1. 3. 6 Reading 1. 3. 7 Teamwork 1. 3. 8 Problem Solving 1. 4 Personal and Social Responsibility (Values) 1. 4. 1 Global Learning 1. 4. 2 Intercultural Knowledge and Competence 1. 4. 3 Civic Knowledge and Engagement 1. 4. 4 Ethical Reasoning 1. 4. 5 Foundations and Skills for Lifelong Learning 2. 3. 2 Critical thinking/analysis 2. 3. 3 Information literacy 2. 4 Personal and Social Responsibility (Values) 2. 4. 1 Intercultural/global perspectives 3. 3 Intellectual and Practical Skills 3. 3. 1 Communication 3. 3. 1 a Written 3. 3. 1 b Oral 3. 3. 2 Critical Thinking 3. 3. 3 Literacy 3. 3. 3 a Information 3. 3. 3 b Quantitative 3. 3. 4 Inquiry and Analysis 3. 3. 5 Creative Thinking 3. 3. 6 Reading 3. 3. 7 Teamwork 3. 3. 8 Problem Solving 3. 4 Personal and Social Responsibility (Values) 3. 4. 1 Global Learning 3. 4. 2 Intercultural Knowledge and Competence 3. 4. 3 Civic Knowledge and Engagement 3. 4. 4 Ethical Reasoning 3. 4. 5 Foundations and Skills for Lifelong Learning 2. 4. 6 Professionalism 1. 5 Integrative Learning 2. 6 Research 3. 5 Integrative Learning

Baccalaureate Learning Goals, Graduate Learning Goals, AAC&U VALUE Rubrics, and DQP (Simple) Refer to the table below to know the relationship between the BLGs and GLGs and how they both relate to the AAC&U VALUE rubrics and DQP. 1. Baccalaureate Learning Goals 2. Graduate Learning Goals (BLGs) (Appendix 8) (GLGs) (Appendix 9) 3. AAC&U (Associations of American Colleges and Universities) VALUE Rubrics 4. DQP (Degree Qualification Profile: Learning Goals for Associate, Bachelor, and Master Degrees) 1. 1 Competence in the Disciplines 3. 1 Competence in the Disciplines 4. 1 Specialized Knowledge 2. 1 Disciplinary Knowledge 1. 2 Knowledge of Human Cultures and the Physical and Natural World (GE) 3. 2 Knowledge of Human Cultures 4. 2 Broad and Integrative and the Physical and Natural Knowledge World 1. 3 Intellectual and Practical Skills 2. 3 Intellectual and Practical Skills 3. 3 Intellectual and Practical Skills 1. 4 Personal and Social Responsibility (Values) 2. 4 Personal and Social Responsibility (Values) 3. 4 Personal and Social Responsibility (Values) 1. 5 Integrative Learning 2. 5 Integrative Learning 3. 5 Integrative Learning 2. 6 Research 4. 3 Intellectual Skills 4. 5 Applied and Collaborative Learning 4. 6 Civic and Global Learning

Baccalaureate Learning Goals, Graduate Learning Goals, AAC&U VALUE Rubrics, DQP, and Bloom’s Taxonomy (Simple) 1. Baccalaureate Learning Goals (BLGs) (Appendix 8) 2. Graduate Learning Goals (GLGs) (Appendix 9) 3. AAC&U (Associations of American Colleges and Universities) VALUE Rubrics 4. DQP (Degree Qualification Profile: Learning Goals for Associate, Bachelor, and Master Degrees) 5. Bloom’s Revised Taxonomy 1. 1 Competence in the Disciplines 2. 1 Disciplinary Knowledge 3. 1 Competence in the Disciplines 4. 1 Specialized Knowledge 5. 1 Bloom’s Revised Taxonomy 3. 2 Knowledge of Human Cultures and the Physical and Natural World 4. 2 Broad and Integrative Knowledge 5. 2 Bloom’s Revised Taxonomy 4. 3 Intellectual Skills 5. 3 AAC&U VALUE Rubrics (Valid Assessment of Learning in Undergraduate Education) 1. 2 Knowledge of Human Cultures and the Physical and Natural World (GE) 1. 3 Intellectual and Practical Skills 2. 3 Intellectual and Practical Skills 3. 3 Intellectual and Practical Skills 1. 4 Personal and Social Responsibility (Values) 2. 4 Personal and Social Responsibility (Values) 3. 4 Personal and Social Responsibility (Values) 1. 5 Integrative Learning 2. 5 Integrative Learning 3. 5 Integrative Learning 2. 6 Research 5. 4 AAC&U VALUE Rubrics 4. 5 Applied and Collaborative Learning 5. 5 AAC&U VALUE Rubrics 4. 6 Civic and Global Learning 5. 6 Civic and Global Learning

Baccalaureate Learning Goals, Graduate Learning Goals, AAC&U VALUE Rubrics, and DQP (Detailed) Refer to the table below to know the relationship between the BLGs and GLGs and how they both relate to the AAC&U VALUE rubrics and DQP. 1. Baccalaureate Learning Goals (BLGs) (Appendix 8) 2. Graduate Learning Goals (GLGs) (Appendix 9) 3. AAC&U (Associations of American Colleges and Universities) VALUE Rubrics 4. DQP (Degree Qualification Profile: Learning Goals for Associate, Bachelor, and Master Degrees) 1. 1 Competence in the Disciplines 2. 1 Disciplinary Knowledge 3. 1 Competence in the Disciplines 4. 1 Specialized Knowledge 3. 2 Knowledge of Human Cultures and the Physical and Natural World 4. 2 Broad and Integrative Knowledge 3. 3 Intellectual and Practical Skills 3. 3. 1 Communication (Written and Oral) 3. 3. 2 Critical Thinking 3. 3. 3 Literacy 3. 3. 3 a Information 3. 3. 3 b Quantitative 3. 3. 4 Inquiry and Analysis 3. 3. 5 Creative Thinking 3. 3. 6 Reading 3. 3. 7 Teamwork 3. 3. 8 Problem Solving 4. 3 Intellectual Skills 1. 2 Knowledge of Human Cultures and the Physical and Natural World (GE) 1. 3 Intellectual and Practical Skills 1. 3. 1 Communication (Written and Oral) 1. 3. 2 Critical Thinking 1. 3. 3 Literacy 1. 3. 3 a Information 1. 3. 3 b Quantitative 1. 3. 4 Inquiry and Analysis 1. 3. 5 Creative Thinking 1. 3. 6 Reading 1. 3. 7 Teamwork 1. 3. 8 Problem Solving 2. 3 Intellectual and Practical Skills 2. 3. 1 Communication 2. 3. 2 Critical thinking/analysis 2. 3. 3 Information literacy 1. 4 Personal and Social Responsibility 2. 4 Personal and Social Responsibility (Values) 1. 4. 1 Global Learning 2. 4. 1 Intercultural/global perspectives 1. 4. 2 Intercultural Knowledge and Competence 1. 4. 3 Civic Knowledge and Engagement 1. 4. 4 Ethical Reasoning 1. 4. 5 Foundations and Skills for Lifelong Learning 2. 4. 6 Professionalism 3. 4 Personal and Social Responsibility (Values) 3. 4. 1 Global Learning 3. 4. 2 Intercultural Knowledge and Competence 3. 4. 3 Civic Knowledge and Engagement 3. 4. 4 Ethical Reasoning 3. 4. 5 Foundations and Skills for Lifelong Learning 1. 5 Integrative Learning 3. 5 Integrative Learning 2. 6 Research 4. 5 Applied and Collaborative Learning 4. 6 Civic and Global Learning

Appendix 3: WSCUC “Rubric for Assessing the Quality of Academic Program Learning Outcomes” http: //www. wascsenior. org/search/site/Rubrics%20 combined Criterion Initial Emerging Developed Highly Developed The list of outcomes is problematic: e. g. , very The list includes reasonable outcomes The list is a well-organized set of reasonable The list is reasonable, appropriate, and 1. incomplete, overly detailed, inappropriate, and but does not specify expectations for outcomes that focus on the key knowledge, skills, comprehensive, with clear distinctions between Comprehensive disorganized. It may include only disciplinethe program as a whole. Relevant and values students learn in the program. It undergraduate and graduate expectations, if List 2. Assessable Outcomes 3. Alignment specific learning, ignoring relevant institutionwide learning. The list may confuse learning processes (e. g. , doing an internship) with learning outcomes (e. g. , application of theory to real-world problems). institution-wide learning outcomes and/or national disciplinary standards may be ignored. Distinctions between expectations for undergraduate and graduate programs may be unclear. includes relevant institution-wide outcomes (e. g. , communication or critical thinking skills). Outcomes are appropriate for the level (undergraduate vs. graduate); national disciplinary standards have been considered. applicable. National disciplinary standards have been considered. Faculty has agreed on explicit criteria for assessing students’ level of mastery of each outcome. Outcomes statements do not identify what students can do to demonstrate learning. “Statements understand scientific method” do not specify how understanding can be demonstrated and assessed. There is no clear relationship between the outcomes and the curriculum that students experience. Most of the outcomes indicate how students can demonstrate their learning. Each outcome describes how students can demonstrate learning, e. g. , “Graduates can write reports in APA style” or “Graduate can make original contributions to biological knowledge. ” Students appear to be given reasonable opportunities to develop the outcomes in the required curriculum. The curriculum is designed to provide opportunities for students to learn and to develop increasing sophistication with respect to each outcome. This design may be summarized in a curriculum map. The program has a reasonable, multi-year assessment plan that identifies when each outcome will be assessed. The plan may explicitly include analysis and implementation of improvements. Outcomes describe how students can demonstrate their learning. Faculty has agreed on explicit criteria statements such as rubrics, and have identified examples of student performance at varying levels of each outcome. Pedagogy, grading, the curriculum, relevant student support services, and co- curriculum are explicitly and intentionally aligned with each outcome. Curriculum map indicates increasing levels of proficiency. The program has a fully-articulated, sustainable, multi-year assessment plan that describes when and how each outcome will be assessed and how improvements based on findings will be implemented. The plan is routinely examined and revised, as needed. Students are well-acquainted with program outcomes and may participate in creation and use of rubrics. They are skilled at self-assessing in relation to the outcome levels of performance. Program policy calls for inclusion of outcomes in all course syllabi, and they are readily available in other program documents. 4. Assessment There is no formal plan for assessing each outcome. Planning 5. The Students know little or nothing about the overall outcomes of the program. Experience The program relies on short-term planning, such as selecting which outcome(s) to assess in current year. Students have some knowledge of program outcomes. Communication is Communication of outcomes to students, e. g. , occasional and informal, left to in syllabi or catalog, is spotty or nonexistent. individual faculty or advisors. Students have a good grasp of program outcomes. They may use them to guide their own learning. Outcomes are included in most syllabi and are readily available in the catalog, on the web page, and elsewhere.

We have used the AAC&U VALUE rubric to organize the previous slide.

Question one! Let’s look at each of the components of a good assessment report! 63

Critical Thinking Information Literacy Written Communication Oral Communication Quantitative Literacy Inquiry and Analysis Creative Thinking Reading Teamwork Problem Solving Plan step 1 Select PLOs/Competencies PLOs: -Select 64

Q 1. 1. Which of the following Program Learning Outcomes (PLOs) … did you assess? [Check all that apply] • • Critical Thinking (WASC) (GLG) • Information Literacy (WASC) (GLG) • • Written Communication (WASC) (GLG) • • Oral Communication (WASC) (GLG) • • Quantitative Literacy (WASC) • • Inquiry and Analysis (GLG) • • Creative Thinking • • Reading • • Team Work • • Problem Solving • Civic Knowledge and Engagement • Intercultural Knowledge, Competency, and Perspectives (GLG) Ethical Reasoning Foundations and Skills for Lifelong Learning Global Learning and Perspectives (GLG) Integrative and Applied Learning Overall Competencies for GE Knowledge Overall Disciplinary Knowledge (GLG) Professionalism (GLG) Other Check here if your program has not collected any data for any PLOs. 9

Assessment Principles Assessment—an ongoing, interactive process using backward design and five basic principles: “Backward Design” for Assessment (Outcomes & Definitions) Q 2. 1, Q 21. 1 Q 2. 3 Q 2. 2. a. Methods and Measures -Assignments -Tests -Projects Q 3. 3. 2 Results -Data Tables -Findings -Conclusions Q 4. 1 -Q 4. 3 66 Report step 2 Expectations 2: -Define Standard of Performance - Design Instruction Help Students Achieve Outcomes Report step 1 Decide on Assessments - Evidence of Learning Outcomes Plan step 4 Expectations 1: -Develop or adopt Rubrics Plan step 3 PLOs: -Select -Define -Curriculum map Plan step 2 Plan step 1 Define Learning Goals - Outcomes & Definitions - Curriculum Map Updated Assessment Action Plan: -Using Assessment Data -Closing the loop Q 20. 2

Revised Bloom’s Taxonomy Table (2001) Cognitive Dimension Knowledge Dimension A. Factual Knowledge Aa. of terminology Ab. of specific details and elements B. Conceptual Knowledge Ba. of classifications and categories Bb. of principles and generalizations Bc. of theories, models, and structures C. Procedural Knowledge Ca. of subject-specific skills and algorithms Cb. of subject-specific techniques and methods Cc. of criteria for determining when to use appropriate procedures D. Metacognitive Knowledge Da. Strategic knowledge Db. Knowledge about cognitive tasks, including appropriate contextual and conditional knowledge Dc. Self-knowledge 1. Remember 1. 1 Recognizing 1. 2 Recalling 2. Understand 3. Apply 4. Analyze 5. Evaluate 2. 1 Interpreting 3. 1 Executing 4. 1 Differentiating 5. 1 Checking 2. 2 Exemplifying 3. 2 Implementing 4. 2 Organizing 5. 2 Critiquing 2. 3 Classifying 4. 3 Attributing 2. 4 Summarizing 2. 5 Inferring 2. 6 Comparing 2. 7 Explaining 6. Create 6. 1 Generating 6. 2 Planning 6. 3 Producing

Assignment 2 1. 1: Can you use verbs to define what is critical thinking for your field of study and then place them in the Bloom’s table? 1. 2: What kind of gap do you see in the teaching, learning, and assessment in the program?

PLO 6: Critical Thinking in the Sociology Graduate Program • 6: Sociology graduate students will demonstrate a habit of systematically exploring social issues, ideas, artifacts, and events before accepting or formulating an opinion or conclusion” (Learning Goal/Outcomes): they will (PLO 6: Critical thinking adopted from the VALUE rubric): • 6. 1: Clearly identify and state the social issue/problem that needs to be considered critically, comprehensively describe the social issue/problem, and deliver all relevant information so it is necessary for a full understanding of the issue/problem (Explanation of issues); • 6. 2: Thoroughly interpret and evaluate sociological perspectives, theories, social methods/statistics, and any other current, credible, and relevant information (research, knowledge, tools, and/or views) to complete a thorough review of relevant literature and the problems to develop a comprehensive analysis or synthesis (Evidence); • 6. 3: Thoroughly analyze their own and others' assumptions and carefully evaluate the relevance of contexts when presenting a position (Influence of context and assumptions); • 6. 4: Consider the complexities (all sides) of a social issue. Limits of position and others' points of view are acknowledged and synthesized within position (Student's position); • 6. 5: Form conclusions, consequences and implications that are logical and reflect student’s informed evaluation and ability to place evidence and perspectives discussed in priority order (Conclusions and related outcomes).

Revised Bloom’s Taxonomy Table (2001) Cognitive Dimension 1. Remember 1. 1 Recognizing 1. 2 Recalling Knowledge Dimension 2. Understand 3. Apply 4. Analyze 5. Evaluate 2. 1 Interpreting 3. 1 Executing 4. 1 Differentiating 5. 1 Checking 2. 2 Exemplifying 3. 2 Implementing 4. 2 Organizing 5. 2 Critiquing 2. 3 Classifying 4. 3 Attributing 2. 4 Summarizing 2. 5 Inferring 2. 6 Comparing 2. 7 Explaining 6. Create 6. 1 Generating 6. 2 Planning 6. 3 Producing A. Factual Knowledge Aa. of terminology Ab. of specific details and elements 6. 1 6. 2 6. 5 B. Conceptual Knowledge Ba. of classifications and categories Bb. of principles and generalizations Bc. of theories, models, and structures 6. 1 6. 2 6. 5 6. 1 6. 3/6. 4 6. 5 C. Procedural Knowledge Ca. of subject-specific skills and algorithms Cb. of subject-specific techniques and methods Cc. of criteria for determining when to use appropriate procedures D. Metacognitive Knowledge Da. Strategic knowledge Db. Knowledge about cognitive tasks, including appropriate contextual and conditional knowledge Dc. Self-knowledge

An Example: The Curriculum Map for the Sociology Graduate Program: Aligning (Linking) Graduate Program Learning Outcomes to Each Course in the Curriculum 1 “I” stands for “Introduced”, “D” for “Developed”, and “M” for “Mastered”. Outcome 1: Outcome 2: Outco Advanced Sociological mes Sociological Contributions Knowledge and Classes Applications Required Classes SOC 200 A I 1 I (Introduction) SOC 200 B (Thesis D, M Prep. ) SOC 214 (Methods) D SOC 215 (Analysis) D SOC 235 (Soc. D D Psych. ) SOC 240 (Theory) D D SOC 500/599 (Thesis M M Work) Elective Classes SOC 210 (Urban Sociology) SOC 220 (Change) SOC 225 (Stratification) SOC 226 (Gender) SOC 230 (Social Org. ) SOC 238 (Environmental) SOC 225 (Crime) SOC 260 (Contem. Issues. ) SOC 265 (Race) SOC 266 (Family) Outcome 3: Outcome 4: Outcome 5: Challenges in Oral Written Sociology Communicati on on Outcome 6: Critical Thinking Outcome 7: Outcome 8: Intercultural Integration: Knowledge & Inquiry and Competence Analysis I I I D, M D, M D I, D I, D I, D D M I, D M I, D I, D I, D I, D I, D I, D I, D D I I, D I, D I, D I, D I, D I, D D I, D

Results -Data Tables -Findings -Conclusions Updated Assessment Action Plan: -Using Assessment Data -Closing the loop Report step 2 Methods and Measures -Assignments -Tests -Projects Report step 1 -Select -Define -Curriculum map Expectations 1: -Develop or Adopt Rubrics 2. Define Standards of Performance Plan step 3 PLOs: Plan step 2 Plan step 1 Review 2017 -2018 Assessment Report Feedback Activity (link) 10