Use QA for Panel discussion Use Chat for

Use Q&A for: Panel discussion Use Chat for: Technology support #aacu. VALUE Slides and recording will be posted online: www. aacu. org/webinar/emerging-landscape-learning-value-initiative

Webinar Speakers Moderator C. Edward Watson Terrel Rhodes Chief Information Officer and Associate Vice President for Quality, Pedagogy, and LEAP Initiatives, Association of American Colleges and Universities (AAC&U) watson@aacu. org Vice President for Quality, Curriculum, and Assessment and Executive Director of VALUE, Association of American Colleges and Universities (AAC&U) rhodes@aacu. org John D. Hathcoat Associate Professor of Graduate Psychology/ Associate Director of University Learning Outcomes Assessment, James Madison University hathcojd@jmu. edu Bonnie Orcutt Professor of Economics, Worcester State University borcutt@worcester. edu Sarah Strout Assistant Vice President for Assessment and Planning, Worcester State University sstrout@worcester. edu

Terrel Rhodes Vice President for Quality, Curriculum and Assessment and Executive Director of VALUE, AAC&U rhodes@aacu. org

The Emerging Landscape of Learning Foundations • VALUE Approach to Assessment, Learning, and Pedagogy • VALUE Institute – starting with data/information that can address questions about quality of learning • Uses of the data to focus on how to improve student understanding of Essential Learning Outcomes • Uses of data for accountability • Uses of data for supporting teaching and learning

VALUE Approach—Assumptions § Learning is a complex process that occurs over time § Student work is representation of student motivated learning § Focus on what students do in terms of key dimensions of learning outcomes—learning is not a single number § Faculty and educator expert judgment § Results satisfy accountability demands § Results are useful and actionable for improvement of learning—by faculty educators, students, programs, institutions

VALUE Approach to Assessment • VALUE rubrics answer the need for measuring the development and application of the Essential Learning Outcomes that college graduates need to be productive in work and in citizenship over time • The VALUE rubrics are available for download—at no cost—on the AAC&U website (https: //www. aacu. org/value).

Sample Data Display—University of X, Overall Results

Sample Data Display—College Y, Evidence vs. No Evidence

Sample Data Display—Community College Z, Pell Eligibility

John D. Hathcoat Associate Professor of Graduate Psychology/Associate Director of University Learning Outcomes Assessment, James Madison University hathcojd@jmu. edu

Two Validity Questions • Do the numbers that we assign to student learning mean what we want them to mean? • What influences the way that we want to use the scores?

Properties of Numbers in Measurement Type Nominal Property Identity Example Ordinal Order Individuals are ranked according to their finishing time in a marathon. Interval Equal intervals and arbitrary 0 Fahrenheit; Celsius Ratio True 0 Kelvin temperature; Distance traveled A number 1 is assigned to students who vote and a 2 is assigned to those who did not vote.

I love numbers. 1 2 Strongly disagree Disagree Score = 1 3 Neutral Score = 3 4 Agree 5 Strongly agree Score = 5

What about VALUE?

“Scorers…assign a ‘zero’ score if the work product does not show evidence of any of the four levels of proficiency for the dimension in question” (On Solid Ground 2017, p. 9). Said differently, a zero is assigned when there is an absence of evidence in the work of a particular rubric element.

Do we maintain the property of order? 0 Absence of evidence Absence of evidence

Assignment vs. 0 Absence of evidence Absence of evidence Student

Assignment A Assignment B Assignment C 3 4 2 3

Assignment A Assignment B Assignment C 3 4 2 Can we predict these scores?

A Quick Summary 1. Ensure that a zero score is a feature of the student as opposed to the assignment before analyzing data. 2. Assignment alignment with each rubric element is extremely important. 3. Assignments should provide an opportunity for the student to demonstrate mastery.

Bonnie Orcutt Professor of Economics, Worcester State University borcutt@worcester. edu

The Importance of the Assignment Prompt for Assessment, Alignment, Equity, and Pedagogy The assignment prompt is central to the assessment of authentic, classroom-based student work and central to the advancement of equity. Alignment: Ensure alignment of the assignment prompt with the learning outcome(s), the grading rubric or the evaluative criteria, and the assessment rubric. Annotate: Annotation is a powerful tool for ensuring alignment in the assignment design and redesign process.

The Assignment Prompt and Classroom Pedagogical Approaches Focus on assignment design and alignment leads to reflections about and changes in classroom pedagogical approaches Learning Outcome: Critical Thinking LEARNING OUTCOME CT DIMENSION: INFLUENCE OF CONTEXT AND ASSUMPTIONS EMBEDDED WITHIN A COURSE LEARNING OUTCOME EXPLICITLY ADDRESSED IN THE ASSIGNMENT PROMPT EXPLICITY ADDRESSED IN THE GRADING RUBRIC OR EVALUATIVE CRITERIA CLASSROOM TEACHING APPROACH AND STUDENT LEARNING OPPORTUNITIES

Using the Grading and Assessment Results for Improvement Did your students perform at levels below or above expectations based upon assessment results and grades? What questions might we ask? Ø Did the assignment prompt explicitly ask students to demonstrate that learning outcome dimension in language accessible to the student? Ø Did your evaluative criteria clearly align with the learning outcome dimension? Ø Did students have earlier opportunity(ies) to practice and demonstrate that learning? Ø Is your pedagogical approach effective for advancing student acquisition of those skills?

Sarah Strout Assistant Vice President for Assessment and Planning, Worcester State University sstrout@worcester. edu

Using the Results of the VALUE Institute • Institutional level assessment is difficult – But, required for many states and accreditors • Many institutions use indirect survey results (i. e. NSSE) • Direct, course-embedded assessments are best practice – VALUE Institute allows this as well as benchmarking

How WSU uses the VALUE Results • Individual course level • Department level (first year vs fourth year) • University level – Yearly – Longitudinal

Written Communication Results Overall

Department Written Communication Results by Credit

Department Written Communication Results Pre-Post

C. Edward Watson Chief Information Officer and Vice President for Quality, Pedagogy, and LEAP Initiatives, AAC&U watson@aacu. org

Curricular and Pedagogical Change • Instructional change and faculty development are long-term formative processes (Hall, 1979; Watson, 2008) • Faculty attitudes, knowledge, and classroom practice have been significantly changed by multi-contact strategies using follow up techniques (Herr, 1998; Long, Sadker, & Sadker, 1986) • Build a foundation for curricular and pedagogical change upon what we know about how students learn and what we know about the success of current practice.

Triangulation of Evidence for Change • Educational research • Scholarship of Teaching and Learning • Assessment activities and results (evidence of student performance) • Mid-semester formative evaluations • Peer review of teaching • Peer mentoring • Pedagogical consulting • End of semester student evaluation data

Approaches to Foster Instructional Change • Employ adult learning practices (establish relevancy, life-centered approach, experiential, self-directed) • Offer incentives (grants, equipment, awards, status, release time, stipends, travel funds/opportunities, GA support) • Provide ongoing support • Integrate peers • Offer well-designed, structured, ongoing programming options, such as…

Faculty Learning Communities / Fellows Programs Set clear goals and expected outcomes Can be structured at the program or department level Can be cross disciplinary Should employ adult learning practices, offer incentives, provide ongoing support, and take place over time • Faculty should be able to articulate rationale for the changes they plan to employ • Consider a closing event where effort is recognized and changes are shared • •

Terrel Rhodes Vice President for Quality, Curriculum and Assessment and Executive Director of VALUE, AAC&U rhodes@aacu. org

Contact and Information Registration Open: https: //www. aacu. org/VALUEInstitute More information and registration: https: //www. aacu. org/VALUEInstitute Email: valueinstitute@aacu. org

Use Q&A for: Panel discussion Use Chat for: Technology support #aacu. VALUE Slides and recording will be posted online: www. aacu. org/webinar/emerging-landscape-learning-value-initiative

Webinar Speakers Moderator C. Edward Watson Terrel Rhodes Chief Information Officer and Associate Vice President for Quality, Pedagogy, and LEAP Initiatives, Association of American Colleges and Universities (AAC&U) watson@aacu. org Vice President for Quality, Curriculum, and Assessment and Executive Director of VALUE, Association of American Colleges and Universities (AAC&U) rhodes@aacu. org John D. Hathcoat Associate Professor of Graduate Psychology/ Associate Director of University Learning Outcomes Assessment, James Madison University hathcojd@jmu. edu Bonnie Orcutt Professor of Economics, Worcester State University borcutt@worcester. edu Sarah Strout Assistant Vice President for Assessment and Planning, Worcester State University sstrout@worcester. edu

Forthcoming AAC&U Webinars Thursday, November 14 1: 30 -2: 30 p. m. Fostering Civic Consciousness through Creativity-Based Pedagogy Wednesday, December 11 2: 00 -3: 00 p. m. STEM Matters: The Formula for Driving Innovation