Agenda Introduction Benchmarks Benchmarking Survey data and benchmarking

Benchmarks Benchmark: H Used to establish “industry standards” based on external and internal comparisons H Comparisons to similar institutions establish benchmarks H CCSSE measures what students are doing

Benchmarks An example from the corporate world An interview with Jeff Immelt, CEO of General Electric, identified Toyota, Dell, and Procter & Gamble as the three companies that GE has benchmarked the most GE looks at Toyota and Dell to learn from their process excellence GE looks at Procter & Gamble to learn from their marketing and commercial excellence

Benchmarks Opportunities abound in educational research Corporate benchmarking tends to be sharing between partners Corporate benchmarking limited to a few competitors in contrast to education that has a much larger universe with which to compare Are these approaches really that different? Should institutions attempt to find similar or different partners for benchmarking?

Benchmarks What are the most important characteristics of a benchmark?

Benchmarks Elements Credibility—reliability and validity Comparative—results examined relative to peers Comprehensive—measures key elements according to the experts Performance & Importance—what has the greatest impact Confidentiality—it takes courage to assess oneself Continuous—it takes time for improvement -Joseph A. Pica, CEO of Educational Benchmarking Inc. in About Campus

Benchmarks The CCSSE survey: H is administered directly to community college students at CCSSE member colleges in randomly selected classes. H is based on research, asking questions about institutional practices and student behaviors demonstrated to promote student learning and retention. H uses a sampling methodology that is consistent across all participating colleges.

Benchmarks The five CCSSE benchmarks: Active and Collaborative Learning Student-Faculty Interaction Academic Challenge Support for Learners Student Effort

Benchmarks Active and Collaborative Learning: Worked with other students on projects during class Worked with classmates outside of class to prepare class assignments Tutored or taught other students (paid or voluntary) Participated in a community-based project as a part of a regular course Made a class presentation Asked questions in class or contributed to class discussions. Discussed ideas from your readings or classes with others outside of class (students, family members, coworkers, etc. )

Criterion Benchmarking How do you determine which measure you use to compare yourself with other institutions?

Benchmarking Normative Compare your college with the mean Criterion Compare your college with a predetermined value

Benchmarking Normative Benchmarks Provide context Determine what the mean you would like to be compared with is Normative Benchmarks Situate Your Results What does it mean to have 80% of your students satisfied? A good place to start, but not necessarily the end point

Benchmarking Normative Benchmarking with CCSSE • Look for Differences of 5 points (a standardized effect size of. 2) • Is. 2 a noteworthy difference?

Criterion Benchmarking with CCSSE • What is the college mission? • What are the college’s accreditation goals? • Are all students equally engaged?

Benchmarking Five ways to colleges can reach for excellence using CCSSE Benchmarks: Compare themselves to national average Compare themselves to high-performing colleges Measure their performance against their least-engaged group Gauge work in areas most strongly valued Compare now to where they want to be

Benchmarking Comparisons yourself with high-performers

Benchmarking Measure performance against least-engaged group H Breakout by race, gender, enrollment status, parental education, traditional vs. non-traditional age H At risk students vs. other students H Define at-risk at your college

Benchmarking Density Curves of Student v. Institutions benchmarks (within institution v. between institution variation)

Benchmarking Gauge work in areas most strongly valued H Focus On Current College Initiatives H Sharing thoughts about how to use CCSSE data to evaluate different programs H Examine institutional mission, vision, and values

Benchmarking Understand values by sharing results Share results with others to determine what is most strongly valued Faculty, students, and administrators will likely have different opinions on what it is that accounts for particular phenomena Hopefully, the questions that are created from benchmarks are more focused questions than the original question

Benchmarking Compare now to where you want to be

Survey Data as Benchmarks Should benchmarks derived from surveys be used to rank colleges? Are there fundamental differences in how census benchmarks (e. g. , graduation rates) and benchmarks derived from surveys should be used?

Survey Data as Benchmarks Ewell’s distinguishes between ‘hard’ statistics and ‘second-order’ statistics ‘Hard’ statistics are clearly enumerated and based on census-type data, such as numbers of students, graduates, and degrees awarded ‘Second-order’ statistics measure phenomena that cannot be directly counted, such as student satisfaction and students’ self-assessments of their behavior, and as such contain some statistical instability ‘Hard’ statistics are preferable for performance funding because they are more statistically stable than ‘secondorder’ statistics Source: Ewell, P. T. (1999). Linking performance measures to resource allocation: Exploring unmapped terrain. Quality in Higher Education, 5 (3), 191209.

Survey Data as Benchmarks Texas Community Colleges Performance Measures: 1. The rate at which students completed courses attempted. 2. The number and types of degrees and certificates awarded. 3. The percentage of graduates who passed licensing exams related to the degree or certificate awarded, to the extent the information can be determined. 4. The number of students or graduates who transfer to or are admitted to a public university. 5. The passing rates for students required to be tested under the Section 51. 306. 6. The percentage of students enrolled who are academically disadvantaged. 7. The percentage of students enrolled who are economically disadvantaged. 8. The racial and ethnic composition of the district’s student body. 9. The percentage of students contact hours taught by full-time faculty. Source: http: //www. thecb. state. tx. us/reports/DOC/1197. DOC

Survey Data as Benchmarks Texas State-Level Benchmarks for Higher Education: • Percent of recent high school graduates enrolled in a Texas public college or university • Percent of first-time, full-time freshmen returning after one academic year • Percent of first-time, full-time freshmen who graduate within four years • Percent of first-time, full-time freshmen who graduate within six years • Percent of two-year college students who transfer to four-year institutions • Percent of two-year transfer students who graduate from fouryear institutions • Percent of population age 24 and older with vocational/technical certificate as highest level of educational attainment • Percent of population age 24 and older with two-year college degree as highest level of educational attainment Source: http: //www. thecb. state. tx. us/reports/DOC/1197. DOC

Survey Data as Benchmarks If performance funding is based on verifiable hard statistics, what role does survey data have in benchmarking? Or, why should we concern ourselves with the student experience? Input and outcome versus process measures To achieve outcomes, we need to understand the process by which they are obtained

Survey Data as Benchmarks “We can tell people almost anything about education except how well students are learning. ” Patrick M. Callan, President, National Center for Public Policy and Higher Education

Survey Data as Benchmarks Input -> Process -> Outcome Model Inputs include costs, numbers admitted, etc. Outputs include graduation rates, retention, graduate satisfaction How do we measure the Process component?

Survey Data as Benchmarks Input and Ranking Inputs are heavily emphasized in media rankings and potentially serve to maintain an establishment Inputs and outputs are naturally correlated The challenge for institutions is to maximize process to improve on the ability of inputs to predict outputs

Survey Data as Benchmarks Input -> Process -> Outcome Model Inputs include costs, numbers admitted, etc. Outputs include graduation rates, retention, graduate satisfaction How do we measure the Process component?

Survey Data as Benchmarks Is there a danger of impacting results by raising the stakes? Increasing the outcome without increasing the process To achieve outcomes, we need to understand the process by which they are obtained Increasing the outcome may not reflect improvement, but increasing the process won’t hurt

Survey Data as Benchmarks CCSSE does not rank There is not a single criteria or set of criteria that can be used universally Institutional characteristics matter Institutional missions differ Benchmarking with CCSSE data is best when presented in a non-threatening manner Improvement requires an understanding of the process Understanding the process in an institution will require hearing different voices and different perspectives on the same issues

Survey Data as Benchmarks Stability of CCSSE Benchmarks Correlate 2005 and 2006 benchmarks for colleges that participated both years 45 institutions 55, 903 students

Survey Data as Benchmarks Student Effort 2006 Student. Faculty Interaction 2005 Student. Faculty Interaction 2006 Support for Learners 2005 Support for Learners 2006 Variable Academic Challenge 2006 Active & Collaborative Learning 2005 Active & Collaborative Learning 2006 Academic Challenge 2005 0. 80 0. 55 0. 44 0. 65 0. 56 0. 59 0. 36 0. 24 0. 10 0. 49 0. 58 0. 56 0. 67 0. 53 0. 34 0. 21 0. 78 0. 55 0. 44 0. 30 0. 15 0. 16 0. 47 0. 61 0. 49 0. 61 0. 28 0. 27 0. 74 0. 39 0. 29 0. 53 0. 46 0. 44 0. 42 0. 49 0. 48 0. 77 0. 47 0. 33 0. 52 0. 41 Academic Challenge 2006 Active & Collaborative Learning 2005 Active & Collaborative Learning 2006 Student Effort 2005 Student Effort 2006 Student-Faculty Interaction 2005 Student-Faculty Interaction 2006 Support for Learners 2005 Student Effort 2005 0. 90

Summary Survey results are ‘second order’ data not ideal for performance funding and Understanding ‘hard’ statistics naturally leads to a discussion of processes. Survey data present an opportunity to understand processes and impact hard, outcome measures As such, survey data presents opportunities for nonthreatening discussions of