Entering Student Assessment A Process Not an Event

Entering Student Assessment: A Process, Not an Event Dr. D. Patrick Saxon Sam Houston State University Associate Professor and Director Doctoral Program in Developmental Education Administration Email: patrick. saxon@shsu. edu References and other materials are posted at: www. developmentaled. com Developmental Education Summit Pennsylvania October 22, 2012

Question How many of us entered developmental education by chance (or serendipity)? Copyright, 2012, D. Patrick Saxon

Good Developmental Education is… A comprehensive program of remedial courses and support services guided by the principles of adult learning and development and highly integrated through communication, coordination, and collaboration. Copyright, 2012, D. Patrick Saxon

Coordination Jenkins (2006) found that the most successful colleges had well aligned and coordinated student and academic support services. “Seamless integration of services from the student’s perspective seems to contribute most to student success. ” Copyright, 2012, D. Patrick Saxon

Support for Mandatory Placement Based on Assessment (Boylan, 2002; Boylan, Bliss, & Bonham, 1997; Casazza & Silverman, 1996; Maxwell, 1997; Mc. Cabe, 2000; Roueche & Roueche, 1999) Each of these scholars has supported mandatory placement. Their work has shown that it contributes to: � higher retention rates, � higher pass rates in developmental English and math, and � higher academic standards at colleges that implement these policies. Copyright, 2012, D. Patrick Saxon

Sorting Students Hundreds, if not thousands of students are coming and in a short period of time you must make a high stakes decision about their lives Copyright, 2012, D. Patrick Saxon

Challenges to Accurate Assessment � Safran & Visher (2010): Most students don’t understand the tests’ purpose or high-stakes nature. � Venezia, Bracco, & Nodine (2010): Students were unprepared, confused about placement policies, and many never received advising. � Scott-Clayton (2012): Our use of assessment instruments lead us to over-diagnose students to developmental courses. Copyright, 2012, D. Patrick Saxon

Why is using a single test score a bad idea? � Standard Error of Measurement (SEM) – All tests have one. � Therefore, you receive “shotgun” accuracy rather than “bullseye” accuracy. what you want what you get Copyright, 2012, D. Patrick Saxon

SEM and the Political Season Clayton Rainey Co. Presidential Polling Candidate A Candidate B 47% 45% Margin of Error ± 3 *Statistical Tie Copyright, 2012, D. Patrick Saxon

What About Our Students? Example: Cut score is 50 and SEM is ± 3 What about a student with 47? (real SEM score may be 44 -50) What about a student with 51? (real SEM score may be 48 -54) Both Students Could Place Either Way Given the SEM Copyright, 2012, D. Patrick Saxon

Consider Using Cut Score Ranges Hypothetical Example: Test Score Course 0 -10 ENG 90 Dev. Writing 1 11 -15 Gray Area 16 -25 ENG 95 Dev. Writing 2 26 -30 Gray Area 31+ ENG 101 College Comp. Copyright, 2012, D. Patrick Saxon

Assessment Must Become a Process Whereby We: inform, teach, assist, administer, collect and interpret data IN ORDER TO make informed, caring decisions about students. Copyright, 2012, D. Patrick Saxon

Components Needed � Well-publicized opportunities to prepare � Staff that are helpful with preparation, policies, and facilities � An inventory of course and support service options � Well-trained and adequately staffed advising programs � Student profiles based on cognitive, affective, and personal characteristics � Systematic formative evaluation Copyright, 2012, D. Patrick Saxon

In The Beginning… � Assessment preparation needs coordination with marketing, registration, and orientation. � Students must understand the purpose and meaning for assessment testing. � Strongly encourage (if not require) preparation and practice. The goal is to get every student in a preparation opportunity prior to the assessment. Copyright, 2012, D. Patrick Saxon

Anecdotal Evidence �Large community college in Texas reports an increase of one course level for students that participate in preparation for the test. (Anecdotal – but suggests promise and an area for experimentation and research) Copyright, 2012, D. Patrick Saxon

Examples http: //www. oaklandcc. edu/mathtest/ Oakland CC offers online information and practice tests http: //www. highland. edu/students/documents/ Placementtestworkshops. pdf Highland CC takes a stronger approach – just short of requiring preparation Copyright, 2012, D. Patrick Saxon

HALFTIME! Stand up, stretch, and tell someone Good morning! Copyright, 2012, D. Patrick Saxon

How many of your colleges offer practice sessions for assessment exams? If you don’t – then do! If you do: � Find out how many students use it. � Research effectiveness. � Refine the process. If you don’t have high participation levels, seek to boost those numbers. Copyright, 2012, D. Patrick Saxon

Wyckoff (1999) reported that the primary negative characteristic linked to student attrition was inadequate academic advising. Astin (1993) found ad What do you think students would have to say about your advising service? Advising �You need an adequate number of advising staff at peak times �Accessible student data �Profiles developed to assist with decision making �Advisors must be well-trained Copyright, 2012, D. Patrick Saxon

What would students say about your advising service? � Wyckoff (1999) found the primary variable linked to student attrition was inadequate academic advising. � Astin (1993) found advising ranked as lowest quality of services by students. � Jenkins (2006) and Ender, Winston, and Miller (1984) found similar problems with advising services. Copyright, 2012, D. Patrick Saxon

Boylan’s T. I. D. E. S. Model Copyright, 2012, D. Patrick Saxon

Boylan’s T. I. D. E. S. Model Copyright, 2012, D. Patrick Saxon

“Janice” 18 -year old African-American student Recent HS graduate with 3. 2 GPA Cognitive assessment results: Missed placing into college algebra by 5 points Affective assessment results: Highly motivated, seeks help willingly, and applies effective study strategies Personal information: Works 10 hours a week, has no children, lives at home with parents Source: Boylan, 2009 Copyright, 2012, D. Patrick Saxon

Prescription for Janice Past success, motivation, and time available for college study indicates that: Janice could enroll in college algebra and be successful if she participated in regular tutoring. Source: Boylan, 2009 Copyright, 2012, D. Patrick Saxon

“Roberto” 26 -year-old Hispanic student Just released from active military service Cognitive assessment results: Missed the cut score for placement into college algebra by five points Affective assessment results: Highly motivated and well disciplined Personal information: Married, has one child, and works 20 hours a week Has been out of high school for 8 years Source: Boylan, 2009 Copyright, 2012, D. Patrick Saxon

Prescription for Roberto Because he is well-disciplined and motivated, but has been out of HS eight years: He may succeed in an “accelerated” online developmental math course. A T. I. D. E. S. advisor might check Roberto’s high school algebra grades to further inform the placement decision. Source: Boylan, 2009 Copyright, 2012, D. Patrick Saxon

The advisor… �Is not a passive reporter of test scores. �Is a well-informed decision maker. �Is seeking to place students in courses where they’ll have the highest probability of success. Copyright, 2012, D. Patrick Saxon

Multiple Measures -High School GPA “High school GPA is an extremely good and consistent predictor of college performance, and it appears to encapsulate all the predictive power of a full high school transcript in explaining college outcomes. ” (Belfield & Crosta, 2012) May be a proxy for student performance and affective attributes But perhaps its usefulness declines over time Copyright, 2012, D. Patrick Saxon

Multiple Measures � Types and number of HS courses taken � High school rank � Background Data: Years since HS, jobs held, financial aid situation � Extracurricular activities: As responsibilities increase – perhaps does the need for dev. ed. � Age: Older students may be more cautious, but motivated. � Student self- choice: With caution when data are confusing, contradictory, or inconclusive. Students may tend to overestimate their proficiencies. Source: Morante Copyright, 2012, D. Patrick Saxon

William Sedlacek and the NCQ Noncognitive Questionnaire Scales � Positive Self Concept � Realistic Self Appraisal � Ability to Handle Racism � Prefers Long Range Goals to Immediate Needs � Availability of a Support Person � Leadership Experience � Demonstrated Community Service � Knowledge Acquired in a Field Source: Sedlacek, 2004 Copyright, 2012, D. Patrick Saxon

Student Readiness Inventory Scales � � � � � Academic Discipline Academic Self-Confidence Commitment to College Communication Skills Emotional Control General Determination Goal Striving Social Activity Social Connection Study Skills Source: Robbins, Allen, Casillas, Peterson& Le, 2006 Copyright, 2012, D. Patrick Saxon

Listing of Student Readiness Assessments Published in Research in Developmental Education by Levine-Brown, Bonham, Saxon, & Boylan. Linked on Developmental. Ed. com Copyright, 2012, D. Patrick Saxon

Other Tips for Assessment � Give a printout or electronic output of the data and placement results � Assist students in finding the right balance between apathy and anxiety � Develop a communication checklist for advisors. � Keep track of the variables considered and their weight in the placement decision � Over time, use these data to refine the process Copyright, 2012, D. Patrick Saxon

Assessment Trends � 13 states and 17 college systems have policies regarding test instrument and cut scores � 3 states mandate regular review of instruments and cut scores � 4 states mandate multiple measures Copyright, 2012, D. Patrick Saxon

Assessment Trends � FL- diagnostic exam linked to modularized courses and state remedial education competencies � VA and NC - diagnostic exam tailored to modularized courses � 4 other states are considering or moving in this direction � California may align tests with Common Core standards. Source: ECS, 2012 Copyright, 2012, D. Patrick Saxon

DEBATE ISSUE Test Scores and Predicting Student Success “Placement test scores are not especially good predictors of course grades in developmental education classes. ” (Belfield & Crosta, 2012, p. 39) Should They? You Decide. Copyright, 2012, D. Patrick Saxon

Another View �Assessment tests are designed to estimate student skills relative to a point where they CANNOT succeed in a given course. �Course grades are a function of many variables such as skills, attitude, dedication, performance, attendance, instructor grading, teaching practices, etc. Source: Gordon (2006) Copyright, 2012, D. Patrick Saxon

Gordon (2006) “Assessment relates only to students’ residual basic skills. We don’t measure dedication, maturity, the quality of tires on the car or the health of the kids, we measure basic skills. ” “While we should be able to predict a failing grade from assessment, we cannot and we should not attempt to, predict a passing grade. ” Copyright, 2012, D. Patrick Saxon

In Closing “Students, by and large, view their matriculation process as a one-shot deal — an isolated event that happens one day with minimal to no advance information. ” “Yet the assessment and placement process involves very high stakes for students…” Venezia, Bracco, & Nodine, 2010 Copyright, 2012, D. Patrick Saxon

Changing the Status Quo Multi-dimensional and highly integrated assessment processes will aid students in accessing course/support options that offer them the highest probability of success Good luck with your efforts to improve assessment! Copyright, 2012, D. Patrick Saxon

Thank You!