Pilot Study of the CAAP Critical Thinking Test

Pilot Study of the CAAP Critical Thinking Test April 27, 2005 Lanette Raymond Research Associate, Suffolk County Community College

CAAP Test Description z 32 item multiple choice test z Designed for use with college students z Administered within a single class meeting, z Appeared relatively credible in an in-class administration protocol z Provided documentation of reliability and validity across community college populations

CAAP Subscore Customization z CAAP Critical Thinking test contents closely match the SUNY CT learning objectives z Variance in the way the results are reported z ACT developed a customized report for these subscores showing normative comparisons against ACT national community college data. z ACT provided the student data files to SCCC, for further analysis of this data

Administration and Sample z Fall, 2004 z Administered in-class to 154 SCCC students in 7 general education courses z Predominately white (77%) z Traditional age (60% 20 years old or younger, 25% between 21 and 25 years old) z 50% male, 50% female z Mostly sophomore status (46%) z Fulltime enrollment (85%)

Student Motivation z No motivational tactics were employed z CAAP-CT instrument included an item that addressed students' self-reported motivation levels z One-third of students (n = 52) did not respond to the motivation item z 5 students indicated that they “gave no effort” (n = 1) or “gave little effort” (n = 4) to the assessment test.

Student Motivation z Lower motivation results in less optimal performance z Less motivated students’ scores are less reliable and less valid z Reporting sample is based on data from the 97 students who reported moderate to best effort z The reliability coefficient (calculated with the data from the original 154 tests) for objective 1 (26 items) is within acceptable range (alpha =. 80) z Due to the small number of items (6 items) contributing to objective 2, its reliability coefficient is much lower (alpha =. 49).

Results z Confirmatory factor analysis substantiates the utility of the CAAP-CT test as a measure of 2 separate but related sets of critical thinking skills based on the 2 GEAR learning objectives

2 -factor model of Critical Thinking based on the GEAR Objectives

Results z All of the items loaded well onto their respective factors, with item 1 being only slightly below 1. 96 (at 1. 81). z The model shows an excellent fit to the data (χ2(463) = 466, p =. 46, CFI =. 94), providing additional context validity to the assessment.

** Data for item 20 should be listed under objective 2

Standards Does not meet standard Approaches standard Meets standard Exceeds standard 59% or less 60% - 69% 70% - 79% 80% or more

Standards