Accountability and College and Career Readiness CCR Theory

Accountability and College and Career Readiness (CCR): Theory and Practice Patrick Kyllonen Director, Center for Academic and Workforce Readiness and Success Educational Testing Service Next Generation Accountability Learning Community (NGALC) explores ESSA from the perspective of small-to-midsize urban districts. (MASSINC) July 28, 2016, Boston, MA 10/21/2021

Outline • ESSA • Which skills are important for College and Career Readiness? • Best approaches for measuring character skills today • Conclusions Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 2

ESSA “goal of fully preparing all students for success in college and careers” ‘‘(v)(I) For all public schools in the State, not less than one indicator of school quality or student success that— ‘‘(aa) allows for meaningful differentiation in school performance; ‘‘(bb) is valid, reliable, comparable, and state-wide (with the same indicator or indicators used for each grade span); and may include one or more of the <the following> measures • student engagement; • educator engagement; • student access to and completion of advanced coursework; • postsecondary readiness; • school climate and safety; and • any other indicator the State chooses that meets the requirements of this clause SOURCES: http: //www 2. ed. gov/policy/elsec/leg/essa/index. html? src=essa-resources 3

Importance of Education • Education benefits individuals and society • More education = more economic growth, less unemployment, better jobs, less crime, higher civic engagement • What is it about education that produces these benefits? • Cognitive skills—mathematics, language, problem solving—but cognitive skills are only part of the benefit • Character skills—punctuality, work ethic, getting along with others, impulse control, listening to the teacher may be as important or more important

Which Skills are Important? For post-secondary readiness 5

“What attributes do you look for on a candidate's resume? ” National Association of Colleges and Employers (NACE) 2015 Survey (260 Employers) • leadership (78%) • work in a team (78%) • written communication skills (73%) Boldface are character skills • problem solving (71%) • strong work ethic (70%) • analytic/quantitative (68%) • technical skills (68%) • verbal communication (67%) • initiative (67%) • computer skills (63%) • flexibility/adaptability (62%) • interpersonal (61%) • detail-oriented (58%) • organizational (42%) https: //www. naceweb. org/about-us/press/class-2015 -skills-qualities -employers-want. aspx

Employers Rating Skill as “Very Important” Oral Communications Teamwork/Collaboration Professionalism/Work Ethic Written Communications Critical Thinking/Problem Solving English Language Ethics/Social Responsibility Leadership Information Technology Creativity/Innovation Lifelong Learning/Self direction Diversity Mathematics Science College graduates 95% 94 94 93 92 88 86 82 81 81 78 72 64 33 High School Graduates 70% 75 80 53 58 62 63 29 53 36 43 52 30 9 Casner-Lotto, J. , Barrington, L. , & Wright, M. (2006) Are they really ready to work? Conference Board, 7 Corporate Voices for Working Families, Partnership for 21 st Century Skills, Society for Human Resource Management

Which Skills are Important? • Correlational Study (Meta-analysis) • Found that Conscientiousness (achievement striving, organization) & Openness (e. g. , intellectual curiosity) predicted educational outcomes from primary to tertiary education as strongly as cognitive ability does, especially when measured by teacher ratings • Arthur Poropat (2009; 2014) • Longitudinal Study • “Misbehavior” ratings* in 8 th grade (9000 8 th graders; 2 teachers’ ratings) predicted • Lower educational attainment (controlling for cognitive scores & socioeconomic status) • Lower earnings at age 26 -27 (controlling for cognitive scores & educational attainment) • Carmit Segal (2013), based on NELS: 88 * Misbehavior = “Yes” to any of 5 questions: “rarely completes homework, ” “frequently absent, ” “frequently tardy, ” “consistently inattentive in class” or “frequently disruptive”)

Background • Many recent proposals for the measurement of noncognitive skills in children and adolescents • • 9 OECD (Skills for Social Progress) NCES (NAEP Questionnaires) California Office to Reform Education (CORE) National Academy of Sciences (4 committees/reports) District/national assessments in Brazil, other countries CASEL’s SECA working group Popular literature (Tough, Duckworth, Dweck, Pink) 10/ 21/ 20

Illinois State Board of Education, Illinois Learning Standards, Social/Emotional Learning (SEL) Section 15(a) of Public Act 93 -0495 http: //www. isbe. net/ils/social_emotional/standards. htm Goal 1 - Develop selfawareness and selfmanagement skills to achieve school and life success Goal 2 - Use socialawareness and interpersonal skills to establish and maintain positive relationships Goal 3 - Demonstrate decision-making skills and responsible behaviors in personal, school, and community contexts Identify & manage one’s emotions and behavior Recognize personal qualities and external supports Demonstrate skills related to achieving personal and academic goals Recognize the feelings and perspectives of others Recognize individual and group similarities and differences Use communication and social skills to interact effectively with others Consider ethical, safety, and societal factors in making decisions Apply decision-making skills to deal responsibly with daily academic and social situations Contribute to the well-being of one’s school and 10 community

OECD Skills Framework Contextual factors: Schools, Family, Community & Workplace Basic cognitive ability • Pattern recognition • Processing speed • Memory Knowledgeacquired • Access • Extract • Interpret Knowledgeextrapolated • Reflect • Reason • Conceptualise Cognitive • Mental capacity to acquire knowledge, thoughts and experience • Interpret, reflect and extrapolate based on the knowledge acquired Social and Emotional Achieving goals • Perseverance • Self-control • Passion for goals Individual capacities that (a) are manifested in consistent patterns of thoughts, feelings and behaviors, (b) can be developed through formal and informal learning experiences, and (c) influence important socioeconomic outcomes throughout the individual’s life. Outcomes: Education, workforce, civic, safety, health & well being, environment Working with others • Sociability • Respect • Caring Managing emotions • Self-esteem • Optimism • Confidence

OECD’s Skills for Social Progress Policy messages • Children need a well-balanced set of cognitive, social, & emotional skills (SES) for life success and societal progress • Children’s capacities to achieve goals, work effectively with others, & manage emotions help improve lifetime outcomes • SES can be raised by improving learning environments & mobilizing intervention programs • “Skills beget skills” and early investment in SES is key to improving life prospects • Regular SES assessments provide valuable information to improve learning contexts and ensure they are conducive to skill development • Policy makers acknowledge the importance of SES, but the level of policies & programs varies 12

Best approaches for measuring CCR skills today In primary and secondary school, college, & the workforce 13

Current best approaches to Measurement • Self ratings (Likert scale) • Most common data collection method • New methods (forced-choice, anchoring vignettes) are promising solutions to known self-ratings problems • Others’ ratings (Teacher, peer assessments) • Provides information that supplements students self evaluations • Situational judgment • Incorporate context, are more “authentic” • Can be used as formative assessments • Not as reliable as ratings

Example: The Mission Skills Assessment Longitudinal Assessment Six character skills Three approaches self ratings teacher ratings supplemental method (e. g. , situational judgment)

self ratings [(a) never/rarely (b) sometimes (c) often (d) usually/always] I am good at helping people work well together. I prefer for other students to take responsibility when I am working in a group. (-) teacher ratings [(a) never/rarely (b) sometimes (c) often (d) usually/always] x leads others effectively x enjoys helping team members. x likes team activities situational judgment test You have been working hard with other students on a group project. A team member gives you some badly written work to include in the project. There is very little time before the project is due. What would you do? a) Suggest ways that the student can improve the work b) Don't worry about it, just put the bad work in c) Tell the teacher about the situation d) Rewrite the work yourself team

self ratings [(a) never/rarely (b) sometimes (c) often (d) usually/always] I remain calm under pressure. I am easily discouraged. I determine what will happen in my life. teacher ratings [(a) never/rarely (b) sometimes (c) often (d) usually/always] x overcomes challenges and setbacks x does not give up easily x is resilient situational judgment test You are feeling stressed about the amount of homework that you have been given by your teacher(s). Below are some ways that you might think, feel, or act in this situation, right at the time that you feel stressed-out. Rate how often you do each activity when you feel stressed. How do you think, feel, or act when you are stressed from having too much homework to do? a) I try to get organized to get on top of my homework. b) I blame my teacher(s) for giving me too much homework in the first place. c) I go out with my friends.

How Do Students, Teachers, & Schools Use this Information? • Students—to know themselves (e. g. , compare with teacher ratings, other “objective” measures) • Teachers—coaching, providing guidance on how to improve, • Schools—determine whether students are progressing adequately • Compare their schools/systems against other schools/systems • Intervene to improve skills growth • Everyone • Common language • Signals the importance of skills otherwise implicit

Conclusions • Regarding CCR, schools do more than teach students mathematics and language, they develop other skills —work ethic, teamwork, communication • These other skills—character skills, social-emotional skills—are demanded by educators & employers • Character skills predict success in school & in the workplace; they are as important as cognitive skills • There has been an explosion of interest in character skills in the last 10 years • We have much better measurement than we had just a few years ago, and this will continue to improve • Today the best approaches for measuring CCR are: • Self reports, particularly “forced-choice” • Teacher reports • Performance measures, such as situational judgment tests • In the future, additional measures