How measuring the nonacademic outcomes of schooling can

How measuring the non-academic outcomes of schooling can help guide school practice Prue Anderson Julian Fraillon ACER Research Conference Perth 2009

non-academic outcomes Successful learners who are able to plan activities independently, collaborate, work in teams and communicate ideas, Confident and creative individuals who have a sense of optimism about their lives and the future Active and informed citizens who act with moral and ethical integrity Melbourne Declaration, 2008

non-academic outcomes Is there such a thing?

Research: Somers School Camp • 10 -day camp for grades 5 & 6 • adventure activities, environmental education and outdoor recreation • students live and work in ‘hut groups’ with students they do not know • aim of camp is to develop students’ independence, confidence and self-esteem and to promote co-operation, communication and tolerance

What does research tell us? Prevailing view • outdoor education programs inherently build character Literature review findings • A general evangelical complacence underlies much of the research into the outcomes of outdoor education programs • Common criticism of poorly constructed, un -standardised measurement instruments

Counter view • Outdoor education programs provide situations that elicit behaviour, such as an unadventurous person willingly trying new activities, but this is a result of the context rather than indicative of any significant change in the person. • for programs to have lasting impact, people need to be able to reflect on their experiences in the broader context of their lives Andrew Brooks La. Trobe University

This process of measuring non-academic outcomes can help guide school practice 1. Defining the outcome 2. Creating opportunities for students to demonstrate the outcome 3. Measuring and improving

Defining the outcome: What are you measuring?

Somers Camp outcomes

Overlapping ideas confuse teaching & assessment Challenge Self Esteem Will move out of comfort zone Is a risk taker Is prepared to take a risk Is a risk taker Will cope with change Copes with failure/ doesn’t always need approval Confident in decision making Is self confident Involved in decision making Demonstrates independence / prepared to participate in a leadership role Sets own goals Demonstrates independence / doesn’t always need approval

Clearly defined outcomes lead to focussed teaching and assessment Self Others Getting to Independence know yourself Happiness and others Humour Honesty Fairness Getting along Friendship Trustworthiness Striving to achieve Understanding Consideration Inclusion Do your best Initiative Persistence Resilience Courage

Defining the outcome: Is more better?

How do students typically grow? • Likert scales administered in WA to measure students’ self-esteem, motivation to achieve potential, positive emotional management, responsibility for self e. g. I usually know how to behave properly. • Y 3 highest mean score • Y 9 lowest mean score For self reports and teacher judgments.

Defining the outcome: What do you want to achieve? Different contexts may have different optimum points of balance Co-operation in class task Co-operation with friends in yard

Defining the outcome: What do you want to achieve? The same context may require different points of balance on different outcomes Co-operation in class task Optimism in class task

Defining the outcome: What do you want to achieve? We suggest: Incremental growth occurs in the increasing sophistication and depth of understanding of self and context that allows students to make good choices about how to behave. Cognitive reflective skills are critical in cultivating non-academic outcomes.

Defining the outcome: What do you want to achieve? Which outcome do you want to cultivate? What does it mean for students to develop in this outcome?

Non-academic outcomes 1. Defining the outcome 2. Creating opportunities for students to demonstrate the outcome 3. Measuring and improving the outcome

Somers Camp example Outcome Striving to achieve: motivation, setting high goals, persistence High ropes course • physically challenging, students set own goals • strong encouragement to ‘have a go’ • high level of teacher support • high chance of success

Somers Camp example Self: Striving to achieve • Assumption that students underestimate what they can do • Students need highly supported, challenges to raise their expectations of what they can achieve • After Somers Camp students will be more confident and set higher goals

prediction orientation

Non-academic outcomes 1. Defining the outcome 2. Creating opportunities for students to demonstrate the outcome 3. Measuring and improving the outcome

reflect on prediction reflect on strategies used to get through course reflect on support provided generalise about learning

apply learning to new situations set higher goals have strategies for managing new challenge

Student responses Reflection on the task • mainly restatement of given strategies • good understanding of support provided • few students able link high ropes activity to personal goals • students generally showed little understanding of how to set personal goals

Improving the outcome RETHINK: what do we want students to learn? • activity is WOW factor • motivation is persisting when things are boring/hard/lonely not thrill seeking • what are better opportunities for students to learn? • how can students be helped to link camp experiences to their everyday lives? • what do students need to be taught about motivational strategies and personal goal setting?

Creating opportunities Is the task really focussed on the outcomes you want to address? Is there enough challenge? Classroom group work • low-bar for routine work • high-bar challenge in a specialised context

Creating opportunities Specialised contexts may be third party • films • novels • issues in English Complex social situations analysed so students • interpret and evaluate how others behave • imagine and explain how they might behave • explore alternative resolutions

How does April manage her feelings? A. She blames herself and thinks she is totally incompetent. B. She tries to blame others for putting her in a horrible position. C. She cannot bear to think about her feelings and wants them to go away. D. She is interested in learning to understand manage her feelings better.

Measuring the outcome: What are you measuring? Social skills assessments • self-affirming? • students reiterating given values? • insightful and revealing? Team work skills • co-operation or task completion? • honesty or tolerance?

Measuring the outcome: What instruments? Our experience: • focussed, probing reflective questions work – show if students are learning what you hope – spread students – show what students need to learn next

Non-academic outcomes 1. Defining the outcome 2. Creating opportunities for students to demonstrate the outcome 3. Measuring and improving the outcome

Somers Camp example Outcome Tolerance of others: Understanding, consideration Inclusion, making friends Opportunities to learn: whole camp

Examples of reflective questions • What was the nicest thing someone you did not know did for you? Why did you like them doing this for you? • What did you mainly do to be friendly to people you did not know (select ‘often’, ‘sometimes’, or ‘hardly ever’) – I noticed if someone was left out. – I tried to meet new people. – I asked people I did not know to do things with me. – I helped shy people feel comfortable. – I did things with different people. • Hut teacher observation: This is a fair self-evaluation yes/no

Examples of reflective questions • What do you usually do to get along with other people? (select ‘often’, ‘sometimes’, or ‘hardly ever’) – I listen carefully to what people say. – I have good ideas. – I make people laugh. – I get everyone to agree about what to do. – I fit in with everyone else. • Are you pleased with how friendly you have been at Somers Camp or do you think you could have been more friendly? Explain • What would you like to change about yourself so you get on better with other people? What would help you make this change?

Example of marking scheme Are you pleased with how friendly you have been at Somers Camp or do you think you could have been more friendly? Explain. 2 pt Evaluates and justifies friendliness. (Level 4) Yes, I tried really hard and I did meet lots of new people. No, I could have tried harder, but I was too shy. Yes, I have been friendly to other people because I help them. I could have been more friendly because I want to make new friends. Yes, because I have met new friends and they like me. 1 pt Evaluates friendliness with no justification. (Level 3) Yes, I was friendly. I could be friendlier. I’m pleased with how friendly I have been. 0 pt vague or irrelevant Everyone was friendly.

Level Interpersonal scale Students show some insight into the factors that 5 contribute to building relationships and team cohesion. 25% They frequently notice and include others who are left out or shy. 4 52% Students identify and use positive cooperative strategies. Students rate their own cooperative skills highly, showing limited understanding of how they might improve. 3 18% 2 4% Students have high expectations of getting along and willingly cooperate, but tend to use negative strategies of threats and guilt. Students do their basic share of team tasks with little sense of responsibility for the overall working of the group.

Somers Camp findings • 75% of students demonstrated a limited capacity to reflect on their behaviour in groups, the dynamics of their interactions with others or to empathise with others in any substantial way • these students require extensive teacher support to reflect more deeply on the challenges the camp offered in building relationships with unknown others, the way they responded to these challenges and how they might continue to build these skills.

Conclusion • experiential learning is not enough to develop non-academic outcomes Students need • a solid foundation of knowledge, skills and language • opportunities to make informed reflections • support to transfer their understandings to new contexts

This process of measuring non-academic outcomes can help guide school practice 1. Defining the outcome 2. Creating opportunities for students to demonstrate the outcome 3. Measuring and improving the outcome