Linking the conceptual framework to assessment and evaluation

Linking the conceptual framework to assessment and evaluation Kathe Rasch Maryville University St. Louis, MO

Acknowledgements n This session is informed by the courageous and pioneering work of Mary Diez and Alverno College n Her willingness to share with all of us (including some of the ideas presented today) continue to inspire us to examine possibilities

Wanted: A clear picture of the teacher n The conceptual framework – Provides a picture of the knowledge, skills, attitudes, values and dispositions of the teacher/advanced professional you aim to develop – Needs to be clear to your faculty, students and partners

The conceptual framework n Can be tested against research, consensus documents like NBPTS propositions and standards, INTASC principles, and state standards n Encompasses what is most important to institutional mission n Articulates fundamental beliefs

The conceptual framework n Is the basis for the design of assessment n Provides the framework for program evaluation

Assessment and Evaluation n Assessment is concerned with “knowing students”, their abilities, knowledge, skills, etc. n Evaluation is about “knowing courses/programs/degrees, ”etc. n Thus, assessment data can inform evaluation studies, not the other way around. – William Scott, University of Bath

How does the conceptual framework help to devise an assessment system? n Knowledge, skills, dispositions manifest themselves in behaviors n What matters most becomes clear n Development of teachers related to the framework takes time n The tenets of the framework cut across the programs

And what the framework lays out……. n NOTE: Knowledge, skills and dispositions are: – Complex – Developmental, i. e. , able to be extended throughout life – Relevant in multiple settings

Assessing this complexity n Begin with the complexity n What are some of the most important abilities for your program? – Are they reflected in the conceptual framework? – How do you think that you develop them now? – What would you like to be able to do?

For example, a central ability n For example, you can use INTASC or your program outcomes to identify a large ability n The ability to – Apply principles of human development and learning in the design of instruction – Use reflection to guide practice

How can such an ability be developed and assessed across the program? n Where do you currently start building an understanding of human development? n How do you assess it? How might you want to assess it?

How can an ability be developed across the courses/experiences of a program? n How do you build the ability to apply that understanding? n How do you assess it? How might you want to assess it?

How can an ability be developed across the courses/experiences of a program? n What should prospective teachers/counselors/principals be able to do regarding this ability when they are ready to be licensed? n How do you assess it? How might you want to assess it?

One example of such a progression (Alverno and Maryville) n Beginning life-span development course – “Take a Learner to Lunch” (analysis of interview with 4 -year old in relationship to Piaget and Erikson) Field Experience logs Observations about groups of learners based upon expected physical, emotional, intellectual characteristics for age

One example of a progression n Methods courses – A case study on a student with reading/language delays Student teaching Work sampling that includes instructional decisions and assessment data for learners at different developmental levels

Conceptual Framework to Assessment System (Diez) n The difference between an assignment and an assessment is in the larger relationships implied in the latter n Etymological definition of assessment: “To sit down beside” n An assessment system requires, over time, successively more complex demonstrations of the target abilities.

Assessment as learning n Public, explicit outcomes tied to the conceptual framework n Prompts/assignments/tasks that allow for the differentiation of performance n Mutually agreed upon criteria (tied to past experience and standards) that describe the expected performance

Assessment as learning and measures of performance n Feedback by a range of assessors, who have a chance also to share their feedback n Self-assessment that develops and measures the learner’s knowledge of the skill and the performance

For Example n Conceptual Framework stresses the importance in the school developing democratic citizens n Long-term task: Candidates asked to assess the differences in access to knowledge and resources in varying school districts in the county and their assessment of educational opportunity

What does this assessment reveal? n Ability to gather information n Values and beliefs about resources n Knowledge of the community n Ability to synthesize the collective information gathered n Willingness to take risks and take a stand

How might this get a range of performances? n Multiple ways that a student can participate n Complexity of evaluation n No clear right/wrong answers

What criteria might describe this performance? Shows the ability to gather and analyze data n Shows how candidates seek and use current information n Demonstrates candidates ability to pool information and work together n Demonstrates candidates understanding of important tenets of democracy n

About dispositions n We have no trouble identifying students who we believe have highly acceptable or highly unacceptable dispositions n It’s those in the middle…. n We can describe behaviors that manifest themselves that might suggest certain dispositions.

About dispositions n We have to identify what they are and where they appear in the conceptual framework n Students must be able to self-assess

And on to a system n How do we make it flow? n How is it usable for both candidates and program improvement? n How can we keep it true to the conceptual framework and serve the external systems of accountability? n How do we acknowledge individual candidate variations?

And on to a system How do we hone our ability to portray candidates and program development over time? n How do we ensure multiple measure and different modalities for students to model their proficiency. n And what about rigor? n What are the key decision points in the system? n

Whom do we involve? n Internal and external constituencies Candidates Faculty Partner school faculty District partners Arts and Sciences faculty

Onto fairness and consistency n Multiple measure requires multiple evaluators and a long term commitment to developing assessments n If they don’t understand the framework in their own terms, it is hard for them to assess n Role of technology

The program should build: n What do we know about assessing the knowledge, skills and dispositions that emerge from the framework – At the beginning – In development – At program exit – In light of P-12 student learning – Across time

Sharing and debating n In program development n In program implementation n As the candidate performance becomes more complex across time

Putting the system into place (Fullan) n Change is a blueprint, not a journey n You can’t mandate what matters n Problems are our friends

Should this help with program n ABSOLUTELY!!!! n If you don’t think about program evaluation at the same time, the efforts will be stymied.

OK, so I have to go home…. . n You are developing a conceptual framework – Don’t assume that your current assessments are not valuable – Use what you have – Work to make it (and the results) more public, explicit and consistent across the program

n Don’t be afraid to look at the real gaps n Look at progression with regard to rigor and complexity n Use your assessment data to look at how your candidate perform on the goals/ outcomes you have set at different levels of the program

Choose new assessments and the system carefully n Choose decision and data points carefully n Think about utility, feasibility, rigor and renews opportunities in designing your system

Resources to help n AACTE: the PETE project n INTASC: The INTASC Academies (July 2003 at Alverno—see www. ccsso. org n NBPTS n Alverno: The Assessment Workshop (June 2003 at Alverno www. alverno. edu The Renaissance Group

Final Thoughts n The conceptual framework is every institution’s most valued and guarded intellectual property n This journey must be for you…not because NCATE made you do it n Your faculty care about producing good teachers