Creating a Teacher Education Assessment System David C

Creating a Teacher Education Assessment System David C. Smith, Dean Emeritus College of Education University of Florida Email: smithdc@aol. com David C. Smith Creating a Teacher Education Assessment System

Why We Need an Assessment System We need to do a better job of assessing what we are doing and how it can be improved. Our assessment systems need to accommodate a paradigm shift from assessing teaching to assessing learning. We need to move from collecting data to implementing an assessment system. David C. Smith Creating a Teacher Education Assessment System 2

We need to be seen as the sources of the best information available about the preparation of professional educators. Each SCDE needs an assessment system. One assessment system should be capable of addressing multiple demands for data. An effective assessment system can help to increase confidence in the preparation of professional educators. David C. Smith Creating a Teacher Education Assessment System 3

There are Multiple Demands for Information Internal - advocacy, analysis, and planning. l Making a case for resources. How do you compare with other units in your institution? l Analyzing units within your SCDE. l How do you compare with “significant others”? l l l Who are “significant others”? Annual Reports. David C. Smith Creating a Teacher Education Assessment System 4

External - Accreditation and evaluation. Accreditation - NCATE and regional. l Reports to the State Department of Education. l Reports to the State Higher Education Board. l Title II information. l David C. Smith Creating a Teacher Education Assessment System 5

The pressure is very real, and it is a high stakes environment. David C. Smith Creating a Teacher Education Assessment System 6

Considerations in Building an Assessment System. The baseline is very important. That may become your standard. (Neither easy or trivial. High tide and low tide. ) Historical data are important insofar as they lead to an understanding of your current needs and direction. (The Past is Prologue. National Archives. ) Collect information that is likely to be valuable and relevant avoid the temptation to collect data which are not meaningful but readily available. l l Trivial information may be readily available. Have a reason for each item in your assessment system. David C. Smith Creating a Teacher Education Assessment System 7

Further Considerations. Consider your audience. (Bob and Gene show. ) A system that is not current is not of much value. (It must be kept up to date. ) It is imperative that your data be accurate. (The price is lost credibility. ) You get big numbers by adding little numbers. (Institutional research. ) Collect data elements at least one level lower than the unit. (College down 10%. ) David C. Smith Creating a Teacher Education Assessment System 8

Considerations. - (Still more. ) Data are inextricably related to resource allocation and the demonstration of productivity. (Without data – what did you do with what you got? Summer budget allocation. ) The first line of defense is to question the accuracy of the data or to ignore dissonance producing data. (Use mine until you have better numbers. ) Try to use data that are difficult for others to reject. (I got the information from your office. ) David C. Smith Creating a Teacher Education Assessment System 9

Considerations. - (Even more. ) Avoid asking for data which require calculations. (Avoid introducing errors. ) Precise measures are better than crude measures; crude measures are better than approximations; and approximations are better than no data at all. (Important variables may be difficult to measure. ) You can make better approximations for your unit than anyone else. (Yo 0 u need not apologize for reasonable approximations. ) David C. Smith Creating a Teacher Education Assessment System 10

Considerations. - (Still more. ) Do not expect sharp changes over a short time interval. (You can’t drive an 18 -wheeler like a sports car. Complex organizations change slowly. ) An assessment system faces conflicting demands; the ability to compare and the need to tailor. (Data from comparable institutions are powerful. Yet your system must fit your needs. ) The system must be capable of easy modification. The assessment system must be capable of supporting a rapid response. David C. Smith Creating a Teacher Education Assessment System 11

Considerations. - (Even more. ) People will attend to those things that are measured, rather than those things which are said to be important. (Achievement tests. ) It doesn’t make sense to think you need to measure with a micrometer if you are going to mark with a piece of chalk and cut with an axe. ( If it is important. Quantitative economics. ) Use data to make your problem important to others. (Abolish elementary education. ) David C. Smith Creating a Teacher Education Assessment System 12

Principles. (Enough – at last. ) A single system is much more powerful and useful than multiple data sources. Information is power. You need to be the best informed persons on the campus with respect to matters relating to the preparation of professional educators. (An assessment system can help you accomplish that goal. ) David C. Smith Creating a Teacher Education Assessment System 13

Data can help us get our act together. David C. Smith Creating a Teacher Education Assessment System 14

The institution almost assuredly collects data. Ø Those data signal what the institution Ø Ø values. The institution can hardly question the accuracy of the data it collects and reports. Those data can be studied and interpreted. What do these data suggest about your SCDE? Some of these data can be imported into your assessment system. David C. Smith Creating a Teacher Education Assessment System 15

What do we mean by the term “assessment system”? A carefully designed and functional system for collecting, organizing, maintaining, analyzing and utilizing meaningful information. A system is made up of sub-systems. To be effective, the system must be incorporated into the regular operation of the SCDE. David C. Smith Creating a Teacher Education Assessment System 16

A Performance Assessment System involves utilizing information to support vital activities of the SCDE. Assessment David C. Smith Analysis Creating a Teacher Education Assessment System Implementation 17

What kinds of data might be included in a Teacher Education Performance Assessment System©? (Establishing a conceptual framework. ) Resource Data Productivity Data Candidate Performance Data Faculty Performance Data David C. Smith Creating a Teacher Education Assessment System 18

Structure of a Teacher Education Assessment System© 1. First Symbol - Classification A = Resource Data 2. B = Productivity Data 3. C = Candidate Performance Data 4. D= Faculty Performance Data 2. Second Symbol - Level 1. 1= Unit Data 2. 2= Institutional Data 3. 3= Calculated Data 3. Third Symbol - Category 1. Sub-systems 4. Fourth Symbol - Number 1. Element Number 1. David C. Smith Creating a Teacher Education Assessment System 19

TEAS© System and Sub-systems A. Resources 1. Unit 1. Human Resources 2. Fiscal Resources 3. Demographic Data 2. Institution 1. Human Resources 2. Fiscal Resources 3. Demographic Data 3. Derived David C. Smith B. Productivity 1. Unit 1. Enrollment 2. Loads and Assignments 3. Program 4. Title II Data 5. External $ Generated 6. Student Demographic Data 2. Institution 1. . Enrollment 2. Loads and Assignments 3. External $ Generated 4. Student Demographic Data 3. Derived 1. Enrollment 2. Loads and Assignments 3. External $ Generated 4. Student Demographic Data Creating a Teacher Education Assessment System 20

TEAS© System and Sub-systems C. Candidate Performance D. Faculty 1. Unit 1. Initial Program Admission 1. Teaching Evaluations 2. Early in Program 2. Scholarly Productivity 3. Mid-program 4. Summative 3. Professional Service 5. Graduate Follow-up 4. Institutional Service 6. Advanced Programs Admission 2. Institution 7. Mid-program 1. Teaching Evaluations 8. Summative 2. Scholarly Productivity 9. Graduate Follow-up 3. Professional Service 2. Institution 4. Institutional Service 1. Undergraduate 3. Derived 2. Graduate 1. Scholarly Productivity 3. Derived 1. Undergraduate 2. Institutional Service 2. Advanced Programs David C. Smith Creating a Teacher Education Assessment System 21

Kinds of Information in TEAS. AACTE – NCATE Annual Report data. Demographic information. l l l A. 1. 3. 1 Number White A. 1. 3. 2 Number Black A. 1. 3. 3 Number Hispanic A. 1. 3. 4 Number Asian A. 1. 3. 5 Number Native American A. 1. 3. 6 Number Pacific Islander A. 1. 3. 7 Number Eskimo A. 1. 3. 8 Number Other A. 1. 3. 9 Total Minority A. 1. 3. 10 Total Faculty A. 1. 3. 11 % Minority Faculty David C. Smith 0 0 #DIV/0! Creating a Teacher Education Assessment System 22

Examples of Use - Resource Information. Section A - Resource Data -- TEAS Unit Data – Resource 2000 Human Resources A. 1. 1. 1 FTE Faculty - Full Time A. 1. 1. 2 FTE Faculty - Part Time A. 1. 1. 3 Summer FTE Faculty A. 1. 1. 4 Total FTE Faculty A. 1. 1. 5 FTE Administration A. 1. 1. 6 FTE Support Personnel David C. Smith Creating a Teacher Education Assessment System 1999 0 23

Comparative Salary Information. Section A - Resource Data -- Derived A. 3. 1 Instr. $ Diff. Inst. – Unit #DIV/0! A. 3. 2 Asst. Prof. $ Diff. Inst. – Unit #DIV/0! A. 3. 3 Assoc. Prof. $ Diff. Inst. – Unit #DIV/0! A. 3. 4 Prof. $ Diff. Inst. – Unit #DIV/0! A. 3. 5 All Ranks $Creating Diff. a Teacher Inst. Education – Unit David#DIV/0! C. Smith Assessment System 24

Productivity Data. Section B - Productivity Data -- Unit Enrollment B. 1. 1. 1 FTE Enroll. Undergrad. B. 1. 1. 2 FTE Enroll. Grad. B. 1. 1. 3 Total FTE Enroll. B. 1. 1. 4 Headcount Enroll. Undergrad B. 1. 1. 5 Headcount Enroll. Grad B. 1. 1. 6 Total Headcount Enroll. B. 1. 1. 7 Total Undergrad. SCH B. 1. 1. 8 Total Grad. SCH B. 1. 1. 9 Total SCH B. 1. 1. 10 Summer SCH Undergrad. B. 1. 1. 11 Summer SCH Grad. B. 1. 1. 12 Total Summer SCH David C. Smith Creating a Teacher Education Assessment System 0 0 25

Data for Staff Planning. A. 1. 2. 18 % Instructors A. 1. 2. 19 % Asst. Prof. A. 1. 2. 20 % Assoc. Prof. A. 1. 2. 21 % Prof. A. 1. 2. 22 Avg. Instructor A. 1. 2. 23 Avg. Asst. Prof. A. 1. 2. 24 Avg. Assoc. Prof. A. 1. 2. 25 Avg. Prof. A. 1. 2. 26 Avg. All Ranks David C. Smith #DIV/0! #DIV/0! #DIV/0! Creating a Teacher Education Assessment System 26

Budget Analysis. A. 1. 2. 27 Consumable Expenses A. 1. 2. 28 Equipment A. 1. 2. 29 Library Budget A. 1. 2. 30 Travel Budget A. 1. 2. 31 Graduate Assistants A. 1. 2. 32 Other A. 1. 2. 33 Total Budget A. 1. 2. 34 Faculty Support Factor David C. Smith Creating a Teacher Education Assessment System $0 #DIV/0! 27

Equity in the System – Over Time. A. 3. 10 Unit % Faculty #DIV/0! A. 3. 11 Unit % Summer Faculty #DIV/0! A. 3. 12 Unit % FTE Support Personnel #DIV/0! A. 3. 13 Unit % Part-time Faculty #DIV/0! A. 3. 14 Unit % Consumable Expenses #DIV/0! A. 3. 15 Unit % Equipment #DIV/0! A. 3. 16 Unit % Administration #DIV/0! A. 3. 17 Unit % Library #DIV/0! A. 3. 18 Unit % Travel #DIV/0! A. 3. 19 Unit % Budget #DIV/0! David C. Smith Creating a Teacher Education Assessment System 28

Monitoring the System. A. 3. 20 Unit % Faculty Change A. 3. 21 Inst. % Faculty Change A. 3. 22 Unit Total Budget % Change A. 3. 23 Inst. Total Budget % Change B. 3. 2. 9 Unit % Change Total SCH #DIV/0! B. 3. 2. 10 Inst. % Change Total SCH #DIV/0! #DIV/0! Patterns become apparent over time! David C. Smith Creating a Teacher Education Assessment System 29

Using Productivity Data Examples. B. 1. 2. 7 Hours Taught/FTE Faculty #DIV/0! B. 2. 2. 4 Hours Taught/FTE Faculty #DIV/0! B. 1. 2. 8 Sch/FTE Faculty #DIV/0! B. 2. 1. 10 Sch/FTE Faculty #DIV/0! B. 1. 3. 1 Cost/SCH #DIV/0! B. 2. 2. 5 Cost/SCH #DIV/0! Compare productivity over time! David C. Smith Creating a Teacher Education Assessment System 30

Candidate Performance Data. Section C - Unit -- Candidate Performance Data Basic Programs Admission C. 1. 1. 1 SAT/ACT C. 1. 1. 2 Writing Sample C. 1. 1. 3 GPA C. 1. 1. 4 Interview David C. Smith Creating a Teacher Education Assessment System 31

Candidate Performance Data. Multiple Measures Early in Program C. 1. 2. 1 Praxis I C. 1. 2. 2 Educ. GPA C. 1. 2. 3 Classroom Obser. - Knowledge. C. 1. 2. 4 Classroom Obser. – Application Clear Expectations David C. Smith Creating a Teacher Education Assessment System 32

Candidate Performance Data. Mid-Program C. 1. 3. 1 Total GPA C. 1. 3. 2 Tutorial Knowledge C. 1. 3. 3 Tutorial ESL C. 1. 3. 4 Small Group Knowledge C. 1. 3. 5 Small Group ESL = Evidence of Student Learning. David C. Smith Creating a Teacher Education Assessment System 33

Candidate Performance Data. Summative C. 1. 4. 1 Final GPA C. 1. 4. 2 GPA in Non-Educ. Courses C. 1. 4. 5 Portfolio C. 1. 4. 6 Student Teaching Grade C. 1. 4. 7 No. Taking Praxis II C. 1. 4. 8 No. Passing Praxis II C. 1. 4. 9 Praxis II Success Rate C. 1. 4. 10 No. Taking State License Exam. C. 1. 4. 11 No. Passing State License Exam. C. 1. 4. 12 State License Exam. Rate C. 1. 4. 13 No. Candidates Rec. Elem. Lisc. C. 1. 4. 14 No. Candidates Rec. Sec. Lisc. C. 1. 4. 15 Total No. Candidates Rec. Lisc. Creating a Teacher Education 0 David C. Smith Assessment System #DIV/0! 34

Candidate Impact Data. How important are schools, colleges and departments of education? C. 1. 4. 16 Potential Candidate Impact =(D 345*25*25)+(D 346*125*25) Movers and shakers. One years’ graduating class. David C. Smith Creating a Teacher Education Assessment System 35

Candidate Performance Data. Graduate Follow-Up C. 1. 5. 1 Prog. Assessment - Graduates C. 1. 5. 2 Graduate Assess. - Principals David C. Smith Creating a Teacher Education Assessment System 36

Faculty Performance Data. Professional Service D. 1. 3. 1 No. State Wrkshps, Elem D. 1. 3. 2 No. State Wrkshps, Sec D. 1. 3. 3 No. State Wrkshps, Total D. 1. 3. 4 Elem. Tchrs. @ State Conf. D. 1. 3. 5 Sec. Tchrs. @ State Conf. D. 1. 3. 6 Total Tchrs. @ State conf. (PS) D. 1. 3. 7 Potential Student Impact D. 1. 3. 8 No. State Wrkshps/Faculty David C. Smith Creating a Teacher Education Assessment System 0 0 0 #DIV/0! 37

Much of this information is important only to deans. NOT SO! Without information of this kind the dean cannot effectively serve as an advocate for the unit. Much of the information can be generated only by the faculty and program heads. Often other units are able to make their data more compelling. David C. Smith Creating a Teacher Education Assessment System 38

Relationships Among Assessment Systems Institution Department Unit Program Remember, you get big numbers by adding little numbers. So you may need to have sub-unit assessment systems. David C. Smith Creating a Teacher Education Assessment System 39

Kinds of Data Top Down Data SCH Budget Data Enrollment Data FTE Faculty FTE Students Student Demographics Most of these data can be acquired. David C. Smith Bottom Up Data Faculty performance data. Candidate performance data. Student performance data. Most of these data can only be generated at the program level and aggregated to the unit level. Creating a Teacher Education Assessment System 40

Focus on Faculty Performance Critical Check Points Admission (GPA – Essay – Interview) Knowledge Dispositions N. A. Skills Evidence of student learning – Early Program Assessment (Classroom Observation – Participation) Knowledge Skills Dispositions Evidence of student learning – N. A. Mid-Program Assessment (Diagnosis - Tutorial) Knowledge Dispositions David C. Smith Skills Evidence of student learning Creating a Teacher Education Assessment System 41

Critical Check Points Exit Assessment (Student Teaching Evaluation – Tests) Knowledge - Skills - Dispositions Evidence of student learning Follow-Up Assessment (Principal Evaluation – Graduate Survey) Knowledge - Skills - Dispositions Evidence of student learning David C. Smith Creating a Teacher Education Assessment System 42

David C. Smith Creating a Teacher Education Assessment System 43

Attributes of this system. (Exclusive of Title II Data) 366 pieces of information are contained in the system. 216 pieces of information (59%) are entered. l l Many of the data are already routinely collected. It is very likely that the institution and unit currently collects some of the data. 150 pieces of information (41%) are generated. Consider the kinds of information generated. l l Comparative data within the system. Comparative data over time – multiple years. Evaluative data. Data for planning. Graphs and tables can be developed from the data in this system. (Enrollment and Budget, % of Faculty and Budget, % of Grad. SCH and Budget) David C. Smith Creating a Teacher Education Assessment System 44

What You Have and What You Need David C. Smith Creating a Teacher Education Assessment System 45

David C. Smith Creating a Teacher Education Assessment System 46

This System and Your System The data in this performance assessment system can be used for: l l Unit planning, advocacy, annual reports and budgeting. Institutional planning and evaluation. External advocacy and information. Accreditation, program approval and review. This system is easily modified. Remember, you do not need to create a performance assessment system annually. The entire assessment system does not need to be put into operation simultaneously. David C. Smith Creating a Teacher Education Assessment System 47

Your Assessment System. Think about the framework for your system. All data elements do not need the same baseline. You can implement your assessment systematically and gradually. An assessment system can help make informed decisions about improving your programs. There is a relationship between resources and productivity. Make unconventional comparisons. (Compare SCH/FTE with candidate evaluations of programs over time. ) David C. Smith Creating a Teacher Education Assessment System 48

Your System This system is available to you. Email: smithdc@aol. com But you need your own system. This system is easily modifiable. There advantages in having some commonality in data sets - within the state, region and nationally. David C. Smith Creating a Teacher Education Assessment System 49

Remember The burden of proof rests with the unit. You do not need to measure everything. You need to present sufficient information for the BOE to make a reasoned and reasonable judgment. David C. Smith Creating a Teacher Education Assessment System 50

The Real Question is. . . Does your assessment system provide sufficient information to permit you and others (in this case NCATE) to draw accurate and reasoned judgments about the quality and performance of your teacher candidates, faculty, unit and the context in which you function? David C. Smith Creating a Teacher Education Assessment System 51

Closing thoughts. Our ability to serve as effective advocates is essential. No effective case can be made without data for support. We must document the level of our effectiveness in a much more sophisticated manner. That requires an assessment system. We need to become much more proactive in portraying the teacher education enterprise. David C. Smith Creating a Teacher Education Assessment System 52