NCATE Institutional Orientation Issues in Assessment Systems Loren

NCATE Institutional Orientation Issues in Assessment Systems Loren Blanchard, Vice President Accreditation & Accountability Louisiana System of Universities Loren Blanchard and David C. Smith, Dean Emeritus College of Education University of Florida Issues in Assessment Systems

Accountability in Higher Education n Establishing criteria for the systematic collection for the collection and analysis of data. n SCDEs often respond to policy makers and accrediting bodies. n The issues should be: program efficacy and program improvement. Loren Blanchard and David C. Smith Issues in Assessment Systems 2

Accountability and Assessment are With Us It is obvious that we are in an age of accountability and that inevitably involves assessment. Like the Ice Age, there is little reason to believe that it will pass quickly. Loren Blanchard and David C. Smith Issues in Assessment Systems 3

Questions and More Questions n What kinds of questions do others ask? (State DOE, Legislature, University, Feds. ) n Why do they ask those kinds of questions? n What do those kinds of questions signal about their concerns? Loren Blanchard and David C. Smith Issues in Assessment Systems 4

What questions do you wish others would ask? Loren Blanchard and David C. Smith Issues in Assessment Systems 5

n How do you respond to important questions regarding program effectiveness and efficiency when you are asked, “what evidence do you have”, and “how do you know”? n What data do you have to support your convictions and contentions? Loren Blanchard and David C. Smith Issues in Assessment Systems 6

n Would you argue that SCDEs should not be accountable for the competence of those they recommend for licensure? Loren Blanchard and David C. Smith Issues in Assessment Systems 7

n. A license is to teachers and other professional educational personnel what accreditation is to SCDEs. Loren Blanchard and David C. Smith Issues in Assessment Systems 8

How do you think that we should be held accountable? Loren Blanchard and David C. Smith Issues in Assessment Systems 9

The nature of change is such that the press for change will always be greater from outside the organization than within the organization. Loren Blanchard and David C. Smith Issues in Assessment Systems 10

Have you thought deeply about the purpose, mission and vision of the organization and related them to the assessment system? There are implications for what you choose to include in your assessment system. Loren Blanchard and David C. Smith Issues in Assessment Systems 11

Goods and Services and Assessment n The assessment of products and services is different. n Products are tangible goods. n Services are intangible goods. n The distinction is important. Loren Blanchard and David C. Smith Issues in Assessment Systems 12

The assessment issues are quite different. Education is not a “product”; it does not have weight and does not take up space. n Education involves providing services that are intangible but have value. n The measurement of services is exponentially more difficult than measuring products. n This distinction should be made clear to policy makers. n Loren Blanchard and David C. Smith Issues in Assessment Systems 13

n We deride single-measure, simplistic measures of educational performance. But what do we offer as a better alternative? n “To every complex problem there is an answer that is short, simple and wrong. ” H. L. Menkin Loren Blanchard and David C. Smith Issues in Assessment Systems 14

Effectiveness and Efficiency n Effectiveness has to do with how well something is functioning. n Efficiency has to do with the effort, time, or cost involved in the preparation or production. Loren Blanchard and David C. Smith Issues in Assessment Systems 15

Inevitable Tension in Assessment n The need to accommodate your situation. n The need to compare with others in similar situations. n It is necessary to compare within and across institutions. (Internal relative productivity and comparison with counterpart units. ) Loren Blanchard and David C. Smith Issues in Assessment Systems 16

A fundamental principle in communication theory is that when the system becomes overloaded with information people will ignore all of the information rather that to ignore the information selectively. (Consider that principle in the context of fact books and assessment systems. ) Loren Blanchard and David C. Smith Issues in Assessment Systems 17

People in an organization focus on what is measured not what is said to be important. Loren Blanchard and David C. Smith Issues in Assessment Systems 18

Consider open versus closed systems. Isn’t it better to have people working from the same numbers? Loren Blanchard and David C. Smith Issues in Assessment Systems 19

It is necessary to compare within and across institutions. (Internal relative productivity and comparison with counterpart units. ) Loren Blanchard and David C. Smith Issues in Assessment Systems 20

n Do not make high-stakes decisions based on soft data. n Consider “directionality” in analyzing data. Loren Blanchard and David C. Smith Issues in Assessment Systems 21

A single assessment system to serve multiple needs is preferable to a set of assessment systems, each of which is designed to serve a single purpose. Loren Blanchard and David C. Smith Issues in Assessment Systems 22

You get big numbers by adding little numbers. It is important to aggregate data at meaningful levels. (Program, department, college, institution. ) Loren Blanchard and David C. Smith Issues in Assessment Systems 23

What we measure is a profound statement about what we believe is important. Loren Blanchard and David C. Smith Issues in Assessment Systems 24

It is one thing to collect data, another thing to be discriminating in collecting data, and still another thing to know how to use data. Loren Blanchard and David C. Smith Issues in Assessment Systems 25

What matters and what matters most? (The need to know and the nice to know. ) Loren Blanchard and David C. Smith Issues in Assessment Systems 26

Multiple measures are important. It is valuable to look at a problem from more than one angle. What are key points at which to access? (At a minimum, entrance, exit and follow-up. ) Loren Blanchard and David C. Smith Issues in Assessment Systems 27

We are not good at being proactive in generating data and we are not good at being creative in generating data. n Be proactive – give people information that they do not ask for but informs them more deeply about the effectiveness of your organization. Think carefully about what that information might be. n Loren Blanchard and David C. Smith Issues in Assessment Systems 28

In higher education it is always teaching, research, and service. Typically policy makers place a higher value on professional public service than does the academy. Loren Blanchard and David C. Smith Issues in Assessment Systems 29

Consider the role of the executive in assessment. Assessment phases: planning, implementing, maintaining, and revising the system. Planning involves knowing what data are needed and meaningful. It also involves assembling a design team. Implementation involves providing the personnel, hardware and software. Maintenance involves using the data in a meaningful fashion. There is a difference between data-informed and data-driven decisions. Revision involves recognizing and accommodating changing needs for data. Loren Blanchard and David C. Smith Issues in Assessment Systems 30

Problems that are regularly seen: People have difficulty in creating an assessment system. n People think more about collecting data than they do about the structure of their assessment system and the kinds of data that they include in their assessment system. n They often want to do it for the wrong reason – for accreditation rather than seeing it as a tool to evaluate and improve what they are doing. n Loren Blanchard and David C. Smith Issues in Assessment Systems 31

They have difficulty in using candidate data three ways. (The aggregation issue. ) n People are not aware of meaningful data that already exists and can be imported into their assessment system. Then they can focus their effort on data that they need to generate. n People often do not know how to use data well. n Loren Blanchard and David C. Smith Issues in Assessment Systems 32

People often do not consider examining relationships among data sets. (FTE and headcount enrollment, enrollment and cost to generate a SCH). n Time is a problem. It is not realistic to expect that busy people can create and maintain an assessment system “on top of everything else”. It is very difficult to develop, implement, maintain and revise an assessment system without additional resources. n Resources – human and technological, are needed. The allocation of resources is a measure of institutional priority. n Loren Blanchard and David C. Smith Issues in Assessment Systems 33

Creating an assessment system is a creative task; it is also tedious and timeconsuming. Loren Blanchard and David C. Smith Issues in Assessment Systems 34

Think about assessment and the development of an assessment system as an opportunity rather than a problem or burden. Loren Blanchard and David C. Smith Issues in Assessment Systems 35

Assessment Assumptions: Bases for Change n Assessment should be based on explicit outcomes. n Assessment must involve performance. n Assessment should provide feedback to the learner. n Assessment should have a multiple context. Loren Blanchard and David C. Smith Issues in Assessment Systems 36

Assessment Assumptions: Bases for Change n Assessment should involve external publics. n Assessment should be developmental. n Assessment should be cumulative. Loren Blanchard and David C. Smith Issues in Assessment Systems 37

Developing an Assessment System that Culminates in a Change Orientation n n Identify what it is that you want to learn or know more about. Determine how you will go about learning more information. Establish a plan for what will happen to data once they are collected. Determine the tools, strategies and resources that you have available or will need to foster change. Loren Blanchard and David C. Smith Issues in Assessment Systems 38

Developing an Assessment System that Culminates in a Change Orientation Be prepared to make changes and continue to study effects of these changes to ensure that there are no adverse effects. Loren Blanchard and David C. Smith Issues in Assessment Systems 39

Two Systems of Performance Assessment n Professional Accountability Support System Using a Portal Approach (PASSPORT) – Statewide Initiative n Teacher Education Assessment System (TEAS) – National Initiative Loren Blanchard and David C. Smith Issues in Assessment Systems 40

PASS-PORT An Electronic Professional Accountability Support System Using a PORTal Approach for Teacher Education Loren Blanchard and David C. Smith Development Funded by the Issues in Assessment Systems Board of Regents CITAL Funds

Evolution of PASS-PORT n Professional collaborative project between a Louisiana public and a Louisiana private higher education institution n State/national demands for outcomebased assessments for teacher education Loren Blanchard and David C. Smith Issues in Assessment Systems 42

Evolution of PASS-PORT n Professional Accountability Support System Using a Portal Approach (PASS-PORT) n n n Highly comprehensive web-based tool Designed primarily for Louisiana teacher preparation programs Allows institutions to gather and statistically analyze performance data on candidates, faculty and program operations at select time frames to glean significant trend data Loren Blanchard and David C. Smith Issues in Assessment Systems 43

Evolution of PASS-PORT is a system designed to assist institutions in fulfilling NCATE Standards, Title II Accountability reporting standards, and select teacher education redesign criteria. n PASS-PORT is designed to assist institutions to better appreciate systematic collection, analysis and use of program data. n Loren Blanchard and David C. Smith Issues in Assessment Systems 44

Evolution of PASS-PORT n PASS-PORT has wealth of useful functionalities n Defines KSD requirements and accompanying assessments for candidates at four different checkpoints across initial and advanced programs Loren Blanchard and David C. Smith Issues in Assessment Systems 45

Loren Blanchard and David C. Smith Issues in Assessment Systems 46

PASS-PORT Functionality 1. Creation of Standards-based Electronic Portfolios (“working” portfolios) § Upload artifacts to PASS-PORT § Create standards-based portfolios (from scratch or § § from template) that are aligned with any set of professional standards (NCATE, ISTE, INTASC, NCTM, etc. ) Request electronic feedback and evaluation Used by faculty as well as teacher candidates Benefit: Standards-based Electronic Portfolios, Unit Reporting 47 Examples: Uploading Artifacts; Working Portfolio From Scratch; From Template

PASS-PORT Functionality 2. Creation of “Portal folios” - special portfolios submitted at various checkpoints (unit specified requirements) in the program: § Portal I: Declaration of Major § Portal II: Entry into Teacher Education Program § Portal III: Entry into Student Teaching § Portal IV: Exit from Teacher Education Program Examples: Creating a Portal Folio; Routing of Portal Folios; Evaluating a Portal Folio Benefit: Meet NCATE expectations for performance-based data 48 collection, analysis, unit improvement and student learning.

Evolution of PASS-PORT n PASS-PORT has wealth of useful functionalities n Emphasizes field/clinical experiences and corresponding sites Loren Blanchard and David C. Smith Issues in Assessment Systems 49

PASS-PORT Functionality 3. Management of Field Experiences Database of field experience sites Students manage own field experiences Add reflections to field experiences Collects data on field experience site and cooperating teacher for reporting § Field experiences can become artifacts to include in portal folios § § Example: Creating Field Experiences Benefit: Meet NCATE expectations for evaluation of 50 field experiences, improve unit and student learning.

Evolution of PASS-PORT n PASS-PORT has wealth of useful functionalities n Draws out and statistically analyzes specific data for varied reporting purposes Loren Blanchard and David C. Smith Issues in Assessment Systems 51

PASS-PORT Functionality 4. Analysis/Reporting and Communication § Maintain record of professional development by candidates and faculty for reporting purposes § Faculty create “taskfolios” (functional portfolios) § Generate reports to meet NCATE, Board of Regents, and/or university requirements, etc. § Create query to analyze any data field (i. e. , field experiences) Examples: Student Gender Analysis, Survey Example, Portal Folio Analysis Benefit: Meet NCATE expectations for collecting and analyzing 52 performance data for the entire college.

PASS-PORT Functions Encourages faculty to develop portfolios that align with NCATE’s Faculty Performance elements n Is a recording mechanism to chronicle program and unit improvements made as a consequence of data analyses n Provides individualized assessment results at each checkpoint for candidates and faculty that include strengths, weaknesses and prescriptions n Loren Blanchard and David C. Smith Issues in Assessment Systems 53

PASS-PORT Functionality 5. NCATE Electronic Documents Room § Ability to create a secure web-based viewing area for displaying documents / artifacts for the NCATE review team § Post example portalfolios for review § Share individual candidate’s artifacts at different points to show growth and feedback/prescriptions § NCATE committee tools, communication Benefit: Facilitate NCATE visit through easy creation 54 of electronic documents room.

EDR Functionality n Creation of Document Rooms Create an unlimited number of secure webbased viewing rooms n Rooms are built by creating pages (from templates) that can be nested and linked together with a simple point and click interface [no web master required] n Provides access for NCATE reviewers prior to site visit n Loren Blanchard and David C. Smith Issues in Assessment Systems 55

PASS-PORT Connection Connects to Louisiana’s K-12 Initiative and Enhances the PK-16+ Education Initiative The Department of Education is developing a similar electronic portfolio system based on PASS-PORT functionality and the seamless exchange of data. As a result, a pre-service teacher’s performance data, initially developed at the undergraduate level, will be accessible upon employment and entrance into the state’s K-12 professional portfolio system. When the teacher enters graduate school, he/she will reenter PASS-PORT at the university. Loren Blanchard and David C. Smith Issues in Assessment Systems 56

PASS-PORT Questions? Contact… Doug Williams, dwilliams@louisiana. edu Loren Blanchard, Loren@uls. state. la. us Juanita Guerin, jguerin@louisiana. edu John Fulwiler, jfulwile@xula. edu Loren Blanchard and David C. Smith Issues in Assessment Systems

What kinds of data might be included in a Teacher Education Performance Assessment System©? (Establishing a conceptual framework. ) n Resource Data n Productivity Data n Candidate Performance Data n Faculty Performance Data Loren Blanchard and David C. Smith Issues in Assessment Systems 58

Structure of a Teacher Education Assessment System© 4 4 Loren Blanchard and David C. Smith First Symbol - Classification √ A = Resource Data √ B = Productivity Data √ C = Candidate Performance Data √ D= Faculty Performance Data Second Symbol - Level √ 1= Unit Data √ 2= Institutional Data √ 3= Calculated Data Third Symbol - Category √ Sub-systems Fourth Symbol - Number √ Element Number Issues in Assessment Systems 59

TEAS© System and Sub-systems A. Resources 1. Unit 1. Human Resources 2. Fiscal Resources 3. Demographic Data 2. Institution 1. Human Resources 2. Fiscal Resources 3. Demographic Data 3. Derived Loren Blanchard and David C. Smith B. Productivity 1. Unit 1. Enrollment 2. Loads and Assignments 3. Program 4. Title II Data 5. External $ Generated 6. Student Demographic Data 2. Institution 1. . Enrollment 2. Loads and Assignments 3. External $ Generated 4. Student Demographic Data 3. Derived 1. Enrollment 2. Loads and Assignments 3. External $ Generated 4. Student Demographic Data Issues in Assessment Systems 60

TEAS© System and Sub-systems C. Candidate Performance 1. Unit 1. Initial Program Admission 2. Early in Program 3. Mid-program 4. Summative 5. Graduate Follow-up 6. Advanced Programs Admission 7. Mid-program 8. Summative 9. Graduate Follow-up 2. Institution 1. Undergraduate 2. Graduate 3. Derived 1. Undergraduate 2. Advanced Programs Loren Blanchard and David C. Smith D. Faculty 1. Unit 1. Teaching Evaluations 2. Scholarly Productivity 3. Professional Service 4. Institutional Service 2. Institution 1. Teaching Evaluations 2. Scholarly Productivity 3. Professional Service 4. Institutional Service 3. Derived 1. Scholarly Productivity 2. Institutional Service Issues in Assessment Systems 61

Kinds of Information in TEAS. n AACTE – NCATE Annual Report data. n Resource Information – Salaries, Budget, Faculty Positions… n Comparative Salary Information – Unit, Institution… Productivity Data – SCH, Headcount Enrollment, FTE Enrollment, Service Activity, Graduate Impact Data for Staff Planning Data for Budget Analysis Advocacy Data - Equity in the System, Monitoring the System n n Loren Blanchard and David C. Smith Issues in Assessment Systems 62

Using Productivity Data - Examples. B. 1. 2. 7 Hours Taught/FTE Faculty #DIV/0! B. 2. 2. 4 Hours Taught/FTE Faculty #DIV/0! B. 1. 2. 8 Sch/FTE Faculty #DIV/0! B. 2. 1. 10 Sch/FTE Faculty #DIV/0! B. 1. 3. 1 Cost/SCH #DIV/0! B. 2. 2. 5 Cost/SCH #DIV/0! Compare productivity over time! Loren Blanchard and David C. Smith Issues in Assessment Systems B. 1. 3. 2 $ Prepare Elem. Teacher Approx. 63

Candidate Impact Data. How important are schools, colleges and departments of education? C. 1. 4. 16 Potential Candidate Impact =(D 345*25*25)+(D 346*125*25) Movers and shakers. One years’ graduating class. Loren Blanchard and David C. Smith Issues in Assessment Systems 64

Faculty Performance Data. Professional Service D. 1. 3. 1 No. State Wrkshps, Elem D. 1. 3. 2 No. State Wrkshps, Sec D. 1. 3. 3 No. State Wrkshps, Total D. 1. 3. 4 Elem. Tchrs. @ State Conf. D. 1. 3. 5 Sec. Tchrs. @ State Conf. D. 1. 3. 6 Total Tchrs. @ State conf. (PS) D. 1. 3. 7 Potential Student Impact D. 1. 3. 8 No. State Wrkshps/Faculty Loren Blanchard and David C. Smith Issues in Assessment Systems 0 0 0 #DIV/0! 65

Attributes of this system. 366 pieces of information are contained in the system. n 216 pieces of information (59%) are entered. n n Many of the data are already routinely collected. It is very likely that the institution and unit currently collects some of the data. 150 pieces of information (41%) are generated. Loren Blanchard and David C. Smith Issues in Assessment Systems 66

n Consider the kinds of information generated. n n n Comparative data within the system. Comparative data over time – multiple years. Evaluative data. Data for planning. Proactive data. Graphs and tables can be developed from the data in this system. (Enrollment and Budget, % of Faculty and Budget, % of Grad. SCH and Budget) Loren Blanchard and David C. Smith Issues in Assessment Systems 67

What You Have and What You Need Loren Blanchard and David C. Smith Issues in Assessment Systems 68

This System and Your System n The data in this performance assessment system can be used for: n n Unit planning, advocacy, annual reports and budgeting. Institutional planning and evaluation. External advocacy and information. Accreditation, program approval and review. This system is easily modified. n Remember, you do not need to create a performance assessment system annually. n The entire assessment system does not need to be put into operation simultaneously. n Loren Blanchard and David C. Smith Issues in Assessment Systems 69

Your Assessment System. n n n Think about the framework for your system. All data elements do not need the same baseline. You can implement your assessment systematically and gradually. An assessment system can help make informed decisions about improving your programs. There is a relationship between resources and productivity. Make unconventional comparisons. (Compare SCH/FTE with candidate evaluations of programs over time. ) Loren Blanchard and David C. Smith Issues in Assessment Systems 70

Your System n n n This system is available to you. Email: smithdc@aol. com But you need your own system. This system is easily modifiable. There advantages in having some commonality in data sets - within the state, region and nationally. Loren Blanchard and David C. Smith Issues in Assessment Systems 71

Loren Blanchard and David C. Smith Issues in Assessment Systems 72