Evaluator Credentialing and Program Accreditation Is Evaluation Ready

Evaluator Credentialing and Program Accreditation: Is Evaluation Ready to Become a Profession? Who Cares Anyway? Jean A. King University of Minnesota

Is evaluation ready to become a profession? • What is the field’s current status? • What will it take to move us toward true professional status? • What are the implications of this situation? [So what? ]

Why do I speak on this topic?

The Louisiana connection • The Progressive era schoolmen of rural Louisiana (on horseback) • My Type A Louisiana Program Evaluator license (valid for LIFE)

The Minnesota connection • The Minnesota Competencies for Program Evaluators • The Essential Competencies for Program Evaluators (ECPE)

Is evaluation ready to become a profession? WHAT IS THE FIELD’S CURRENT STATUS?

Criteria for Judging Whether Program Evaluation Has Become a Profession (Worthen, 1994) Criteria Satisfied? 1. A need for evaluators 2. Certification or licensure of evaluators 3. Exclusion of unqualified practitioners 4. Unique knowledge and skills of evaluation 5. Preparation programs for evaluators 6. Professional associations 7. Accreditation of preparation programs 8. Stable career opportunities 9. Standards of practice ? ? ? ? ?

Turn to a neighbor. . . 1. In your opinion, to what extent is program evaluation currently a profession? 2. What supports your opinion? 3. What difference does this make in the world?

Worthen’s 1994 conclusions Criteria 1. A need for evaluators 2. Certification or licensure of evaluators 3. Exclusion of unqualified practitioners 4. Unique knowledge and skills of evaluation 5. Preparation programs for evaluators 6. Professional associations 7. Accreditation of preparation programs 8. Stable career opportunities 9. Standards of practice Satisfied? Yes No No Yes Yes No Yes

Almost 20 years later. . .

Current (2012) status in my opinion Criteria 1. A need for evaluators 4. Unique knowledge and skills 5. Preparation programs for evaluators 6. Professional associations 8. Stable career opportunities 9. Standards of practice Evidence We’re employed; increasing number of jobs Yes Yes Yes

Current (2012) status Criteria 1. A need for evaluators 4. Unique knowledge and skills 5. Preparation programs for evaluators 6. Professional associations 8. Stable career opportunities 9. Standards of practice Evidence We’re employed; increasing work ECPE+

“Unique knowledge and skills” The Essential Competencies for Program Evaluators

But a challenge. . . Public health knowledge and skills Evaluation knowledge and skills NGO knowledge and skills “Pick your subject area” knowledge and skills

Current (2012) status Criteria 1. A need for evaluators Evidence 4. Unique knowledge and skills We’re employed; increasing work ECPE+ La. Velle & Donaldson (2010) 6. Professional associations 8. Stable career opportunities 9. Standards of practice Yes Yes 5. Preparation programs for evaluators

Compilation of preparation programs (La. Velle & Donaldson, 2010)

Current (2012) status Criteria 1. A need for evaluators Evidence We’re employed; increasing work 4. Unique knowledge and skills ECPE+ 5. Preparation programs for evaluators La. Velle & Donaldson (2010) 6. Professional associations 8. Stable career opportunities 9. Standards of practice AEA, CES, plus Yes

Current (2012) status Criteria 1. A need for evaluators Evidence We’re employed; increasing work 4. Unique knowledge and skills ECPE+ 5. Preparation programs for evaluators La. Velle & Donaldson (2010) 6. Professional associations AEA, CES, plus 8. Stable career opportunities 9. Standards of practice Increase in evaluation jobs/careers Yes

Current (2012) status Criteria 1. A need for evaluators 4. Unique knowledge and skills 5. Preparation programs for evaluators 6. Professional associations 8. Stable career opportunities 9. Standards of practice Evidence We’re employed; increasing work ECPE+ La. Velle & Donaldson (2010) AEA, CES, plus Increase in evaluation jobs Program Evaluation Standards+

The Joint Committee Program Evaluation Standards

Our field definitely has standards • Three editions to date: 1981, 1994, and 2011 • There also standards for personnel and student evaluation • See the Western Michigan University website for additional information

The Joint Committee Program Evaluation Standards The five categories of the PES: • Utility • Feasibility • Propriety • Accuracy • Evaluation Accountability Remember UFPAE!

Other standards available • AEA’s Guiding Principles for Evaluators • The US (GAO) Government Auditing Standards (Stufflebeam & Shinkfield, 2007)

Two untouched criteria Criteria 1. A need for evaluators 2. Certification or licensure of evaluators Satisfied? Yes No 3. Exclusion of unqualified practitioners No 7. Accreditation of preparation programs No 4. Unique knowledge and skills of evaluation 5. Preparation programs for evaluators 6. Professional associations 8. Stable career opportunities 9. Standards of practice Yes Yes Yes

Where action is occurring Criteria Satisfied? 2. Certification or licensure of evaluators *** 1. A need for evaluators 3. Exclusion of unqualified practitioners 4. Unique knowledge and skills of evaluation 5. Preparation programs for evaluators 6. Professional associations 7. Accreditation of preparation programs 8. Stable career opportunities 9. Standards of practice Yes No Yes Yes

Almost 20 years later. . . On the one hand, there must be reasons we haven’t moved ahead on the three missing criteria • “If it ain’t broke, don’t fix it!” • No felt need--or we would have moved on this

Almost 20 years later. . . On the other hand, the field is missing out on the potential benefits of professional status

Where action is occurring “Certification” of evaluators • The Evaluators’ Institute (TEI) certificate • University program certificates (e. g. , Claremont, U of MN) • The Canadian Evaluation Society (CES) Credentialed Evaluators (CE) Designation

Important distinctions Term Definition A process whereby an external body Certification (usually a professional society) assesses a person’s skills and competencies in a given area A set of courses/experiences a person Credentialing goes through to receive a document or status that entitles her to specific authority A process through which a government body awards a credential that allows a Licensure person to practice in a certain field

CE: A critical development • A credentialing system created by a national professional association for its members 1. A comprehensive, approved set of competencies for professional evaluators in Canada 2. A set of rubrics for assessing competency 3. A voluntary system for credentialing (“professional designation, ” “credentialed evaluator” [CE]) 4. A process that has been established, promoted, and used • A set of committed professionals who have made it all happen

Different contexts Canada USA • Treasury Board (an active • Government Accountability and committed participant) Office (GAO) + Office of Management and Budget • Canadian Evaluation (OMB) Society (CES) • American Evaluation Association (AEA) • 10 th amendment to the US constitution

Informal international scan Barbara Rosenstein- Israeli Evaluation Society Certification has been an issue since the organization was founded in 1999 A “difficult and messy kettle of worms” A workshop devoted to it in 2004, but nothing followed Aware of the efforts in Canada

Informal international scan Wolfgang Boettcher- De. GEval (Austria/Germany) Not working on a certification system Debating ethical standards “I believe that some of our master evaluators would not object [to a] a system of certification in which they would play a dominant role” (personal opinion)

Informal international scan Ryo Sasaki- Japanese Evaluation Society Three levels of JES certification: • Certified Professional Evaluator (CPE)training followed by a written exam [N=200 in four years] • Certified Specialty Evaluator (CSE/School Evaluator) • Certified Advanced Evaluator (CAE)

Informal international scan Donna Podems- South African Monitoring and Evaluation Association Conference on competencies in 2010 The government has recently asked for three sets of competencies For evaluators For evaluation managers For those who commission evaluations

Is evaluation ready to become a profession? WHAT WILL IT TAKE TO MOVE TOWARD TRUE PROFESSIONAL STATUS?

Comparisons highlight issues Profession Medical doctors Issues Raised • Lives are at stake; no one dies if an evaluation is poorly done • Meaningful testing is a challenge Lawyers • Great financial rewards for success • A key role for a powerful professional association CPAs (certified • Not everyone chooses to be certified • Who pays for the certification? public accountants) Teachers and • Different systems in different states administrators • Licensure does not guarantee competence

Who is likely to benefit? • Current evaluators, who may receive increased status and more contracts • Novice evaluators who will know what to study • People who commission and manage program evaluations • Attorneys, who will be able to sue the credentialing body for various reasons • The evaluation field—one additional step toward status as a true profession

Who might be hurt? • Current practicing evaluators • Anyone who has to pay for the credential • Good evaluators who lack official training in evaluation • The charlatans we assume are out there • Those responsible for developing and implementing the credentialing process • Individuals • Professional association(s)

Is evaluation ready to become a profession? WHAT ARE THE IMPLICATIONS OF THIS SITUATION?

What if we do nothing?

Questions I have come to. . . • What motivates the development of a credentialing system and sustains it once developed? • Lives are not at stake; no one dies if an evaluation is poorly done • Some group/individual needs to make this a priority; a potential role for a professional association • How do you pay for this to begin? On an ongoing basis? • To what extent is flexibility a good idea? • Should people have an option to be credentialed or not? • Should different states/fields have different systems? • What process would link the credential to competent performance? Must it?

A quick thought (if I were in charge, which I’m not) • Purposeful professional development for practicing evaluators • Voluntary accreditation of evaluation training programs

Try a new perspective. . .