Essential Competencies for Program Evaluators Jean A King

  • Slides: 39
Download presentation
Essential Competencies for Program Evaluators Jean A. King 2012 AEA/CDC Summer Institute

Essential Competencies for Program Evaluators Jean A. King 2012 AEA/CDC Summer Institute

Agenda for the session • Identifying competencies intuitively • Beliefs about evaluator competencies •

Agenda for the session • Identifying competencies intuitively • Beliefs about evaluator competencies • The ECPE and how they work in different settings • Case application • Do-it-yourself concept maps • Self-assessment and planning for professional development

People create these things! IDENTIFYING COMPETENCIES

People create these things! IDENTIFYING COMPETENCIES

Identifying competencies intuitively What are the knowledge, skills and dispositions that a program evaluator

Identifying competencies intuitively What are the knowledge, skills and dispositions that a program evaluator needs to successfully complete the evaluation?

Evaluator relationships • A relationship exists among – The evaluator – The client –

Evaluator relationships • A relationship exists among – The evaluator – The client – Program staff and other evaluation stakeholders • Primary responsibility for two tasks: – Making evaluation decisions – Implementing the evaluation

Evaluator relationships • Primary responsibility for making and implementing evaluation decisions may lie (1)

Evaluator relationships • Primary responsibility for making and implementing evaluation decisions may lie (1) With the evaluator (2) With someone else in the setting (3) Jointly with both • The evaluation relationship may shift during the study

Interactive participation quotient (IPQ) HIGH Program leaders, staff, community Involvement in decision making and

Interactive participation quotient (IPQ) HIGH Program leaders, staff, community Involvement in decision making and implementation Evaluator LOW Evaluatordirected Collaborative ZONES Participantdirected

What do you believe? COMPETENCY STATEMENTS

What do you believe? COMPETENCY STATEMENTS

Question: To what extent do your personal beliefs matter in a discussion of evaluator

Question: To what extent do your personal beliefs matter in a discussion of evaluator competencies?

Presenting. . . ESSENTIAL COMPETENCIES FOR PROGRAM EVALUATORS

Presenting. . . ESSENTIAL COMPETENCIES FOR PROGRAM EVALUATORS

How were the ECPE developed? • Began: With an idea in a graduate course

How were the ECPE developed? • Began: With an idea in a graduate course • Then: “What does the literature tell us? ” • Then: A pilot study using the initial set of competencies, AJE publication (2001) • Then: A revision process including a crosswalk • Then: ECPE published in AJE (2005) • Then: Articles in CJPE (2005) and AJE (2006) • Then: CES’s professional designation program • Next: A national validation study

1) Professional practice Ø Fundamental norms and values of evaluation practice - Applies evaluation

1) Professional practice Ø Fundamental norms and values of evaluation practice - Applies evaluation standards - Works ethically - Respects all stakeholders - Considers the general and public welfare - Contributes to the knowledge base

2) Systematic inquiry 1. Technical aspects of evaluation practice 1. Understands knowledge base 2.

2) Systematic inquiry 1. Technical aspects of evaluation practice 1. Understands knowledge base 2. Knows quantitative and qualitative methods 3. Knows mixed methods 4. Conducts literature reviews 5. Specifies program theory 6. Frames evaluation questions 7. Designs the evaluation PLUS. . .

2) Systematic inquiry (cont. ) ØCollects and interprets data ØMakes judgments and recommendations ØProvides

2) Systematic inquiry (cont. ) ØCollects and interprets data ØMakes judgments and recommendations ØProvides rationales for decisions ØReports procedures and results ØNotes evaluation strengths and limitations ØConducts meta-evaluations

3) Situational analysis Ø Unique interests, issues, and contextual circumstances of evaluation Describes program

3) Situational analysis Ø Unique interests, issues, and contextual circumstances of evaluation Describes program Ø Determines program evaluability Ø Identifies stakeholders’ interests Ø Serves intended users’ informational needs Ø Addresses conflicts Ø Attends to evaluation use PLUS. . . Ø

3) Situational analysis (cont. ) Ø Examines organizational and political context Ø Attends to

3) Situational analysis (cont. ) Ø Examines organizational and political context Ø Attends to organizational change Ø Respects site and client uniquenesses Ø Remains open to input Ø Modifies evaluation as needed

4) Project management Ø “Nuts and bolts” of evaluation work ØResponds to RFPs ØNegotiates

4) Project management Ø “Nuts and bolts” of evaluation work ØResponds to RFPs ØNegotiates with clients ØWrites formal agreements ØCommunicates with client ØBudgets evaluation ØJustifies cost PLUS. . .

4) Project management (cont. ) ØIdentifies needed resources ØUses technology as appropriate ØSupervises and

4) Project management (cont. ) ØIdentifies needed resources ØUses technology as appropriate ØSupervises and trains others ØConducts evaluation in a nondisruptive manner ØPresents work in timely manner

5) Reflective practice Ø One’s own evaluation expertise and need for growth - Knows

5) Reflective practice Ø One’s own evaluation expertise and need for growth - Knows self - Reflects on practice - Pursues professional development: evaluation and content areas - Builds professional relationships

6) Interpersonal competence Ø “People skills” necessary for evaluation practice - Uses written and

6) Interpersonal competence Ø “People skills” necessary for evaluation practice - Uses written and verbal/listening communication skills - Uses negotiation skills - Uses conflict resolution skills - Facilitates constructive interpersonal interaction - Demonstrates cross-cultural competence

A recent research study AN INITIAL STUDY TO DEVELOP INSTRUMENTS AND VALIDATE THE ECPE

A recent research study AN INITIAL STUDY TO DEVELOP INSTRUMENTS AND VALIDATE THE ECPE

Wilcox (2012) research study • Top- and lowest-rated competencies (rated on a scale of

Wilcox (2012) research study • Top- and lowest-rated competencies (rated on a scale of 1 [not at all necessary] to 5 [extremely necessary]) • Subject area differences not as important as evaluator role • Results of 9 expert interviews

Top weighted competencies (Wilcox, 2012) Label Competency Acts ethically and strives for integrity and

Top weighted competencies (Wilcox, 2012) Label Competency Acts ethically and strives for integrity and honesty in conducting evaluations Respects clients, respondents, program 1. 4 participants, and other stakeholders Uses verbal/listening communication 6. 2 skills 2. 14 Interprets data 6. 1 Uses written communication skills 1. 2 Weighted Mean 4. 87 4. 76 4. 67 4. 63 4. 55

Lowest weighted competencies (Wilcox, 2012) Label Competency Conducts literature reviews Specifies program theory Responds

Lowest weighted competencies (Wilcox, 2012) Label Competency Conducts literature reviews Specifies program theory Responds to requests for 4. 1 proposals Contributes to the knowledge base 1. 6 of evaluation 2. 20 Conducts meta-evaluations 2. 5 2. 6 Weighted Mean 3. 61 3. 55 3. 20 3. 09 2. 70

Lowest weighted competencies (Wilcox, 2012) Label 2. 5 2. 6 4. 1 1. 6

Lowest weighted competencies (Wilcox, 2012) Label 2. 5 2. 6 4. 1 1. 6 2. 20 Competency Conducts literature reviews Specifies program theory Responds to requests for proposals Contributes to the knowledge base of evaluation Weight ed Mean 3. 61 3. 55 3. 20 3. 09 Conducts meta-evaluations 2. 70

Original and final interviewee categories Original Final Education evaluators Public health evaluators Evaluators in

Original and final interviewee categories Original Final Education evaluators Public health evaluators Evaluators in formal organizations Evaluators of non-profits Evaluations as independent consultants

Summary of expert interviews (Wilcox, 2012) ECPE Category In Education In Formal Organizations As

Summary of expert interviews (Wilcox, 2012) ECPE Category In Education In Formal Organizations As Independent Consultants Professional practice Primary Systematic inquiry Essential [Team building] Essential [Collaboration] Very important [Reliance on assessment data] Interpersonal competence Very important

Summary of expert interviews (Wilcox, 2012) ECPE Category In Education In Formal Organizations As

Summary of expert interviews (Wilcox, 2012) ECPE Category In Education In Formal Organizations As Independent Consultants Situational analysis Project management Seldom Sometimes Frequently Not so much All the time Reflective practice Wish to be able to do more regularly Senior staff’s responsibility Could do better Trying to keep up

Applying the competencies. . . 1. Back to the IPQ for discussion/reflection 2. A

Applying the competencies. . . 1. Back to the IPQ for discussion/reflection 2. A case application

Now the fun part. . . ECPE CONCEPT MAPPING

Now the fun part. . . ECPE CONCEPT MAPPING

Work context and program evaluator competencies

Work context and program evaluator competencies

Create a concept map-In your work context, what are the relationships between and among

Create a concept map-In your work context, what are the relationships between and among the competencies?

What’s next for you? SELF-ASSESSMENT

What’s next for you? SELF-ASSESSMENT

ECPE self-assessment Reflecting on your own practice, how do you rate your current knowledge,

ECPE self-assessment Reflecting on your own practice, how do you rate your current knowledge, skills, and dispositions to effectively carry out program evaluations?

ECPE rating scale Entry / Novice Proficient/ Skilled Mastery/ Expert

ECPE rating scale Entry / Novice Proficient/ Skilled Mastery/ Expert

Now what? What is your professional development plan?

Now what? What is your professional development plan?

References Ghere, G. , King, J. A. , Stevahn, L. , & Minnema, J.

References Ghere, G. , King, J. A. , Stevahn, L. , & Minnema, J. (2006). A professional development unit for reflecting on program evaluator competencies. American Journal of Evaluation, 27, 108 -123. Stevahn, L. , King, J. A. , Ghere, G. , & Minnema, J. (2005). Establishing essential competencies for program evaluators. American Journal of Evaluation, 26, 43 -59.

Feel free to contact me at kingx 004@umn. edu or see www. evaluation. umn.

Feel free to contact me at kingx 004@umn. edu or see www. evaluation. umn. edu