Essential Competencies for Program Evaluators Jean A King
- Slides: 39
Essential Competencies for Program Evaluators Jean A. King 2012 AEA/CDC Summer Institute
Agenda for the session • Identifying competencies intuitively • Beliefs about evaluator competencies • The ECPE and how they work in different settings • Case application • Do-it-yourself concept maps • Self-assessment and planning for professional development
People create these things! IDENTIFYING COMPETENCIES
Identifying competencies intuitively What are the knowledge, skills and dispositions that a program evaluator needs to successfully complete the evaluation?
Evaluator relationships • A relationship exists among – The evaluator – The client – Program staff and other evaluation stakeholders • Primary responsibility for two tasks: – Making evaluation decisions – Implementing the evaluation
Evaluator relationships • Primary responsibility for making and implementing evaluation decisions may lie (1) With the evaluator (2) With someone else in the setting (3) Jointly with both • The evaluation relationship may shift during the study
Interactive participation quotient (IPQ) HIGH Program leaders, staff, community Involvement in decision making and implementation Evaluator LOW Evaluatordirected Collaborative ZONES Participantdirected
What do you believe? COMPETENCY STATEMENTS
Question: To what extent do your personal beliefs matter in a discussion of evaluator competencies?
Presenting. . . ESSENTIAL COMPETENCIES FOR PROGRAM EVALUATORS
How were the ECPE developed? • Began: With an idea in a graduate course • Then: “What does the literature tell us? ” • Then: A pilot study using the initial set of competencies, AJE publication (2001) • Then: A revision process including a crosswalk • Then: ECPE published in AJE (2005) • Then: Articles in CJPE (2005) and AJE (2006) • Then: CES’s professional designation program • Next: A national validation study
1) Professional practice Ø Fundamental norms and values of evaluation practice - Applies evaluation standards - Works ethically - Respects all stakeholders - Considers the general and public welfare - Contributes to the knowledge base
2) Systematic inquiry 1. Technical aspects of evaluation practice 1. Understands knowledge base 2. Knows quantitative and qualitative methods 3. Knows mixed methods 4. Conducts literature reviews 5. Specifies program theory 6. Frames evaluation questions 7. Designs the evaluation PLUS. . .
2) Systematic inquiry (cont. ) ØCollects and interprets data ØMakes judgments and recommendations ØProvides rationales for decisions ØReports procedures and results ØNotes evaluation strengths and limitations ØConducts meta-evaluations
3) Situational analysis Ø Unique interests, issues, and contextual circumstances of evaluation Describes program Ø Determines program evaluability Ø Identifies stakeholders’ interests Ø Serves intended users’ informational needs Ø Addresses conflicts Ø Attends to evaluation use PLUS. . . Ø
3) Situational analysis (cont. ) Ø Examines organizational and political context Ø Attends to organizational change Ø Respects site and client uniquenesses Ø Remains open to input Ø Modifies evaluation as needed
4) Project management Ø “Nuts and bolts” of evaluation work ØResponds to RFPs ØNegotiates with clients ØWrites formal agreements ØCommunicates with client ØBudgets evaluation ØJustifies cost PLUS. . .
4) Project management (cont. ) ØIdentifies needed resources ØUses technology as appropriate ØSupervises and trains others ØConducts evaluation in a nondisruptive manner ØPresents work in timely manner
5) Reflective practice Ø One’s own evaluation expertise and need for growth - Knows self - Reflects on practice - Pursues professional development: evaluation and content areas - Builds professional relationships
6) Interpersonal competence Ø “People skills” necessary for evaluation practice - Uses written and verbal/listening communication skills - Uses negotiation skills - Uses conflict resolution skills - Facilitates constructive interpersonal interaction - Demonstrates cross-cultural competence
A recent research study AN INITIAL STUDY TO DEVELOP INSTRUMENTS AND VALIDATE THE ECPE
Wilcox (2012) research study • Top- and lowest-rated competencies (rated on a scale of 1 [not at all necessary] to 5 [extremely necessary]) • Subject area differences not as important as evaluator role • Results of 9 expert interviews
Top weighted competencies (Wilcox, 2012) Label Competency Acts ethically and strives for integrity and honesty in conducting evaluations Respects clients, respondents, program 1. 4 participants, and other stakeholders Uses verbal/listening communication 6. 2 skills 2. 14 Interprets data 6. 1 Uses written communication skills 1. 2 Weighted Mean 4. 87 4. 76 4. 67 4. 63 4. 55
Lowest weighted competencies (Wilcox, 2012) Label Competency Conducts literature reviews Specifies program theory Responds to requests for 4. 1 proposals Contributes to the knowledge base 1. 6 of evaluation 2. 20 Conducts meta-evaluations 2. 5 2. 6 Weighted Mean 3. 61 3. 55 3. 20 3. 09 2. 70
Lowest weighted competencies (Wilcox, 2012) Label 2. 5 2. 6 4. 1 1. 6 2. 20 Competency Conducts literature reviews Specifies program theory Responds to requests for proposals Contributes to the knowledge base of evaluation Weight ed Mean 3. 61 3. 55 3. 20 3. 09 Conducts meta-evaluations 2. 70
Original and final interviewee categories Original Final Education evaluators Public health evaluators Evaluators in formal organizations Evaluators of non-profits Evaluations as independent consultants
Summary of expert interviews (Wilcox, 2012) ECPE Category In Education In Formal Organizations As Independent Consultants Professional practice Primary Systematic inquiry Essential [Team building] Essential [Collaboration] Very important [Reliance on assessment data] Interpersonal competence Very important
Summary of expert interviews (Wilcox, 2012) ECPE Category In Education In Formal Organizations As Independent Consultants Situational analysis Project management Seldom Sometimes Frequently Not so much All the time Reflective practice Wish to be able to do more regularly Senior staff’s responsibility Could do better Trying to keep up
Applying the competencies. . . 1. Back to the IPQ for discussion/reflection 2. A case application
Now the fun part. . . ECPE CONCEPT MAPPING
Work context and program evaluator competencies
Create a concept map-In your work context, what are the relationships between and among the competencies?
What’s next for you? SELF-ASSESSMENT
ECPE self-assessment Reflecting on your own practice, how do you rate your current knowledge, skills, and dispositions to effectively carry out program evaluations?
ECPE rating scale Entry / Novice Proficient/ Skilled Mastery/ Expert
Now what? What is your professional development plan?
References Ghere, G. , King, J. A. , Stevahn, L. , & Minnema, J. (2006). A professional development unit for reflecting on program evaluator competencies. American Journal of Evaluation, 27, 108 -123. Stevahn, L. , King, J. A. , Ghere, G. , & Minnema, J. (2005). Establishing essential competencies for program evaluators. American Journal of Evaluation, 26, 43 -59.
Feel free to contact me at kingx 004@umn. edu or see www. evaluation. umn. edu
- Washington evaluators
- Evaluators institute
- Characteristics of essential learning competencies
- Essential non essential fatty acids
- Jean-gabriel monnet
- Jean monnet jean-gabriel monnet
- The king is dead long live the king
- King ___________ of france called himself "the sun king."
- Kontinuitetshantering
- Novell typiska drag
- Nationell inriktning för artificiell intelligens
- Ekologiskt fotavtryck
- Varför kallas perioden 1918-1939 för mellankrigstiden?
- En lathund för arbete med kontinuitetshantering
- Adressändring ideell förening
- Tidbok
- A gastrica
- Densitet vatten
- Datorkunskap för nybörjare
- Boverket ka
- Tes debattartikel
- Magnetsjukhus
- Nyckelkompetenser för livslångt lärande
- Påbyggnader för flakfordon
- Lufttryck formel
- Svenskt ramverk för digital samverkan
- Bo bergman jag fryser om dina händer
- Presentera för publik crossboss
- Vad är ett minoritetsspråk
- Vem räknas som jude
- Treserva lathund
- Epiteltyper
- Bästa kameran för astrofoto
- Centrum för kunskap och säkerhet
- Byggprocessen steg för steg
- Mat för idrottare
- Verktyg för automatisering av utbetalningar
- Rutin för avvikelsehantering
- Smärtskolan kunskap för livet
- Ministerstyre för och nackdelar