In Search of Evaluator Competency Inventories Jeanette Gurrola

In Search of Evaluator Competency Inventories Jeanette Gurrola jeanette. gurrola@cgu. edu Psychology Department School of Behavioral & Organizational Sciences Claremont Graduate University American Evaluation Association Conference 2010

Kirkhart: Defining evaluator competencies: New light on an old issue Scriven: Types of evaluation and types of evaluator Stevahn et al. : Establishing essential competencies for program evaluators 2009 Guiding Principles for Evaluators are published 2001 The American Evaluation Association is formed 1995 2009 1986 1980 2005 Program Evaluation Standards are published 1996 1981 1994 Evaluator Competencies CES: Competencies for Canadian evaluation practice King et al. : Toward a taxonomy of essential evaluator competencies

Competency Domains Kirkhart (1981) Scriven King et al. Stevahn et al. (1996) (2001) (2005) Empirically derived Competency Domains Interpersonal practice Professional ethics Project management Reflective practice Situational analysis Systematic inquiry X X X CES (2009) X X X X X X

Research Questions Do evaluation approaches emphasize that evaluators be competent in certain skills and knowledge? Additionally, do evaluation approaches vary on which evaluator competencies they emphasize?

Donaldson Christie & Alkin (2008) Evaluation Theory Tree

Theorists Donaldson ◦ Methods branch ◦ Theory-driven program evaluation Greene ◦ Values branch ◦ Responsive, inclusive, mixed methodology King ◦ Use branch ◦ Evaluation capacity building

Sample and Methodology Refereed journal articles Published in the past ten years Practical application of an evaluation Content analysis Stevahn et al. (2005) ECPEs ◦ 61 competencies across 6 domains ◦ Rated degrees of referenced Not referenced, extensively referenced Descriptive and Chi-square analyses

Competency Domains Professional Practice Systematic Inquiry Situational Analysis Project Management Reflective Practice Interpersonal Competence

Competency Domains � � � Professional Practice Systematic Inquiry Situational Analysis Project Management Reflective Practice Interpersonal Competence • Applies professional evaluation standards • Acts ethically and strives for integrity and honesty in conducting evaluations • Conveys Personal evaluation approaches and skills to potential clients • Respects clients, respondents, program participants, and other stakeholders • Considers the general and public welfare in evaluation practice • Contributes to the knowledge base of evaluation

Competency Domains � � � Professional Practice Systematic Inquiry Situational Analysis Project Management Reflective Practice Interpersonal Competence • Understands the knowledge base of evaluation • Knowledgeable about methodology • Conducts literature reviews • Specifies program theory • Frames evaluation questions • Develops evaluation designs • Identifies data sources • Collects data • Assesses validity and reliability of data • Analyzes data • Interprets data • Makes judgments • Develops recommendations • Provides rationales for decisions throughout the evaluation • Reports evaluation procedures and results • Notes strengths and limitations of the evaluation • Conducts meta-evaluations

Competency Domains � � � Professional Practice Systematic Inquiry Situational Analysis Project Management Reflective Practice Interpersonal Competence • Describes the program • Determines program evaluability • Identifies the interests of relevant stakeholders • Serves the information needs of intended users • Addresses conflict • Examines the organizational context of the evaluation • Analyzes the political consideration relevant to the evaluation • Attends to the issues of evaluation use • Attends to the issues of organizational change • Respects the uniqueness of the evaluation site and client • Remains open to input from others • Modifies the study as needed

Competency Domains � � � Professional Practice Systematic Inquiry Situational Analysis Project Management Reflective Practice Interpersonal Competence • Responds to request for proposals • Negotiates with clients before the evaluation begins • Writes formal agreements • Communicates with clients throughout the evaluation process • Budgets an evaluation • Justifies costs given information needs • Identifies needed resources for evaluation • Uses appropriate technology • Supervises others involved in conducting the evaluation • Trains others involved in conducting the evaluation • Conducts the evaluation in a nondisruptive manner • Presents work in a timely manner

Competency Domains � � � Professional Practice Systematic Inquiry Situational Analysis Project Management Reflective Practice Interpersonal Competence • Aware of self as an evaluator • Reflects on personal evaluation practice • Pursues professional development in evaluation • Pursues professional development in relevant content areas • Builds professional relationships to enhance evaluation practice

Competency Domains � � � Professional Practice Systematic Inquiry Situational Analysis Project Management Reflective Practice Interpersonal Competence • Uses written communication skills • Uses verbal/listening communication skills • Uses negotiation skills • Uses conflict resolution skills • Facilitates constructive interpersonal interaction • Demonstrates cross-cultural competence

a Stewart Donaldson Competency Domains Absent Present Professional Practice 67% 33% Systematic Inquiry 50%2 Situational Analysis 77% 23% Project Management 75% 25% Reflective Practice 90% 10% Interpersonal Competence 67% 33% Total 67% 33% Pearson X 2 statistically significant at. 05 level. b More than 25% of cells have an expected count less than 5. 1 Observed percentage statistically significantly less than expected percentage. 2 Observed percentage statistically significantly greater than expected percentage. a

ab Stewart Donaldson Degrees of Referenced Competency Domains Absent Referenced Extensive Professional Practice 67% 25% 8% Systematic Inquiry 50% 19% 31% Situational Analysis 77% 4% 19% Project Management 75% 19% 6% Reflective Practice 90% 5% 5% Interpersonal Competence 67% 4% 29% Total 67% 14% 19% Pearson X 2 statistically significant at. 05 level. b More than 25% of cells have an expected count less than 5. 1 Observed percentage statistically significantly less than expected percentage. 2 Observed percentage statistically significantly greater than expected percentage. a

Stewart Donaldson Competency Domains % Extensive Systematic Inquiry 31% Interpersonal Competence 29% Situational Analysis 19% Professional Practice 8% Project Management 6% Reflective Practice 5%

Stewart Donaldson Specific Competencies (4 articles) Competency Domains % Extensive Systematic Inquiry 31% Interpersonal Competence 29% Situational Analysis 19% Professional Practice 8% Project Management 6% Reflective Practice 5% • (4) Collects data • (2) Knowledgeable about qualitative methods • (2) Conducts literature reviews • (2) Specifies program theory • (2) Frames evaluation questions • (2) Develops evaluation design • (2) Identifies data sources • (2) Assesses validity of data • (2) Assesses reliability of data • (2) Analyzes data

Stewart Donaldson Specific Competencies (4 articles) Competency Domains % Extensive Systematic Inquiry 31% Interpersonal Competence 29% Situational Analysis 19% Professional Practice 8% Project Management 6% Reflective Practice 5% • (2) Uses verbal/listening communication skills • (2) Demonstrates cross-cultural competence

a Jennifer Greene Competency Domains Absent Present Professional Practice 71% 29% Systematic Inquiry 58% 42% Situational Analysis 38%1 62%2 Project Management 62% 38% Reflective Practice 89%2 11%1 Interpersonal Competence 60% 41% Total 59% 41% Pearson X 2 statistically significant at. 05 level. b More than 25% of cells have an expected count less than 5. 1 Observed percentage statistically significantly less than expected percentage. 2 Observed percentage statistically significantly greater than expected percentage. a

a Jennifer Greene Degrees of Referenced Competency Domains Absent Referenced Extensive Professional Practice 71% 17% 12% Systematic Inquiry 58% 14% 29% Situational Analysis 38%1 21% 41%2 Project Management 62% 30%2 8%1 Reflective Practice 89%2 9% 3%1 Interpersonal Competence 60% 31% 10% Total 59% 20% 21% Pearson X 2 statistically significant at. 05 level. b More than 25% of cells have an expected count less than 5. 1 Observed percentage statistically significantly less than expected percentage. 2 Observed percentage statistically significantly greater than expected percentage. a

Jennifer Greene Competency Domains % Extensive Situational Analysis 41% Systematic Inquiry 29% Professional Practice 12% Interpersonal Competence 10% Project Management 8% Reflective Practice 3%

Jennifer Greene Specific Competencies (7 articles) Competency Domains % Extensive Situational Analysis 41% Systematic Inquiry 29% Professional Practice 12% Interpersonal Competence 10% Project Management 8% Reflective Practice 3% • (5) Describes the program • (5) Serves the information needs of intended users • (4) Identifies the interest of relevant stakeholders • (4) Attends to the issues of evaluation use

Jennifer Greene Specific Competencies (7 articles) Competency Domains % Extensive Situational Analysis 41% Systematic Inquiry 29% Professional Practice 12% Interpersonal Competence 10% Project Management 8% Reflective Practice 3% • (7) Collects data • (6) Knowledgeable about quantitative methods • (5) Reports evaluation procedures and results

a Jean King Competency Domains Absent Present Professional Practice 75%2 25%1 Systematic Inquiry 38% 63% Situational Analysis 21%1 79% Project Management 50% Reflective Practice 45% 55% Interpersonal Competence 21% 79% Total 39% 61% Pearson X 2 statistically significant at. 05 level. b More than 25% of cells have an expected count less than 5. 1 Observed percentage statistically significantly less than expected percentage. 2 Observed percentage statistically significantly greater than expected percentage. a

a Jean King Degrees of Referenced Competency Domains Absent Referenced Extensive Professional Practice 75%2 8% 17% Systematic Inquiry 38% 25% Situational Analysis 21%1 21% 58%2 Project Management 50% 27% 23% Reflective Practice 45% 30% 25% Interpersonal Competence 21% 50% 29% Total 39% 30% 31% Pearson X 2 statistically significant at. 05 level. b More than 25% of cells have an expected count less than 5. 1 Observed percentage statistically significantly less than expected percentage. 2 Observed percentage statistically significantly greater than expected percentage. a

Jean King Competency Domains % Extensive Situational Analysis 58% Interpersonal Competence 29% Systematic Inquiry 25% Reflective Practice 25% Project Management 23% Professional Practice 17%

Jean King Specific Competencies (4 articles) Competency Domains % Extensive Situational Analysis 58% Interpersonal Competence 29% Systematic Inquiry 25% Reflective Practice 25% Project Management 23% Professional Practice 17% • (4) Identifies the interest of relevant stakeholders • (4) Attends to the issues of evaluation use • (4) Remains open to input from others • (3) Determines program evaluability • (3) Addresses conflict • (2) Describes the program • (2) Examines the organizational context of the evaluation • (2) Attends to the issues of organizational change

Jean King Specific Competencies (4 articles) Competency Domains % Extensive Situational Analysis 58% Interpersonal Competence 29% Systematic Inquiry 25% Reflective Practice 25% Project Management 23% Professional Practice 17% • (3) Facilitates constructive interpersonal interaction • (2) Demonstrates cross-cultural competence

Domain Emphases SA 60% Professional Practice 50% Systematic Inquiry SA 40% SI IC 30% IC SI 1 20% Project Management Reflective Practice 10% 0% Situational Analysis S. Donaldson J. Greene J. King Interpersonal Competence

Approach by Competencies Donaldson ◦ Theory-driven program evaluation Emphasis on Systematic Inquiry Greene ◦ Responsive, inclusive, mixed methods approach Emphasis on Situational Analysis Lack of emphasis on Reflective Practice King ◦ Evaluation capacity building Emphasis on Situational Analysis Lack of emphasis on Professional Practice

Limitations and Implications Representative to only the three theorists Articles not written with competencies in mind Focus on specific competencies Inform training, professional development, education, etc. Contributes to research on evaluation Future research needed to expand findings on evaluation approaches

References Canadian Evaluation Society (2009, May 31). Companion Document for Competencies for Canadian Evaluation Practice. Retrieved from http: //www. evaluationcanada. ca/txt/20090531_competencies_companion. pdf. Christie, C. A. & Alkin, M. C. (2008). Evaluation theory tree re-examined. Studies in Educational Evaluation, 34, 131 -135. King, J. A. , Stevahn, L. , Ghere, G. , & Minnema, J. (2001). Toward a taxonomy of essential evaluator competencies. American Journal of Evaluation, 22 (2), 229 -247. Kirkhart, K. (1981). Defining evaluator competencies: New light on an old issue. American Journal of Evaluation, 2, 188 -192. Perrin, B. (2005). How can information about the competencies required for evaluation be useful? The Canadian Journal of Program Evaluation, 20 (2), 169 -188. Mc. Guire, M. & Zorzi, R. (2005). Evaluator competencies and performance development. The Canadian Journal of Program Evaluation, 20 (2), 73 -99. Scriven, M. (1996). Types of evaluation and types of evaluator. American Journal of Evaluation, 17 (2), 151 -161. Stevahn, L. , King, J. A. , Ghere, G. , & Minnema, J. (2005). Establishing essential competencies for program evaluators. American Journal of Evaluation, 26 (1), 4359.

Contact Jeanette Gurrola jeanette. gurrola@cgu. edu Psychology Department School of Behavioral & Organizational Sciences Claremont Graduate University