Organizational Consultant Critical Friend and Evaluator The Value

Organizational Consultant, Critical Friend, and Evaluator: The Value and Challenge of Flexible Roles Kristine L. Chadwick, Ph. D. Evaluator for The Spatial Intelligence and Learning Center (SILC) The Science of Learning Center on Visual Language and Visual Learning (VL 2) www. edvantia. org

SILC and VL 2 Evaluations • NSF encouraged centers to change evaluators • Have served as SILC’s evaluator since its third year of operation (2008 -2009) • Served as VL 2’s evaluator during its fourth year of operation (2009 -2010) (and will continue in Years 6 -10) www. edvantia. org 2

What is SILC? (http: //spatiallearning. org) “The Spatial Intelligence and Learning Center (SILC) brings together scientists and educators from Temple University, Northwestern University, the University of Chicago, the University of Pennsylvania, and the Chicago Public Schools (CPS) to pursue the overarching goals of – understanding spatial learning – using this knowledge to develop programs and technologies that will transform educational practice, helping learners to develop the skills required to compete in a global economy SILC participants include researchers from cognitive science, psychology, computer science, education, and neuroscience, as well as practicing geoscientists and engineers who are particularly interested in spatial thinking in their fields, and teachers in the CPS. ” www. edvantia. org 3

What is VL 2? (http: //vl 2. gallaudet. edu) “The purpose of VL 2 is to gain a greater understanding of the biological, cognitive, linguistic, sociocultural, and pedagogical conditions that influence the acquisition of language and knowledge through the visual modality. Gallaudet University houses VL 2, and brings together deaf and hearing researchers and educators from a variety of disciplines and institutions to study how language and literacy develop in deaf individuals. ” www. edvantia. org 4

SILC and VL 2 Evaluation Priorities • Satisfy NSF evaluation “requirements” • Present well at Year 4 high-stakes site visit • Consult on center management issues • Gather and report data on aspects of center PIs knew less about (e. g. , outreach for SILC, training at affiliate IHEs for VL 2) www. edvantia. org 5

Articulating Center Theory • Theory of the center, not its research per se – Value added of center-mode research – Challenging for the center scientists • Immediate and intermediate outcome categories – – – 6 Centerness Dissemination Interdisciplinarity Outreach Training Tools and resources www. edvantia. org

Program Theory / Logic Model • Long-term Outcomes (> 10 years) – – Social Capital (reputation, networks) Intellectual Capital (research advancement) Physical Capital (tools and resources) Human Capital (diverse interdisciplinary researchers) • Impacts based on NSF goals – advancing science – applying science to specific technological, educational, and workforce challenges – enabling science to capitalize on new opportunities and discoveries and respond to new challenges www. edvantia. org 7

Evaluation Approach • Operationalizing, measuring (or documenting) resources, activities, outputs, immediate outcomes in logic model • Reflect their growth as centers • Consulting on center processes • Outsider viewpoint, critical friend www. edvantia. org 8

NSF Requirements • Center-level evaluation requirements not very clear – Show value added of center-mode functioning – Present outcomes of center at site visits • NSF need: To demonstrate to the National Science Board the worth or value of the Science of Learning Centers www. edvantia. org 9

Limited Resource Tensions External Evaluator • Show merit or worth of the center • Demonstrate how centermode functioning better than individual RO 1 grants • Focus on outcomes Consultant & Critical Friend • Fix some process gaps • Work through the logic model (are activities happening and resulting in expected outputs? ) • Attend to developmental stage of center • Focus on process and some immediate outcomes www. edvantia. org 10

Not enough resources to attend to all needs Caption for this vacation photo: “Up a Tree” www. edvantia. org 11

Conflicting Role Example • SILC Year 3 Evaluation – Process focused – Organizational consulting – Conversation and informal feedback and suggestions to Pis – SILC leadership satisfied – Site visit team wants outcome data www. edvantia. org 12

Example • SILC & VL 2 Year 4 Evaluation – Outcomes oriented – Bibliometric analyses • Limited corpus of publications • Not useful to center at this early stage – Less feedback mid-year – PIs less satisfied – Fewer questions/critiques from site visit team www. edvantia. org 13

Balancing Tensions in a Most Unusual Way • Years 6 -7 will bring back focus on process (new 5 -year funding cycle) • Update logic model and key measures • Focus also on intermediate outcomes (now appropriate in center’s lifecycle) • “SLC Evaluation Network” economies of scale may result in better – center-specific evaluation – SLC program-wide evaluation www. edvantia. org 14

Thank You! Kristine Chadwick, Ph. D. Executive Director Evaluation and Applied Research Initiatives 304. 347. 0429 kristine. chadwick@edvantia. org www. edvantia. org