In Insights Organizational cultures in public libraries Gatekeepers

In Insights: Organizational cultures in public libraries: Gatekeepers to effective outcome evaluation? Defining New Metrics For Libraries Success i. School, University of Toronto Symposium April 23, 2014 Dr. Bill Irwin, Huron University College Paul St-Pierre, PHD Candidate, FIMS, UWO

Project’s Goal The key goal of our research is to examine existing evaluation systems of public libraries, in an effort to replace the current reliance on a predominantly output based model of performance measure, with new models of evaluation based on capturing outcomes. 2 Organizational Culture & Evaluation

Project’s Goal Outcome models assist policy makers, both inside and outside of organizations, to address fundamental questions of resource allocation, delivery methods, and agency design and purpose. 3 Organizational Culture & Evaluation

Defining Outcome Evaluation 4 Organizational Culture & Evaluation

Project Objectives exploring the limitations and challenges inherent in the current qualitative evaluation system creating a better understanding of the cultural impacts and policy implications of a sociological performance framework, on library organizations, their constituencies, and their stakeholders in the establishment capture changes in the Practice-Program-Policy continuum and the downstream implications for how libraries are viewed and valued upon the introduction of a new system of performance measurement 5 Organizational Culture & Evaluation

OPLA Child and Youth Services Committee Teen Services Benchmarks and Statistical Report 2013 6 Does your library measure outcomes or impacts of teen programming? % Yes All 46. 7% 100, 000+ 38. 1% 50, 000<100, 000 40. 0% 15, 000<50, 000 46. 4% 5, 000<15, 000 75. 0% <5, 000 42. 9% Organizational Culture & Evaluation Less than 50% of the libraries reported measuring outcomes or impacts of teen programming, although several libraries did note that they are planning to implement outcomebased measurement in the future.

Q: Are you satisfied with the methods used by your library to evaluate teen programs? • It would be helpful to have a clearer idea of what a youth program should be i. e. what is the purpose, what is the desire outcome. • It is not consistent • We do not evaluate anything past attendance. • I would like to find out what the participant got out of program • Difficult to measure anecdotal reports • Evaluation is based only on circulation and attendance performance indicators 7 Organizational Culture & Evaluation

Q: Are outcomes/impacts measured in other parts of your library? • Lack of staff capacity • Do not have anyone trained in outcome/impact measurement • Probably the best answer for why it hasn't happened yet is "too time consuming. " • Lack of time. • Not currently considered a priority by our governing body • Assessing outcomes falls short of our priority list. • No formal criteria which is implemented system-wide 8 Organizational Culture & Evaluation

Q: You indicated that you do not assess outcomes/impacts of teen programs. Why not? Check all that apply. 9 Organizational Culture & Evaluation

Q: Do the agencies that fund your teen programs require that you report outcome/impact measures? 10 Organizational Culture & Evaluation

Organizational Culture and Impact Sub-cultures are currently the greatest inculcator of values (Schein, 1992) regarding: 1. the worth and purpose of evaluations, 2. postulation of negative influence, 3. evaluation seen as “busy work” 11 Organizational Culture & Evaluation

Sub-Culture as Inhibitor Common Responses from Followup Interviews Seen as “busy-work” Rely on informal feedback Some staff value the process, some question it, seen as additional work Against staff comfort level Comments are filtered –each manager chooses Staff wouldn’t want it (new system) to be more onerous on them Staff question why, “Are you evaluating me? ” 12 Organizational Culture & Evaluation

Inhibitors Lack of education, not able to produce meaningful results and change Lack of inclusion: “Big picture” relevance Required training in best practices Librarians feel skill set is inadequate to the task, so they are reluctant to engage – (Similar circumstances to the introduction of technology in mid to late 1990 s in public libraries (Author’s observation)) Technological impasse Consider the law of unintended consequences (how can/does it apply here? ) 13 Organizational Culture & Evaluation

Sustainable change requires cultural focusing (1) "Cultural change is one of the most important factors to consider but one of the hardest to implement" (Preston, 2004). “In general, we found a number of library staff skeptical of quantitative or qualitative data from customers, preferring instead to rely on their own assumptions and past practices to make decisions. ” (Hiller et al, 2008) “Due to the ‘everyone does the same thing’ culture and operational model it was impossible to make a change to workflow in one area without it directly impacting other areas - therefore systemic change was necessary. ” (Nussbaumer & Merkley, 2012) 14 Organizational Culture & Evaluation

Sustainable change requires cultural focusing (2) 15 Organizational Culture & Evaluation

Integration of Culture & Evaluation Model (1) The following model was developed to demonstrate the steps an organization goes through in regards to moving from an input based to an outcome based evaluation system. It highlights the operational and cultural transformation process. This model can be used to determine where an organization is situated (at what stage) in terms of its use of outcome evaluation. The model is designed to serve as a “roadmap”, to assist organizations in their successful movement through a series of steps, by infusing a culture of outcome evaluation. Once an organization has achieved stage 4, evaluation then serves as a tool for: realizing organizational priorities, educating stakeholders and funders of the range of quantitative and qualitative program and service impacts, and identifying operational issues and challenges. 16 Organizational Culture & Evaluation

Integration of Culture & Evaluation Model (2) Creating an outcome evaluation culture Stage 0 Stage 1 Stage 2 Stage 3 Stage 4 Purpose Complacency Justification Self Awareness Alignment Actualization Motivation Inertia Fear & Survival Toward sense of self-efficacy Enlightenment Success, Internalization Tactical: Short-term Dimension Organization al Impact Tactical / Strategic Organization al Impact Internal / External Inhibitors & Enablers Implications 17 Resistance Evaluation is ‘busy work’ Strategic: Long-term Buy-in Shift from systems to individual to patron/staff focus Habituation Inculcation Closed system Internal (staff) Toward an open system Broad stakeholder inclusion Infusion (beware of gaming) Lack of education, lack of meaningful results, inertia Lack of personal involvement. Buy-in not enough Cautious engagement Enthusiasm; Coordinating many voices Unintended consequences (positive or negative) Shared leadership Assessment is continual, naturally occurring Diminishing budgets, failure Awakening Organizational learning

Integration of Culture & Evaluation Model (3) Kotter (2008) argues that getting buy-in is not enough because it only engages the head, not the heart. Moving through the stages is akin to Kotter’s step 6, creating short term wins, building staff confidence, an incremental approach Somewhere in the transition between step 1 to 2 need to create and articulate a vision/mission statement. Why do we evaluate? What is it all about? Building cultural assessment seen as a strategic priority – otherwise over shadowed by more “important work” “When faculty and staff perceive that the administrators are only motivated to create a culture of assessment for accreditation, cynicism and low-motivation will likely result” (Lakos and Phipps, 2004). 18 Organizational Culture & Evaluation

Next Steps Formalize the Integration of Culture & Evaluation Model Create an organizational evaluation audit to: Assess the stage of the model an organization is at Develop strategies to assist moving it forward Create a sustainability approach Design and deliver the next phase of the research 19 Organizational Culture & Evaluation

Next Steps Q: Do the agencies that fund your teen programs require that you report outcome/impact measures? Ontario-wide survey to elected municipal officials focused on understanding what information they consider when in their deliberations on financial support to libraries’ budgets. In part this research will provide insight for libraries in how to structure, and communication the results of, their institutional evaluations. 20 Organizational Culture & Evaluation