Why assess What is assessment What do we

Why assess? What is assessment? What do we mean by actionable data? Martha Kyrillidou Planning for Results: Making Data Work for You The Cato Institute, Washington, DC February 20, 2008

What’s bringing us together … Association of Research Libraries http: //www. arl. org/stats

Overview 1. Initiate (Why do we assess? ) 2. Plan (What do we assess? ) 3. Design “Why assess? What is assessment? What do we mean by actionable data? ” - Martha Kyrillidou (1&2) “Survey Design” – Neal Kaske (2&3) “Performance Measures” – Rick Garcia (2&3) (How do we assess? ) 4. Implement “Analysis and Interpretation” - Ray Lyons (5) (Design in action) 5. Analyze and Interpret (What do the results mean? ) Lib. QUAL+®, MINES™, and Other Tools; Packaging Results to Communicate to Stakeholders Gary Roebuck and Kristina Justh (2, 3, 4, &5) 6. Reflect and Initiate (What do we do with the results? ) Association of Research Libraries http: //www. arl. org/stats

Clear and present danger • Bangor University considers removing librarians posted by Blake on Thursday January 27, 2005 @07: 30 AM -753 hits Ms Information writes "News from the University of Wales Bangor in the UK. senior management no longer feel that subject librarians / academic liaison librarians are needed in the modern academic library. They have made restructuring proposals which include removing all but one of the subject librarians and a tier of the library management, including the Head of Bibliographic Services. The university management thinks that technology has 'deskilled' literature searching. As far as I know, this proposal is unprecedented in the United Kingdom. In essence, there will remain 4 professional librarians serving a 'research-led' university of 8, 000 plus FTEs and with 8 library sites. These will be the university librarian, cataloguing librarian, acquisitions librarian and Law librarian. • Has anything like this happened anywhere that you know of? If so, what have been the effects? Association of Research Libraries http: //www. arl. org/stats

Assessment Librarian

Library of the Future Will Also Need. . . To have it’s own data collection and management personnel, individuals who constantly collect, analyze and prepare reports on data regarding what services are being used, which portions of the collection are getting the highest usage, what materials are being lent through interlibrary loan, and who patrons are. Documenting the libraries contributions to quality teaching, student outcomes, research productivity will become critical. Association of Research Libraries http: //www. arl. org/stats

Technical Skills needed • Quantitative Analysis: SPSS • Qualitative Analysis: Atlas. ti • Data collection: Custom Web interface, Survey Monkey, Form. Spring • Data storage: SQL Server, Excel • Reporting: Crystal Reports, SPSS, SQL Reporting Services, Excel, Word • Web delivery: Dundas Charts • Web development: ASP. NET, C#, Cold. Fusion Association of Research Libraries http: //www. arl. org/stats

Communication Skills needed • Communicating organizational vision and service ethic • Leadership in capturing the future and making it happen • Ability to take initiative and willingness to take risks and learn from mistakes • Troubleshooting and being an independent thinker • Ability to work collaboratively and move forward team and organizational goals Association of Research Libraries http: //www. arl. org/stats

. . . chart a course for every endeavor that we take the people's money for, see how well we are progressing, tell the public how we are doing, stop the things that don't work, and never stop improving the things that we think are worth investing in. – President William J. Clinton, on signing the Government Performance and Results Act of 1993 Association of Research Libraries http: //www. arl. org/stats

What’s driving the ‘assessment’ agenda • “Times are changing” – Exploding growth in use and applications of technology – Increased customer expectations for services, including quality and responsiveness • Shrinking budgets – Justification for spending $$$ on resources, programs, and services – Increasing competition for resources – Cost savings and/or reallocation – Fighting to remain relevant Association of Research Libraries http: //www. arl. org/stats

The Internet Goes to College Early data from ethnographic interviews – “I use Google because I heard it searches for more things” (than other sources). – “I believe I can find anything on the Internet. There hasn’t been anything I haven’t been able to find. ” – “Because I’m lazy. ” – Books have “so much information that no one can go through it all. ” – I use “the Internet first because it is more convenient. ” – I go to the library “because that’s what teachers like. ” – “Google has gotten me through college. ” Source: Steve Jones, The Internet Goes to College, ARL Talk Association of Research Libraries http: //www. arl. org/stats

Libraries Remain a Credible Resource in 21 st Century 98% agree with statement, “My … library contains information from credible and known sources. ” Note. Digital Library Federation and Council on Library and Information Resources. (2002). Dimensions and Use of the Scholarly Information Environment. Association of Research Libraries http: //www. arl. org/stats

OCLC Reports • Sharing, Privacy and Trust in Our Networked World • College Student’s Perceptions: Libraries and Information Resources • Perceptions of Libraries and Information Resources » OCLC: http: //www. oclc. org/reports/sharing/default. htm Association of Research Libraries http: //www. arl. org/stats

‘Universal’ library brand • Amazing stability of library user’s desired expectations across – – Types of users Languages Countries Some differences across disciplines • 297, 158 Lib. QUAL+® survey respondents Association of Research Libraries http: //www. arl. org/stats

Towards what end The emerging library user sees the need for a different kind of library – a library that serves as a source of information ready to be accessed any time, from anywhere, using modern technologies. Can we effectively translate the comfort and assurance of human interactions into a virtual reality that is trustworthy, reliable, and ever present? And will learning be quite as effective or deep in this virtual world of never-ending interaction with the collective digital-knowledge base? • Thompson, Kyrillidou, Cook “Library Users’ Service Desires” Library Quarterly (2008) Association of Research Libraries http: //www. arl. org/stats

Library Assessment and its Global Dimensions • Markets and people exposed to economic and social frameworks unheard of before • Competing internationally • Library users exposed to global forces • Libraries facing similar challenges • Libraries as the Internet • Libraries as Google • Libraries as Collaborative Spaces Association of Research Libraries http: //www. arl. org/stats

Defining Success in a Digital Environment • Crafting new measures of success. • Moving from measuring inputs to outputs and outcomes • Understanding impact of library roles and services. • Agreeing on qualitative measures of success: user perceptions, user success, creating value, advancing organizational goals. • Reallocating and managing capabilities to focus on new definitions of success. Association of Research Libraries http: //www. arl. org/stats

Thinking Strategically About Libraries’ Futures • What is the central work of the library and how can we do more, differently, and at less cost? • What important set of services does the library provide that others can’t? What new roles are needed? • What advantages does the library possess? • What will be the most needed by our community of users in the next decade? How is user behavior changing? • What should our libraries aspire to be ten years from now? What are the implications of technology driven change? • What are the essential factors responsible for the success of the library? Association of Research Libraries http: //www. arl. org/stats

Library Assessment Library assessment provides a structured process to learn about our communities, their work and the libraries connection to what they do The information acquired through library assessment is used in an iterative manner to improve library programs and services and make our libraries responsive to the needs of our communities. Academic libraries do not exist in a vacuum but are part of a larger institution. Assessment within the institution may take place in individual areas as well as at the broad institutional level. Association of Research Libraries http: //www. arl. org/stats

Assessment is… “…a structured process to learn about communities and evaluate how well the library supports them. ” - Steve Hiller, Director of Assessment & Planning University of Washington Libraries Association of Research Libraries http: //www. arl. org/stats

Free speech wall, Charlottesville, Sept 2006 Association of Research Libraries http: //www. arl. org/stats

Evaluation – foci by Peter Brophy We have 2 million records Inputs We have the technology to transform data into information -rich products Processes Outputs 26 students have read this paper Outcomes As a result of our service, humankin d has beaten malaria Impacts We have delivered one million document s Increasing difficulty ………… Increasing usefulness ………… Association of Research Libraries http: //www. arl. org/stats

IFLA: Measuring Quality • Resources, infrastructure: What services does the library offer? • Use: How are the services accepted? • Efficiency: Are the services offered costeffectively? • Potentials and Development: Are there sufficient potentials for future development? (an adaptation of the Balanced Score Card) Association of Research Libraries http: //www. arl. org/stats

Assessment at ARL • A gateway to assessment tools: Stats. QUAL®: – ARL Statistics -- E-Metrics – Lib. QUAL+® -- Service Quality and Satisfaction – Digi. QUAL® – MINES for Libraries® • • • Library Assessment Conferences – libraryassessment. org Library Assessment Blog – libraryassessment. info SPEC Kits Service Quality Evaluation Academy ESP Assessment (Making Library Assessment Work) – Effective, Sustainable, Practical Association of Research Libraries http: //www. arl. org/stats

Qualitative Profile Developing New Metrics (per Yvonna Lincoln) • • Uniqueness of collections Defining the value of consortia Administrative and budgetary efficiencies Student outcomes/student learning/graduate success Contributions to faculty/stakeholders productivity Social frameworks/intellectual networks Generating new knowledge Creating the collective good with reusable assets Association of Research Libraries http: //www. arl. org/stats

How do you describe a Library? • Breadth and quality of collections and services • Sustained institutional commitment to the library • Distinctive resources in a variety of media • Services to the scholarly community • Preservation of research resources • Contributions of staff to the profession • Effective and innovative use of technology • Engagement of the library in academic planning Association of Research Libraries ‘Principles of Membership’ Association of Research Libraries http: //www. arl. org/stats

Group exercise • Share important aspects of your – Services – Collections – Collaborative Relations Association of Research Libraries http: //www. arl. org/stats

Tell the story … • • Facilities: number and type (library buildings, classrooms, studios, computer labs, cafes, offices outside of the library, etc. ) University centers, services and responsibilities (managed by the library or housed in the library): slide libraries, media centers, teaching and learning centers, education centers, writing centers, distance ed or online course development/instructional design, i. Tunes. U, laptop support, technology help desk, escholarship centers, university press or other publishing entity, university archives, university records management, etc. Digital collections: institutional/digital repositories, images, articles, simulations, data, learning objects, etc. Grants and development: numbers, types, friends organizations, advisory boards/councils, etc. Partnerships: nature and number, on campus and external Consortial memberships and roles Outreach: web hits, computer use hours, learning hours/contact/impacts (in classrooms, Info Lit, etc. ), visits, external teaching, volunteering, programs offerings, sponsored lectures, etc. Leadership: on campus and external committees, boards, initiatives, projects, etc. Publications/scholarship: numbers, types (articles, books, presentations, poster sessions, performances, etc. ) » Email from ARL Library Director – February 13, 2008 Association of Research Libraries http: //www. arl. org/stats

Measuring Impact • Concerns Based Adoption Model (CBAM) – Stages of concern • How engaged are subjects in the service /innovation? • How much interest do they show? – Levels of use • How much use do they make? • Avoid dichotomy of user / non-user – Levels of Impact • Draw together concern and use • Assess effects on behaviour » Peter Brophy (South Africa, August 2007) Association of Research Libraries http: //www. arl. org/stats

Measuring Impact • Surface learning – Skim, memorise, regurgitate for tests • Little long-term impact • Deep learning – Involved, questioning, interactive, integration with own knowledge • Significant long-term impact • A habit of critical thinking » Peter Brophy (South Africa, August 2007): Association of Research Libraries http: //www. arl. org/stats

“If you think you're too small to have an impact, try going to bed with a mosquito. ” » Anita Roddick Association of Research Libraries http: //www. arl. org/stats

Actionable Data • What is the desired outcome of assessment? – An organization that strives to be customer-centered and acts on information about environmental factors and especially on user input. » Raynna Bowlby and Martha Kyrillidou Association of Research Libraries http: //www. arl. org/stats

Align with mission, vision and goals of parent organization – Are there prominent themes/directions – research support, instruction, physical facilities; priority user groups; expansion and growth or conservation and reallocation of resources; emphasis on current technologies and methods or more traditional approaches – Are there strategically favored programs/initiatives and/or individual champions with key agendas? – Are there stated goals for the organization and the library? Association of Research Libraries http: //www. arl. org/stats

Address users’ top priorities • Are you undertaking initiatives that users indicate are their highest wants and are you doing enough? • What budget/resources are being re/allocated to these initiatives? • Are enough budget/resources being directed to these initiatives? • Have the departments/staff that most affect these initiatives been informed? • Do these departments/staff understand the data (and comments/specifics) and have goals and improvement plans in place? • Are there improvement targets; how is change being measured? Association of Research Libraries http: //www. arl. org/stats

Identify resources that can be reallocated • Is the work that you are performing addressing areas of strong dissatisfaction? • Are you doing too much of what you’ve always done, rather than the work that results in user wants & needs? • What budget/resources are being allocated to legacy work that is not a user priority and is it too much? • Have the departments/staff that are currently doing this work been informed of changing user wants and needs? • Do these departments/staff understand the data (and comments/specifics) and can they redirect their time and resources? • Are there improvement targets; is change being measured? Association of Research Libraries http: //www. arl. org/stats

Continuous improvement and assessment • How will you incorporate the results of your analysis and findings into your Library’s strategic and annual plans? • How can you develop some of your more resourceintensive improvements over several years? • How can you insure that your assessment results are always available when and where decisions are being made? • How and when will you be assessing the user improvements implemented after your user survey? Association of Research Libraries http: //www. arl. org/stats

The importance of accurate data in decision making Association of Research Libraries http: //www. arl. org/stats

Your library … • Organization and use of information is focused on disciplines and increasingly interdisciplinary centers • Subject-oriented staff provide information services required and integrate them in the research process • Developing and maintaining relationships with key stakeholders in the disciplines/departments • Acquainting specialized stakeholders with the broad range of information and information services available outside their specialty • Assuring adequate performance of network and technology based information services • Assuring adequate performance of the centralized basic support functions Association of Research Libraries http: //www. arl. org/stats

Exercise • What are the key indicators of this library's work? • How do you monitor each indicator? How do you record the data for each indicator? With what frequency? • To whom and how do you communicate information about the library's performance? Association of Research Libraries http: //www. arl. org/stats

Association of Research Libraries http: //www. arl. org/stats

Making Library Assessment Work • ARL project approved in 2004 and funded by participating libraries • Site visits by Steve Hiller (U of Washington) and Jim Self (U of Virginia) – Presentation – Interviews and meetings – Report to the Library • 24 libraries in U. S. and Canada visited in 2005 -06 • Effective, Sustainable and Practical Library Assessment in 2007 – Open to all libraries – 6 libraries participated in 2007 Association of Research Libraries http: //www. arl. org/stats

What We Found • Strong interest in using assessment to improve customer service and demonstrate value of library • Many libraries uncertain on how to establish, maintain, and sustain effective assessment • Effectiveness of assessment program not dependent on library size or budget • Each library has a unique culture and mission. No “one size fits all” approach works. • Strong customer-focus and leadership support were keys to developing an effective and sustainable assessment Association of Research Libraries http: //www. arl. org/stats

What are the lessons learned? • Understanding changes in users approach to information resources. • Service quality improvement is a key factor. • Understanding the impact of e-resources on library services - TRL. • Learning how to compete or collaborate with Google. • Upfront investment in design and development. • Making the assessment service affordable, practical, & effective. • Assessment needs to be satisfying and fun. Association of Research Libraries http: //www. arl. org/stats

Multiple methods • • • Transactional surveys* Mystery shopping New, declining, and lost-customer surveys Focus group interviews Customer advisory panels Service reviews Customer complaint, comment, and inquiry capture Total market surveys* Employee field reporting Employee surveys Service operating data capture *A SERVQUAL-type instrument is most suitable for these methods Note. A. Parasuraman. The SERVQUAL Model: Its Evolution And Current Status. (2000). Paper presented at ARL Symposium on Measuring Service Quality, Washington, D. C. Association of Research Libraries http: //www. arl. org/stats

Assessment can… • • Support decision making Establish accountability Strengthen organizational mission, goals, and objectives Realign library goals and mission statements with the needs of users • Identify trends in user behaviors and technology • Measure learning outcomes and impact • Improve programs and services Association of Research Libraries http: //www. arl. org/stats

Use Assessment Data for Good! Association of Research Libraries http: //www. arl. org/stats

Association of Research Libraries http: //www. arl. org/stats

What’s in a word? quality library What makes a ? “Quality much like beauty is in the eye of the beholder” Association of Research Libraries http: //www. arl. org/stats

What’s in a “Library”? A word is not crystal, transparent and unchanged; it is the skin of a living thought, and may vary greatly in color and content according to the circumstances and time in which it is used. --Justice Oliver Wendell Holmes Association of Research Libraries http: //www. arl. org/stats

Interpreting Data Association of Research Libraries http: //www. arl. org/stats