How do we measure impact Developing a Framework

How do we measure impact? Developing a Framework for Measuring Reuse of Digital Objects Liz Woolcott Santi Thompson Elizabeth Kelly Caroline Muglia Genya O’Gara Ayla Stein Research Data Access & Preservation Summit Chicago, Illinois 2018

#digreuse Project Team Santi Thompson University of Houston sathomp 3@central. uh. edu | @Santi. Thompson Elizabeth J. Kelly Loyola University, New Orleans ejkelly@loyno. edu | @Elizabeth. Jelly Genya O’Gara Virtual Library of Virginia, George Mason Univ. gogara@gmu. edu | @genyao Ayla Stein University of Illinois Urbana-Champaign astein@illinois. edu | @The. Stacks. Cat Caroline Muglia Liz Woolcott University of Southern California Utah State University muglia@usc. edu liz. woolcott@usu. edu | @lizwoolcott reuse. diglib. org

Advisory Board Hannah Frost Emily Gore Krystyna K. Matusiak Sandra Phoenix Dorothea Salo Ali Shiri

What are your practices? How do you measure the use/reuse of your digital repository material?

Use vs. Reuse Use Reuse Discovering and browsing objets in a digital repository, often described as “clicks” or “downloads, ” without knowing the specific context for the use. How often and in what ways digital repository materials are utilized and repurposed, knowing the context of the use. Ex. Web analytics; CMS supplied analytics Ex. Social media use; data citation

Why study reuse? Digital Library Federation (DLF) Assessment Interest Group ○ Surveying the landscape: Use and Usability Assessment of Digital Libraries (2015 White Paper: https: //osf. io/9 nbqg ) Developing a Framework for Measuring the Reuse of Digital Objects ○ IMLS 2017 National Leadership Grant LG-7317 -0002 -17

Grant Outline Project Deliverables Implementation

Project Deliverables Products Functional requirements and use cases for a reuse assessment toolkit: ○White paper o Sustainable and vetted assessment techniques o Methods to deepen understanding of users o Developing approaches and best practices to communicate impact ○Grant Report ○Project Data Where to find them: Website: https: //reuse. diglib. org OSF Repository: https: //osf. io/y 5 tjh

Implementation Surveys Focus groups Foundational Survey In person ○Fall 2017 ○ Digital Library Federation (Fall 2017) ○ CODE 4 LIB (Winter 2018) Follow up Survey ○ Personal Digital Archiving (Spring 2018) ○Spring 2018 Virtual ○ December 2017 ○ March 2018 ○ April 2018 (tentative)

Survey Administration Findings

19 questions 10 “core” questions 302 responses “usable” reponses, that is (409 total, 130 core) 56% academic Largest response group. Next largest: 10% public libraries

Do you collect USE statistics?

Do you collect REUSE statistics?

Motivations for Assessment Top reasons for gathering assessment metrics ○ Improve digital collections services ○ Demonstrate the impact of collections on scholarship and research ○ Prove the return on investment ○ Improve collection selection decisions

Assessment Platforms Use ○ Google Analytics ○ CMS Platform statistics Reuse ○ Social media metrics ○ Alert Services ○ Reverse Image Lookup

Assessment Data Use ○ Number of visitors ○ Number of downloads ○ Number of clicks Reuse ○ Citations in scholarship ○ Publish/re-posted digital objects ○ Published or replicated in physical media (e. g. print books)

Barriers to Assessment Use ○ No/New digital Reuse ○ Lack of methodology or collections system ○ ○ ○ Lack of staff Other responsible parties ○ Not a priority Lack of staff/time Not a focus (of institution) Lack of ability or resources

“ “. . . collating data from [digital collection] use and or re-use would cost manpower hours that the board of directors would not understand or approve. ”

Top 10 barriers to assessment

“ “Most of our understanding of reuse comes anecdotally. We haven't figured out a good quantitative way to measure reuse that isn't too labor intensive for us. This is a pain point, since reuse is highly valued by stakeholders!”

Focus Groups Structure Findings

Focus Group Structure Focus Group Type Sessions When Participants Focus Group 1 In person 2 October 2107 9 Information Gathering Group 2 Virtual 2 December 2017 7 Information Gathering Group 3 In person 2 February 2018 8 Technology/ Standards Group 4 Virtual 2 March 2018 7 Technology/ Standards Group 5 In person 1 April 2018 6 Cultural/ Privacy Group 6 Virtual 1 April 2018 TBD Cultural/ Privacy

Focus Group Findings Need for technology to track their digital content, such as: ○ “Sticky” metadata ○ Data dashboards ○ Unique identifiers for all digital objects

Focus Group Findings Need for common standards and training: ○ Standards for gathering use/reuse data ○ Turning data into impact statements and stories ○ Guidelines on making collection selection/digitization decisions

Focus Group Findings Need for common repositories/data sharing ○ Sharing collections data □ Success and failures ○ Shared usage/reusage data repositories

Focus Group Findings Need for cultural understanding and privacy ○ Recognizing the cultural and ethical implications of our assessment practices ○ Protecting the privacy of the communities we serve

Key Takeaways Toolkit Training Common Standards To pull common standards, use cases, trainings, and resources into one central location. To support staff/time investment to enable institutions of any level to gather assessment metrics. To ensure the profession is measuring gathering and comparing the same kind of data. Technology Privacy Common Repository To support and streamline assessment processes, such as dashboards and sticky metadata. To provide conscientious assessment data that protects the privacy of the communities we serve. To help examine the larger trends in digital repositories to support funding requests and show impact.

#digreuse Project Team Santi Thompson University of Houston sathomp 3@central. uh. edu | @Santi. Thompson Elizabeth J. Kelly Loyola University, New Orleans ejkelly@loyno. edu | @Elizabeth. Jelly Genya O’Gara Virtual Library of Virginia, George Mason Univ. gogara@gmu. edu | @genyao Ayla Stein University of Illinois Urbana-Champaign astein@illinois. edu | @The. Stacks. Cat Caroline Muglia Liz Woolcott University of Southern California Utah State University muglia@usc. edu liz. woolcott@usu. edu | @lizwoolcott reuse. diglib. org

THANKS! Any questions? @lizwoolcott #digreuse liz. woolcott@usu. edu

Credits Special thanks to all the people who made and released these awesome resources for free: ○ Presentation template by Slides. Carnival ○ Photographs by Unsplash ○ Backgrounds by Subtle. Patterns