Putting Evidencebased Practice to Work Usability Testing at

Putting Evidence-based Practice to Work: Usability Testing at Northwestern University Library Frank Cervone AUL for Information Technology Monday October 29, 2007 Internet Librarian 2007

Overview n n The problem of web site design Using more formal methods Formal methods in practice Outcomes

The problems of web site design n Web development/HCI is an intricate mix of technology and design – While there are exceptions, the majority of librarians haven’t been trained in HCI – Gaps in understanding the significant differences between the on-line and in-person experience n Once a site is done, most people are not eager to have others suggest adjustments or changes, particularly to those things they “know” are right Liberally adapted from Problems with in-house website development by “The IT Guy” http: //www. newfangled. com/contentmgr/showdetails. php/id/7441

Web site design and organizational responsiveness n n The more an organization depends upon its publics for achieving its mission, the more it should employ dialogic features into its Web site design The challenge is to move the tasks associated with Web site design and maintenance away from being a "B-list" job to being an imperative for the survival of highly stakeholder-dependent organizations Kent, M. L. , Taylor, M. , and White, W. J. (2003). The relationship between Web site design and organizational responsiveness to stakeholders. Public Relations Review, 29(1), pp. 6377.

Which leads to … n Evidence-based information practice – “…an approach to information practice that promotes the collection, interpretation, and integration of valid, important, and applicable user-reported, librarian-observed, and researchderived evidence” Andrew Booth Senior Lecturer in Evidence-based Healthcare Information School of Health and Related Research University of Sheffield

What is evidence-based librarianship? n Data provides the primary evidence for making decisions – Not anecdotal stories – Not “common sense” – Evaluation occurs early in the process n How is this different from what often happens now? – Decisions are made based on < Beliefs of what is needed – Frequently, these beliefs are inherently biased < Assumptions, anecdotal evidence, and preferences – Evaluation, if it occurs, happens afterward

Evidence-based professional practice n n Derived from the evidence-based model of medicine Fundamental precepts – Study the phenomenon – Contrast results to other studies of the same or related phenomena – Combine results to better understand the phenomenon at hand

Evidence-based practice process Define the problem Redefine the problem Evaluate change Find the evidence Apply results of the evaluation Evaluate the evidence

Defining the problem n n Very similar to concepts used in “user-centered design” Five points – Yes, there’s an acronym - SPICE – SETTING < Where is this being used; what is the context? – POPULATION < Who are the (potential) users? – INTERVENTION < What is being done to/for them? – COMPARISON < What are the alternatives? – EVALUATION < What does success mean?

Levels of evidence 1. 2. Systematic review of a wide-body of rigorous studies Systematic review of a wide-body of less rigorous studies 3. Randomized controlled trials 4. Controlled-comparison studies 5. Cohort studies 6. Descriptive surveys 7. Case studies 8. Decision analysis 9. Qualitative research (focus groups, ethnography, historic, Delphi techniques, interviews) 10. Surveys, audits Adapted from Eldredge, J. (2000). Evidence-based librarianship: An overview. Bulletin of the Medical Library Association, 88(4). pp: 289 -302. Online at http: //www. pubmedcentral. nih. gov/articlerender. fcgi? artid=35250

Which brings us to the Northwestern story… n n First usability test in 2001 Focused in on “Electronic Resources” – A home-grown resource finder n n Investigation and questions based on anecdotal evidence with ER interface Learning experience – Provided model for organization n Whetted people’s appetite for usability testing

The second test – Searching in NUcat n n Began summer 2002 How are people using Voyager? – Are there discernable patterns? – Most importantly are there patterns of errors? – Can we do something to fix that? n What we did – Looked at searching patterns in our Voyager system – Used a more rigorous research approach < Based on grounded theory < Simple data mining techniques

And in the process… Searches by type

…dispelled some myths No hits within searches by type

Which lead to our tackling a bigger problem n Existing site did not work especially well for the Library's various user communities – Both students and faculty expressed frustration with the web site < Ability to navigate the site < Overall look and feel – Problem areas determined by gathering information in focus groups, online surveys, interaction with librarians and staff < Hours and locations < Collections < Electronic resources

Moving to a more evidencebased process n n Restructuring of the web development process Creation of a active web group – Library Web Advisory Group < < Representation from all divisions of the Library – Reference/Instruction – Collection management – Bibliographers – Branch libraries – Technical Services – Library Public Relations – Digital Library Systems Development – Library Information Technology Support Services Web Communications Group at University Relations

Induction process n Required reading list – Extensive list of supplemental readings n Required training – In usability – Conducting a usability test – Other “soft skills” < n Evidence-based procedures Required participation in design and testing – – Usability test moderator Usability test observer Transcriber Missionary/representative

Current results n Overall, site usability has improved – And we can provide it with statistical measures! – People find resources more quickly and easily < But n there is still an unfortunate amount of confusion Debates are less rancorous about how to proceed – We can always go back to the data – We can always go back to the studies n Easier to develop a strategy for incremental improvements over time – No longer locked into a tight academic schedule

Remaining issues n Jargon – I don’t know what – Metasearch – E-journals – Virtual reference – Electronic resources means! n Why should I go here? I can find everything in Google (or Google Scholar) – To some degree, this is true…

The big issue n Designing for a world where they don’t even come to the library web site – When the day comes that Google Scholar or Microsoft Academic Live do provide a “universal index” – Pluggable modules for the University portal

Thanks n Frank Cervone Assistant University Librarian for Information Technology Northwestern University Library 1970 Campus Drive Evanston, IL 60208 email: f-cervone@northwestern. edu AIM/YAHOO: hfcervone Skype: frank. cervone