Partnering with an Evaluator Presented by Joanne Kahn

Partnering with an Evaluator Presented by Joanne Kahn 1

Session goals 2 ¡ Understand how to hire and partner with a consultant to assist you with evaluation. ¡ Set the stage for an effective evaluation.

Steps in the partnership 1 - Project definition 2 - Vendor selection 3 - Contract negotiation 4 - Data collection 5 - Reporting 3

Project Definition: what to include? Things you could do in-house: l Library use statistics l Point-of-use inquiry by staff l Skills tests l Feedback forms/surveys (but could get design help) Outside firm helpful for: l Survey design l Focus groups l Interviews l Observation l Instructor assessments 4

Steps in the partnership Project definition Vendor selection Contract negotiation Data collection Reporting 5

Vendor selection Does your library have an existing relationship with a particular research firm? To find an evaluator, ask peers or check the American Evaluators Association website www. eval. org. Members commit to: l Systematic inquiry l Competence l Integrity/honesty l Respect for people l Responsibilities for general and public welfare 6

Vendor selection Interview for: l Compatibility with your goals l Flexibility l Experience with methods you want to use l Staff who speak/write in needed languages l Schedule availability l Willing to work within your budget constraints l A portfolio of similar work l References — are they easy to work with? See handout: article from “Usable Knowledge. ” www. usablellc. net/White_Papers/Hiring%20 an%20 Evaluation%20 Consultant. pdf 7

Points to discuss with the evaluator 8 ¡ Timing and frequency ¡ Who will do each task? ¡ Which participants? How many? ¡ Protecting the confidentiality of participants’ personal information ¡ How will participants be recruited? ¡ Incentives and perhaps transportation for participants

Evaluator’s perspective* ¡ Focus on the story you want to tell stakeholders, and collect the data you need to tell it. ¡ If you change staff (or delegate), make sure everyone knows the goals. ¡ Plan how you will handle unexpected or negative feedback. The evaluator will take the approach that all information is useful. *Special thanks to Juliette Mackin of NPC Research. 9

Steps in the partnership Project definition Vendor selection Contract negotiation Data collection Reporting 10

Contract negotiation ¡ Ask for a written proposal and fixed-price bid, and understand how the total was reached (hourly rates and line-item costs). ¡ If bid exceeds budget, look for ways to reduce costs: l Junior staff members may be less expensive than partners l You may be able to reduce their time by: ¡ ¡ ¡ 11 Arranging the venue, incentives, and refreshments Providing pre-screened participants with phone/email contact info Bundling this project with another

Contract negotiation 12 ¡ Set a schedule that includes time for review and revisions. ¡ Agree up front: l What intermediate deliverables will be provided? l Who needs to sign-off on the report?

Steps in the partnership Project definition Vendor selection Contract negotiation Data collection Reporting 13

Data collection — surveys 14 ¡ You may draft questions for the evaluator to review and include. ¡ Have you made the process as participant-friendly as possible? ¡ Measure before/after results. ¡ Are the questions clear? For multiple choice questions, are the choices adequate? ¡ Are there open-ended questions?

Data collection — focus groups 15 ¡ How will you recruit/screen participants? ¡ Have you made the logistics as participant-friendly as possible? ¡ The evaluator will probably want you out of the room. ¡ Review script for completeness, especially follow-on questions. ¡ Will there be a transcript or recording?

Steps in the partnership Project definition Vendor selection Contract negotiation Data collection Reporting 16

Reporting Respect the firm’s objectivity! But it’s fair to ask them to have a positive outlook. 17

Reporting: make it readable! o Executive summary conveys the main points o Outline is easy to follow o Visually attractive – photos!! o Do tables/graphs improve the report? o Are patron comments or anecdotes included? o Patterns — do data sources corroborate each other? o Conclusions should follow from the data 18