Reference Assessment Programs Evaluating Current and Future Reference

  • Slides: 32
Download presentation
Reference Assessment Programs: Evaluating Current and Future Reference Services INTERVIEWS & FOCUS GROUPS National

Reference Assessment Programs: Evaluating Current and Future Reference Services INTERVIEWS & FOCUS GROUPS National RUSA Institute: “Serving 21 st Century Users: Opportunities & Challenges” Marie L. Radford, Ph. D. Associate Professor, School of Information & Library Science, Pratt Institute October 12 -14, 2000 Omni Inner Harbor Hotel Baltimore, MD 1

Introductions • Presenter- Marie L. Radford, Ph. D. Associate Professor, School of Information &

Introductions • Presenter- Marie L. Radford, Ph. D. Associate Professor, School of Information & Library Science Pratt Institute, 200 Willoughby Ave. Brooklyn, NY 11205 -3897 mradford@prodigy. net (718) 636 -3512 (o) (718) 636 -3733 (fax) • Workshop Participants 2

AGENDA • Interviews – – – Types of interviews Questioning types Listening effectively Advantages

AGENDA • Interviews – – – Types of interviews Questioning types Listening effectively Advantages & Uses Disadvantages Planning & Conducting Interviews • Focus Groups – Advantages & Uses – Disadvantages – Planning & Conducting FGs • Analysis • Reporting Results • Limitations 3

INTERVIEWS • What is an Interview? “An interview is a conversation involving two or

INTERVIEWS • What is an Interview? “An interview is a conversation involving two or more people, which is guided by a predetermined purpose. ” (Lederman, 1996, p. 5) 4

TYPES of INTERVIEWS • Person-to. Person with Individuals • Telephone • Focus Groups 5

TYPES of INTERVIEWS • Person-to. Person with Individuals • Telephone • Focus Groups 5

KEY COMPONENTS è Good Questions èGood Listening Skills èGood Interpersonal Skills 6

KEY COMPONENTS è Good Questions èGood Listening Skills èGood Interpersonal Skills 6

TYPES OF QUESTIONS • 4 valuable types for information gathering interviews – OPEN •

TYPES OF QUESTIONS • 4 valuable types for information gathering interviews – OPEN • “What is it like when you visit the library? ” – DIRECTIVE • “What happened when you asked for help at the reference desk? ” – REFLECTIVE • “It sounds like you had trouble with the online catalog? ” – CLOSED • “Have I covered everything you wanted to say? ” 7

NEUTRAL, LEADING, AND LOADED QUESTIONS • NEUTRAL – “What are your impressions of the

NEUTRAL, LEADING, AND LOADED QUESTIONS • NEUTRAL – “What are your impressions of the M. L. Radford Memorial Library? ” • LEADING – “You don’t like the librarians at the M. L. Radford Memorial Library, do you? ” • LOADED – “How many other unreasonable requests have you made to the reference librarians? ” 8

Questioning Do’s & Don’ts DO’S ÔBe clear ÔBe focused Ô 1 idea per question

Questioning Do’s & Don’ts DO’S ÔBe clear ÔBe focused Ô 1 idea per question ÔUse specific questions ÔGive sufficient time ÔPRETEST! DON’TS ÔHypotheticals ÔWandering interviewee ÔToo many choices ÔAmbiguous or abstract questions ÔLeading or loaded questions 9

Effective Listening Do’s & Don’ts • • DO’S Pay attention Listen actively Invite talk

Effective Listening Do’s & Don’ts • • DO’S Pay attention Listen actively Invite talk Open mind Supportive atmosphere Show acceptance & understanding Paraphrase Note NV cues DON’TS • Rush to fill silences • Give opinions • Be defensive • Forget to listen for feelings • Hesitate to clarify • Glaze over 10

In-Person Interviews: Advantages & Uses • Face-2 -face interaction • In-depth info. • Understand

In-Person Interviews: Advantages & Uses • Face-2 -face interaction • In-depth info. • Understand experiences & meanings • Highlight individual’s voice • Preliminary information to “triangulate” • Control of sampling • Greater range of topics 11

Telephone Interviews • 3 Types of Sampling 1. All who phone are asked to

Telephone Interviews • 3 Types of Sampling 1. All who phone are asked to participate in a short interview immediately 2. Phoning users are asked to participate in a call-back interview 3. Random dialing of community members (samples non-users) • Disadvantages – Not all users have phones – Nonverbal cues missing – Follow-up may be difficult 12

Interviews: Disadvantages • Time Factors – Varies by # & depth – Transcription: 1.

Interviews: Disadvantages • Time Factors – Varies by # & depth – Transcription: 1. 5 hrs. per hr. – Lots of prep. & administration • Cost Factors – Higher the #, higher the cost – Training interviewers – Tape purchase & transcription • Additional Factors – Self-report data – Errors in transcription or note 13 taking possible

Library Applications • Academic – Reference Encounter • Public – Family Place Project 14

Library Applications • Academic – Reference Encounter • Public – Family Place Project 14

ACADEMIC The Reference Encounter (Radford, 1999) • Interviews & observation • 3 sites –

ACADEMIC The Reference Encounter (Radford, 1999) • Interviews & observation • 3 sites – Community College – Undergraduate College – Research University • Qualitative methodology to capture complexity & depth • 27 pairs of librarians/users • Analysis – Critical Incident – Paired Perceptions – Categories 15

PUBLIC Family Place Project (Middle Country Public Library, Centereach, NY & Libraries For the

PUBLIC Family Place Project (Middle Country Public Library, Centereach, NY & Libraries For the Future, NYC) • Interviews & questionnaire • 5 sites – 3 Urban – 1 Suburban – 1 Rural • Qualitative methodology to evaluate change process • Analysis – Critical Incident – Categories 16

Phase I: Planning • Determine purpose of interviews in overall assessment • Formulate Q’s

Phase I: Planning • Determine purpose of interviews in overall assessment • Formulate Q’s & pretest • Identify: Interviewees & Interviewers (& train ‘em) • Choose notes or tape recorder • Decide when & where • Decide f-2 -f or telephone • Plan schedule of interviews • Choose method of data analysis 17

Phase II: Conducting Interviews • Be on time & don’t waste their time •

Phase II: Conducting Interviews • Be on time & don’t waste their time • Obtain permission to use info. (report and/or publication) & if taping • If taping check equipment & have back-up • Create safe climate, assure confidentiality • Be prepared, flexible, & stay on task • Listen & know when to probe • Accept that some interviews won’t go well 18 • Thank them!

Focus Group Interviews • What is a Focus Group? “A focus group is an

Focus Group Interviews • What is a Focus Group? “A focus group is an in-depth, face-to-face interview of a group of 8 to 12 people representing some target group and centered on a single topic. ” (Zweizig, Johnson, Robbins, & Besant, 1996) 19

Focus Groups: Advantages & Uses • Advantages of individual interviews plus… • Take less

Focus Groups: Advantages & Uses • Advantages of individual interviews plus… • Take less time • Creates synergy: – Comments stimulate others – Unexpected insights – More complete info. – Elicit strong & common opinions – Less inhibiting, less formal • Access needs of under-served or underrepresented groups 20 • Control of sampling

Focus Groups: Disadvantages • Skilled moderator required • Participants may be too quiet or

Focus Groups: Disadvantages • Skilled moderator required • Participants may be too quiet or too outspoken • Analysis, summarization & interpretation of responses may be difficult • Loads of planning & administration time • Transcription of tapes is time consuming & costly 21

Library Applications • ACADEMIC – Assessment of AV Delivery Service – Undergraduate College –

Library Applications • ACADEMIC – Assessment of AV Delivery Service – Undergraduate College – Focus Groups of students, faculty, users and non-users of service – Used to develop survey to evaluate and improve service • PUBLIC – Preparation for service excellence training – Urban library system – Focus Groups of staff (mixed librarians & non librarians) and users – Used to develop survey to inform 22 training

Phase I: Planning • Determine role of focus group study in overall assessment •

Phase I: Planning • Determine role of focus group study in overall assessment • Identify: – target group(s) – moderator & assistant • Decide # of groups (3 -4 groups per targeted group) • Develop interview guide (& pretest) • Plan schedule (1. 5 -2 hours each), when, & where • Choose notes or tape recorder • Choose method of data analysis 23 • Con’t. . .

More Planning • RECRUIT VOLUNTEERS – Identify pool of potentials – Plan for 8

More Planning • RECRUIT VOLUNTEERS – Identify pool of potentials – Plan for 8 -12 per group (over-recruit and remind) – Offer reward if possible (is not $, free ILLs, photocopying, food) – Try to get representative members – Form homogeneous groups: • Academic (e. g. , faculty, undergrads, grad. students) • Public (e. g. , adults, teenagers, nonusers) 24

Moderator Characteristics • • • Excellent communication skills Experience in group dynamics Knows when

Moderator Characteristics • • • Excellent communication skills Experience in group dynamics Knows when to probe Restrained Ability to: – Involve all participants – Ask neutral questions – Quickly establish rapport – Summarize areas of agreement/disagreement – Direct but not control discussion 25

Phase II: Conducting Focus Groups • Be on time, don’t waste time • Bring/check

Phase II: Conducting Focus Groups • Be on time, don’t waste time • Bring/check supplies (flip chart, markers, masking tape, etc. ) • Obtain permission to use info. (report and/or publication) & if taping • If taping check equipment & have back-up • Begin by creating safe climate • More. . . 26

Conducting Focus Groups Continued • Use effective listening • Help quiet people talk, limit

Conducting Focus Groups Continued • Use effective listening • Help quiet people talk, limit talkative people • Design well constructed guide, including: – Introduction (purpose, ground rules) – Ice breaker or warm-up set of questions – Relevant major/probe questions – Summary or closing • Thank them! 27

Interviews & Focus Groups Phase III: Analysis • Listen/transcribe tapes, review notes • Ways

Interviews & Focus Groups Phase III: Analysis • Listen/transcribe tapes, review notes • Ways to analyze data: – Code data into pre-determined categories – Use data to identify categories – Use data as basis for summary statements “capture the essence” – Interpret the data- intensive analytic method • • Compare all interviews/focus groups Compile and summarize Look for trends or problems to study Do not overgeneralize from results! 28

Interviews & Focus Groups Phase IV: Reporting Results • Parts of Report – Statement

Interviews & Focus Groups Phase IV: Reporting Results • Parts of Report – Statement of purpose – List of issues purpose – Explanation of how data was collected analyzed – Summary of findings: What was said on each issue – Sample quotes (anonymous) – Interviewer’s or moderator’s impressions – Recommendations • Short term (low hanging fruit) • Long term 29

Limitations: Interviews & Focus Groups • • Difficult to generalize May not be representative

Limitations: Interviews & Focus Groups • • Difficult to generalize May not be representative Analyzes perceptions, not facts Subjective analysis (reliability requires more than 1 person) • Raw data could be misleading • Results can be limited by: – Poor/inexperienced moderator/ interviewer – Poorly constructed discussion guide/interview questions • Compensate by combining methods (survey/questionnaires/ observation)30

31

31

Summary & More Information • Where to get more information • List of books,

Summary & More Information • Where to get more information • List of books, articles, in packet. 32