Reference Assessment Programs Evaluating Current and Future Reference
- Slides: 32
Reference Assessment Programs: Evaluating Current and Future Reference Services INTERVIEWS & FOCUS GROUPS National RUSA Institute: “Serving 21 st Century Users: Opportunities & Challenges” Marie L. Radford, Ph. D. Associate Professor, School of Information & Library Science, Pratt Institute October 12 -14, 2000 Omni Inner Harbor Hotel Baltimore, MD 1
Introductions • Presenter- Marie L. Radford, Ph. D. Associate Professor, School of Information & Library Science Pratt Institute, 200 Willoughby Ave. Brooklyn, NY 11205 -3897 mradford@prodigy. net (718) 636 -3512 (o) (718) 636 -3733 (fax) • Workshop Participants 2
AGENDA • Interviews – – – Types of interviews Questioning types Listening effectively Advantages & Uses Disadvantages Planning & Conducting Interviews • Focus Groups – Advantages & Uses – Disadvantages – Planning & Conducting FGs • Analysis • Reporting Results • Limitations 3
INTERVIEWS • What is an Interview? “An interview is a conversation involving two or more people, which is guided by a predetermined purpose. ” (Lederman, 1996, p. 5) 4
TYPES of INTERVIEWS • Person-to. Person with Individuals • Telephone • Focus Groups 5
KEY COMPONENTS è Good Questions èGood Listening Skills èGood Interpersonal Skills 6
TYPES OF QUESTIONS • 4 valuable types for information gathering interviews – OPEN • “What is it like when you visit the library? ” – DIRECTIVE • “What happened when you asked for help at the reference desk? ” – REFLECTIVE • “It sounds like you had trouble with the online catalog? ” – CLOSED • “Have I covered everything you wanted to say? ” 7
NEUTRAL, LEADING, AND LOADED QUESTIONS • NEUTRAL – “What are your impressions of the M. L. Radford Memorial Library? ” • LEADING – “You don’t like the librarians at the M. L. Radford Memorial Library, do you? ” • LOADED – “How many other unreasonable requests have you made to the reference librarians? ” 8
Questioning Do’s & Don’ts DO’S ÔBe clear ÔBe focused Ô 1 idea per question ÔUse specific questions ÔGive sufficient time ÔPRETEST! DON’TS ÔHypotheticals ÔWandering interviewee ÔToo many choices ÔAmbiguous or abstract questions ÔLeading or loaded questions 9
Effective Listening Do’s & Don’ts • • DO’S Pay attention Listen actively Invite talk Open mind Supportive atmosphere Show acceptance & understanding Paraphrase Note NV cues DON’TS • Rush to fill silences • Give opinions • Be defensive • Forget to listen for feelings • Hesitate to clarify • Glaze over 10
In-Person Interviews: Advantages & Uses • Face-2 -face interaction • In-depth info. • Understand experiences & meanings • Highlight individual’s voice • Preliminary information to “triangulate” • Control of sampling • Greater range of topics 11
Telephone Interviews • 3 Types of Sampling 1. All who phone are asked to participate in a short interview immediately 2. Phoning users are asked to participate in a call-back interview 3. Random dialing of community members (samples non-users) • Disadvantages – Not all users have phones – Nonverbal cues missing – Follow-up may be difficult 12
Interviews: Disadvantages • Time Factors – Varies by # & depth – Transcription: 1. 5 hrs. per hr. – Lots of prep. & administration • Cost Factors – Higher the #, higher the cost – Training interviewers – Tape purchase & transcription • Additional Factors – Self-report data – Errors in transcription or note 13 taking possible
Library Applications • Academic – Reference Encounter • Public – Family Place Project 14
ACADEMIC The Reference Encounter (Radford, 1999) • Interviews & observation • 3 sites – Community College – Undergraduate College – Research University • Qualitative methodology to capture complexity & depth • 27 pairs of librarians/users • Analysis – Critical Incident – Paired Perceptions – Categories 15
PUBLIC Family Place Project (Middle Country Public Library, Centereach, NY & Libraries For the Future, NYC) • Interviews & questionnaire • 5 sites – 3 Urban – 1 Suburban – 1 Rural • Qualitative methodology to evaluate change process • Analysis – Critical Incident – Categories 16
Phase I: Planning • Determine purpose of interviews in overall assessment • Formulate Q’s & pretest • Identify: Interviewees & Interviewers (& train ‘em) • Choose notes or tape recorder • Decide when & where • Decide f-2 -f or telephone • Plan schedule of interviews • Choose method of data analysis 17
Phase II: Conducting Interviews • Be on time & don’t waste their time • Obtain permission to use info. (report and/or publication) & if taping • If taping check equipment & have back-up • Create safe climate, assure confidentiality • Be prepared, flexible, & stay on task • Listen & know when to probe • Accept that some interviews won’t go well 18 • Thank them!
Focus Group Interviews • What is a Focus Group? “A focus group is an in-depth, face-to-face interview of a group of 8 to 12 people representing some target group and centered on a single topic. ” (Zweizig, Johnson, Robbins, & Besant, 1996) 19
Focus Groups: Advantages & Uses • Advantages of individual interviews plus… • Take less time • Creates synergy: – Comments stimulate others – Unexpected insights – More complete info. – Elicit strong & common opinions – Less inhibiting, less formal • Access needs of under-served or underrepresented groups 20 • Control of sampling
Focus Groups: Disadvantages • Skilled moderator required • Participants may be too quiet or too outspoken • Analysis, summarization & interpretation of responses may be difficult • Loads of planning & administration time • Transcription of tapes is time consuming & costly 21
Library Applications • ACADEMIC – Assessment of AV Delivery Service – Undergraduate College – Focus Groups of students, faculty, users and non-users of service – Used to develop survey to evaluate and improve service • PUBLIC – Preparation for service excellence training – Urban library system – Focus Groups of staff (mixed librarians & non librarians) and users – Used to develop survey to inform 22 training
Phase I: Planning • Determine role of focus group study in overall assessment • Identify: – target group(s) – moderator & assistant • Decide # of groups (3 -4 groups per targeted group) • Develop interview guide (& pretest) • Plan schedule (1. 5 -2 hours each), when, & where • Choose notes or tape recorder • Choose method of data analysis 23 • Con’t. . .
More Planning • RECRUIT VOLUNTEERS – Identify pool of potentials – Plan for 8 -12 per group (over-recruit and remind) – Offer reward if possible (is not $, free ILLs, photocopying, food) – Try to get representative members – Form homogeneous groups: • Academic (e. g. , faculty, undergrads, grad. students) • Public (e. g. , adults, teenagers, nonusers) 24
Moderator Characteristics • • • Excellent communication skills Experience in group dynamics Knows when to probe Restrained Ability to: – Involve all participants – Ask neutral questions – Quickly establish rapport – Summarize areas of agreement/disagreement – Direct but not control discussion 25
Phase II: Conducting Focus Groups • Be on time, don’t waste time • Bring/check supplies (flip chart, markers, masking tape, etc. ) • Obtain permission to use info. (report and/or publication) & if taping • If taping check equipment & have back-up • Begin by creating safe climate • More. . . 26
Conducting Focus Groups Continued • Use effective listening • Help quiet people talk, limit talkative people • Design well constructed guide, including: – Introduction (purpose, ground rules) – Ice breaker or warm-up set of questions – Relevant major/probe questions – Summary or closing • Thank them! 27
Interviews & Focus Groups Phase III: Analysis • Listen/transcribe tapes, review notes • Ways to analyze data: – Code data into pre-determined categories – Use data to identify categories – Use data as basis for summary statements “capture the essence” – Interpret the data- intensive analytic method • • Compare all interviews/focus groups Compile and summarize Look for trends or problems to study Do not overgeneralize from results! 28
Interviews & Focus Groups Phase IV: Reporting Results • Parts of Report – Statement of purpose – List of issues purpose – Explanation of how data was collected analyzed – Summary of findings: What was said on each issue – Sample quotes (anonymous) – Interviewer’s or moderator’s impressions – Recommendations • Short term (low hanging fruit) • Long term 29
Limitations: Interviews & Focus Groups • • Difficult to generalize May not be representative Analyzes perceptions, not facts Subjective analysis (reliability requires more than 1 person) • Raw data could be misleading • Results can be limited by: – Poor/inexperienced moderator/ interviewer – Poorly constructed discussion guide/interview questions • Compensate by combining methods (survey/questionnaires/ observation)30
31
Summary & More Information • Where to get more information • List of books, articles, in packet. 32
- Future perfect future continuous exercises
- Future perfect continuous and simple
- Evaluating your current fitness level
- Disadvantages of waterfall model in software engineering
- What is the importance of evaluating the list of references
- Current trends in media and information include
- Line current and phase current
- Power formula three phase
- Energy band diagram of pn junction diode
- Ac theory 3 lesson 4
- Drift vs diffusion current
- Ceramic composition resistors
- In a delta connected source feeding a y connected load
- Slideplayer
- Diffusion current density
- Current and future issues in corrections
- The constant-current area of a fet lies between
- The shielded metal arc welding machines voltage output
- Touch current vs leakage current
- Non planar circuit
- See future continuous
- Tense
- Future plans and finished future actions
- Future continuous and future perfect
- Nulti i prvi kondicional
- Current state vs future state diagram
- Reference node and non reference node
- Reference node and non reference node
- Future perfect presentation
- Past future tense adalah
- Future nurse programme
- Present continuous in future
- Vacation plan example