Research Training Institute RTI Assessment Results Two Years

Research Training Institute (RTI) Assessment Results Two Years After: Building a Research Support System for Health Sciences Librarian-Researchers Jodi L. Philbrick, MSLS, Ph. D, AHIP Senior Lecturer, Department of Information Science, University of North Texas Lorie Kloda, MLIS, Ph. D, AHIP Associate University Librarian, Planning & Community Relations, Concordia University Editor-in-Chief, Evidence Based Library and Information Practice Susan Lessick, MA, MLS, AHIP, FMLA Project Director, MLA Research Training Institute Librarian Emerita, University of California, Irvine

Goals of MLA Research Training Institute • Increase research competencies • Increase research quality, quantity, and dissemination • Build research capacity to contribute to health and library improvements

Features of the Research Training Institute One-year institute that includes: 1. Immersive training workshop • Online coursework (~15 hours) and supporting resources • Face-to-face 5 -day workshop 2. Mentoring and monitoring 3. Online community of practice 4. Capstone presentation at MLA Annual meeting

RTI Cohorts Cohort 1 2018 – 2019 20 fellows Cohort 2 2019 – 2020 20 fellows Pre- and post-test assessment data Research progress RTI impact Cohort 3 2020 – 2021 20 fellows

Research Questions 1. Is the RTI effective for improving fellows’ confidence and research output? 2. How do Cohort 1 and the Cohort 2 fellows’ confidence and research output compare?

RTI Fellows’ Prior Research Experience & Research Education Activities Prior research experience since obtaining LIS master’s degree Cohort 1 (N=19) Cohort 2 (N=20) Have conducted research since master’s degree 12 11 Prior research education activities of participants Cohort 1 (N=20) Cohort 2 (N=20) Continuing education programs 13 13 Formal master’s degree and information science course 11 9 Staff development programs provided by your institution 6 1 Formal degree non-LIS course 4 6 Self-education activities 2 12 None 2 1 Formal doctoral degree LIS course 1 0

Fellows’ Reasons for Participating in the RTI REASONS (N=20) GROUP % IN AGREEMENT C 1 100% C 2 100% Will increase likelihood I will conduct program evaluations and assessments C 1 100% C 2 95% Will advance the profession C 1 95% C 2 100% Will provide opportunity to partner with and understand the needs of researchers C 1 95% C 2 95% Will increase likelihood I will engage in evidence-based decision making C 1 95% C 2 95% Will help demonstrate the value of my library to my administration and users C 1 85% C 2 100% Will support my tenure and/or promotion efforts C 1 75% C 2 50% Will help me contribute to research and scholarship

Confidence Levels of Participants • A pre- and post- assessment instrument based on Librarian Research Confidence Scale (LRCS-10) (Brancolini & Kennedy, 2017) • Pre-assessment survey deployment: • Cohort 1: May 3 -31, 2018 • Cohort 2: May 15 -31, 2019 • Post-assessment survey deployment: • Cohort 1: August 14 -31, 2018 • Cohort 2: August 22 -29, 2019 • Used the Wilcoxon Signed Ranks Test for each cohort to determine if there was statistically significant difference in the self-reported research confidence of the fellows before and after the RTI workshop

Fellows’ Research Confidence Levels Before and After Workshop (1) Rated with Likert scale: 5: Very Confident; 4 Confident; 3 Moderately Confident; 2 Slightly Confident; 1 Not At All Confident COHORT 1 Questions about skills needed for a research project COHORT 2 Median Rating (Pre) Median Rating (Post) Z score p. Median value Rating (Pre) 3 4 -3. 087 . 002 2. 5 4 -3. 630 3. Selecting methods and procedures for my question. 2 3 4. Developing plan and timeline for my study. 2 5. Identifying appropriate information sources in which to conduct my literature search. 4 1. Turning my topic into a question. 2. Designing a project to answer my question. Median Rating (Post) Z score p-value 3 4 -3. 491 . 000 3 4 -3. 815 . 000 -3. 352 . 001 2 4 -3. 971 . 000 4 -3. 534 . 000 3 4 -1. 973 . 049 5 -2. 221 . 026 2. 5 5 -3. 769 . 000

Fellows’ Research Confidence Levels Before and After Workshop (2) Rated with Likert scale: 5: Very Confident; 4 Confident; 3 Moderately Confident; 2 Slightly Confident; 1 Not At All Confident COHORT 1 COHORT 2 Questions about skills needed for a research project Median Rating (Pre) Median Rating (Post) Z score p-value 6. Using relevant keywords and search strategies to discover literature about the research topic. 4 5 -2. 804 . 005 4 5 -2. 299 . 022 7. Assessing and synthesizing literature that is relevant to your research question. 3 4 -2. 984 . 003 4 4 -0. 758 . 448 8. Using a theoretical framework to inform the research design of your study. 1 3 -3. 022 . 003 1. 5 3 -2. 702 . 007 9. Identifying sources of research funding and funding agency requirements. 2 3 -3. 570 . 000 10. Choosing an appropriate data gathering procedure. 2 3. 5 -4. 011 . 000 2 4 -3. 787 . 000

Fellows’ Research Confidence Levels Before and After Workshop (3) Rated with Likert scale: 5: Very Confident; 4 Confident; 3 Moderately Confident; 2 Slightly Confident; 1 Not At All Confident COHORT 1 COHORT 2 Questions about skills needed for a research project Median Rating (Pre) Median Rating (Post) Z score p-value 11. Determining which members of a population to include in your study. 2 4 -3. 672 . 000 2 4 -3. 676 . 000 12. Knowing how to design a focus group. 2 3 -3. 804 . 000 2 3. 5 -3. 903 . 000 13. Knowing how to run a focus group. 2 3 -3. 682 . 000 2 3 -3. 677 . 000 14. Knowing how to design an interview. 2 4 -3. 685 . 000 2 4 -3. 794 . 000 15. Knowing how to conduct an interview. 2 4 -3. 499 . 000 2 4 -3. 903 . 000

Fellows’ Research Confidence Levels Before and After Workshop (4) Rated with Likert scale: 5: Very Confident; 4 Confident; 3 Moderately Confident; 2 Slightly Confident; 1 Not At All Confident COHORT 1 Questions about skills needed for a research project COHORT 2 Median Rating (Pre) Median Rating (Post) Z score p-value 2 4 -3. 839 . 000 2. 5 4 -3. 250 18. Knowing institutional processes and standards to ensure that your study is conducted ethically. 3 4 19. Knowing what method of data analysis you would use for your study. 1 20. Knowing what type of assistance you might need to undertake data analysis (e. g. , data/statistics consulting, transcription, software). 2 16. Knowing how to design a survey. 17. Knowing how to administer a survey. Median Rating (Pre) Median Rating (Post) Z score p-value . 001 2. 5 4 -3. 703 . 000 -3. 274 . 001 3 4 -3. 469 . 001 3 -3. 668 . 000 1. 5 4 -3. 872 . 000 4 -3. 809 . 000 1 4 -3. 864 . 000

Fellows’ Research Confidence Levels Before and After Workshop (5) Rated with Likert scale: 5: Very Confident; 4 Confident; 3 Moderately Confident; 2 Slightly Confident; 1 Not At All Confident COHORT 1 Questions about skills needed for a research project COHORT 2 Median Rating (Pre) Median Rating (Post) Z score p-value 21. Knowing how to manage the data you have gathered. 2 3. 5 -3. 668 . 000 4 4 -. 924 . 356 22. Knowing how to code qualitative data to identify themes and sub-themes. 1 3 -3. 660 . 000 2 4 -3. 560 . 000 23. Reporting results in written format. 2 3 -3. 486 . 000 3 4 -3. 787 . 000 24. Reporting results verbally. 2 3 -3. 463 . 001 2. 5 4. 5 -3. 677 . 000 25. Identifying appropriate places to disseminate results. 3 4 -3. 640 . 000 3 4 -3. 405 . 001 26. Tracking the dissemination and impact of your research. 3 4 -3. 458 . 001 2. 5 4 -3. 072 . 002

Research Progress of Cohort 1 (as of June 2019, one-year post-workshop) 7 5 4 Write-Up 1** 2** 1** Problem Identified Research Questions Literature Review Research Method Design IRB Approval (if applicable) Data Collection Data Analysis **Had to postpone research projects due to job changes and/or work-related issues. 15/20 fellows submitted e-posters about their research projects for the MLA Annual Meeting in May 2019.

Research Progress of Cohort 2 (as of June 2020, one-year post workshop) 4 2 6 1 2** Write-Up 2** 3** Problem Identified Research Questions Literature Review Research Method Design IRB Approval (if applicable) Data Collection Data Analysis **Had delays due to COVID-19 pandemic, job changes, and/or other issues. 12 fellows in 2019 cohort and 3 fellows in 2018 cohort submitted e-posters about their research projects for the MLA Annual Meeting in August 2020.

Impact of RTI on Cohort 1 & Cohort 2 & their Institutions Type of Impact C 1 Frequency C 2 Frequency Formed internal and external research collaborations 13 12 Shared RTI experience with colleagues through informal and formal venues 12 3 Provided leadership to strengthen research capacity 9 2 Strengthened relationships with individuals outside of the library 8 6 Increased visibility of the library and its services 7 3 Improved or initiated new library services 6 3 Gained a better understanding of the users served 5 4 Received recognition for research 5 1 Decided to pursue more research education 4 1 Participated in other research activities 4 7 Increased confidence in conducting research 3 7 Developed research support materials 2 2 Impacted way research is conducted at institution 1 2 Gained institution’s interest in study findings 1 5 Increased the research culture at institution 1 n/a Guided and educated users about the research process n/a 8 Received research funding/support n/a 3 Communicated about research with others n/a 2

Conclusions Based on our assessment: • RTI has increased the research confidence of both cohorts • Cohort 1 and Cohort 2 are consistent in their amount of research progress at the one-year mark* • The most frequently reported impact of the RTI for both cohorts is forming internal and external research partnerships

Reference • Brancolini, K. R. , & Kennedy, M. R. (2017). The development and use of a research self-efficacy scale to assess the effectiveness of a research training program for academic librarians. Library and Information Research, 41(24), 44 -84. https: //doi. org/10. 29173/lirg 760

Comments/Questions? • For additional information about RTI: • RTI web site • MLANET, under “Professional Development” link at top of page • http: //www. mlanet. org/p/cm/ld/fid=1333 • Contact Us: • Jodi Philbrick (Jodi. Philbrick@unt. edu) • Lorie Kloda (lorie. kloda@concordia. ca) • Susan Lessick (slessick@uci. edu) This project was made possible in part by the Institute of Museum and Library Services (RE-95 -17 -0025 -17).

RTI Program Features • Objective application process • IMLS, AAHSL, and MLA scholarship support for RTI Fellows • 5 -member teaching faculty • Research training is designed to meet special needs of health sciences librarians; curriculum focuses on advanced research methods and emphasizes use of theoretical frameworks • Research projects of Fellows address research questions, topics, and populations of importance and interest to HS librarians • Structured mentor-based support after workshop as Fellows complete research projects • Active online RTI Community of Practice • Capstone research presentation • Comprehensive assessment plan; results shared and used for program improvement