Foundations of Assessment II Methods Data Gathering and

Foundations of Assessment II Methods, Data Gathering and Sharing Results

What do you still have questions about? • • How to effectively speak to students about problems How to include all stakeholders (develop partnerships) Developing effective questions What tools or mine-able data already exist that we can use in our assessment projects • When are you asking too much info at once (i. e. mid-year reviews)? • How to narrow down the focus of the assessment • How to stay on top of assessment while working on other programs

What do you still have questions about? • How to draw conclusions from assessment data (especially data from long-term assessments) • Quick assessments and what they look like • How long an assessment process should take • How to choose the right assessment method • How to conduct an assessment interview • How to incentivize participation in data collection • Further understanding how to “measure” qualitative data • How much is too much data collection? How do you predict the info you will need or want?

Goals of Today’s Presentation 1. Understand the various methods available for assessment 2. Learn how and why to share results 3. Gain confidence to plan and conduct assessments in your department

The Assessment Cycle Website: Assessment Resources Forms

A few questions • What methods have you used for data collection? • How did you determine that those methods were the best way to gather evidence?

Common Methods for Gathering Evidence § § § § Surveys Rubrics Interviews Pre/Post evaluation Reflection Quick assessments Document Analysis Reviewing existing data § § § § Focus Groups Demonstrations Observations Written papers, projects Posters and presentations Portfolios Mobile data collection Tests, exams, quizzes Contributing Source: Campus Labs

Things to consider when selecting a method § § § How will you use the data? Available resources (i. e. , time, materials, budget, expertise) Potential for collaboration Timing: Religious holidays, large events, student schedules Direct vs. indirect Quantitative vs. qualitative Contributing Source: Campus Labs

What Type of Data Do I Need? § Direct Methods: Any process employed to gather data which requires students to display their knowledge, behavior, or thought processes. § Indirect Methods: Any process employed to gather data which asks students to reflect upon their knowledge, behaviors, or thought processes. Source: Campus Labs

What Type of Data Do I Need? Quantitative § Focus on numbers/numeric values § Easier to report and analyze § Can generalize to greater population with larger samples § Less influenced by social desirability § Sometimes less time, money Qualitative § Focus on text/narrative from respondents § More depth/robustness § Ability to capture “elusive” evidence of student learning and development § Specific sample

Questions for Choosing a Method § Is your method going to provide you with the evidence you need to make a statement about the learning that occurred? § If you are assessing satisfaction or service effectiveness, is the method going to give you the most detailed and accurate information? § Do you have the time and resources to use the specific method? Adapted from Campus Labs

Surveys Advantages Disadvantages § Useful when you have a large population. § Easily administered § Easy to compare longitudinally § You can ask a variety of questions § Quick data turnaround § Good for statistical reporting § Indirect measure § Interpretation – lack of skill in understanding results § Low response rates § May need secondary direct measure to learn more information

Focus Groups: What is a focus group? § Qualitative research § Small group, open discussion § Gauges opinions, perceptions and attitudes

Focus Groups Advantages Disadvantages § Able to collect a lot of rich data in short time § Facilitation requires skill § Check perceptions, beliefs and options § Explore specific themes § Participants build off each other’s ideas § Can be a good follow-up to a survey § Not generalizable to population § Time needed for preparation and analysis § Lack of control over discussion § Can be difficult to attract participants § IRB may be required

Quick Assessment Techniques § Quick, easy, and systematic § Assessment does not HAVE to be time-consuming § Minimal resources § “Pulse” on how things are going § Specific techniques available on BC website

Rubrics § Scoring tool for subjective assessment § Assess student performance on learning objectives § Clearly define acceptable and unacceptable performance § Training programs, interviews, projects

Rubric Example

Mobile Devices

How/When to Use Mobile Devices Campus Pulse Survey § Larger audience and more diverse sample § Dining Halls, residence halls, computer lounge § Topic suggestions: food, campus facilities, hot topics, world events Point of Service Survey § Distribute to students after they stop an office on campus § Focus on service experience § Locations: Career Center, SABSC, Residential Life After an Event Survey § Immediate response § After a concert, meeting, or dance

Activity: Choosing a Method • Use your assessment project/idea • What are the outcomes you are assessing? • What is the best way to gather this information? Why? • Report out to the group

Analyzing and interpreting your data requires as much diligence and strategy as the design process

Qualitative Data Analysis § Read the data with an eye for themes, categories, patterns, and relationships § Have multiple people read the data and discuss the key themes § Identify contradictions surprises Interpretation § Make assumptions, add meaning, and come up with conclusions – keeping your own assumptions and beliefs visible. § Do not disregard outliers: Data that is surprising, contradictory, or puzzling can be insightful Source: In. Sites 2007

Quantitative Data Analysis § § § Campus Labs - sort your data, use crosstabs, and view it in graphs or pie charts. Campus Labs does not help you analyze or interpret your data! Your responsibility is to describe the data as clearly, completely, and concisely as possible. Interpretation § For each learning outcome, compare the results with the level of intended outcome. What does the data show?

Reporting Assessment Results • In what ways have you seen assessment results reported? Source: Campus Labs

Why Focus on Reporting Results? § § § Role Modeling Buy-in Historical Documentation Evidence trail What about sensitive data and/or campus politics? Source: Campus Labs

Formats for Audiences Students • Email invitations • PR campaign (flyers, newspaper, TVs) • Student government meetings Internal campus partners • • Cabinet meetings Elevator speech Exec. summary Annual reports Staff • • • Roadshows Brief emails Newsletters Retreat Full reports External Constituents • Presentations • Website • Press releases Source: Student. Voice

Policymakers are more likely to read information if: • Information is in short bulleted paragraphs, not large blocks of type • Charts or graphs are used to illustrate key points • If provided in print rather than electronically • Recommendations and implications are presented Source: Campus Labs

Questions to Consider when Planning a Report üWhen do you need to report your results? üWho is the audience of the report? üWhy is this information important to this audience? üWhat are your options for reporting the results? üWhat is the best method for reporting your results? üWhat exactly should be or needs to be included?

Basic Structure of Assessment Report 1. 2. 3. 4. Executive summary Purpose of assessment Methods Description of participants 5. Findings 6. Discussion/implications and conclusion

Questions/Discussion