Data 102 Using Data to Drive Change March

Data 102: Using Data to Drive Change March 9, 2016 Preparing a Diverse Population for Work

INTRODUCTION Preparing a Diverse Population for Work.

Learning Outcomes Participants will: • • Understand the data request process and where to start. Learn how to identify the necessary data points needed. Understand different variables/factors to consider when interpreting data. Have a basic understanding of how to use data to drive change. Preparing a Diverse Population for Work.

WHAT DO YOU NEED? Preparing a Diverse Population for Work.

FUN FACT #1: Our ABE/GED students transition into college-level courses at a rate of 27. 5%? A. True B. False

What Data Do You Actually Need? • What are you trying to figure out? • What are possible data elements? • Know the terms and what is available • Making assumptions Preparing a Diverse Population for Work.

ASSUMPTIONS AND BIASES Preparing a Diverse Population for Work.

What is Confirmation Bias? A. The tendency to search for, interpret, favor, and recall information in a way that confirms one’s beliefs or hypotheses. B. When a belief persists even after evidence to the contrary has been presented. C. A type of cognitive bias. D. All of the above. E. Only A and C.

Assumptions and Biases “Each [person] approaches scientific problems with a theoretical viewpoint, whether explicit or implicit. Theory guides inquiry through the questions raised, the framework of inquiry, and the interpretation of results. Each [person] seeks to find “facts” to assimilate into his or her world view. Thus, each of us is biased by human tendency to seek “facts” that are congruent with our prior beliefs. ” (Scarr, 1985). In a nutshell, we are all biased based on our life experiences! Preparing a Diverse Population for Work.

Don’t be like this guy! Preparing a Diverse Population for Work.

Assumptions and Biases continued… If everyone is biased, how can I avoid it? • Know in advance what your biases are. Remember, anecdotal evidence is not always what is REALLY happening. • Get a second opinion! • Take a “break” from the data and come back to it later with a clear head. • Ask yourself if your bias is clouding your interpretation of the data. Preparing a Diverse Population for Work.

The “File Drawer Problem” Your hypothesis was not confirmed…. What do you do? Don’t just file it away and pretend it didn’t happen, “null” results are sometimes just as important. Preparing a Diverse Population for Work.

FUN FACT #2: Over the past 5 years, our IPEDS completion rate increased from ___% to ___%? A. B. C. D. 57%; 65%; 57% 61%; 63% 56%; 63%

RESEARCH TERMINOLOGY Preparing a Diverse Population for Work.

When can you determine causality? A. When you observe or measure variables of interest without manipulating them. B. When you conduct a “true” experimental study. C. When you conduct an in-depth case study.

Correlation vs. Causation • At RTC, almost all of the research and data analysis we do can only show correlational relationships. • Causality determined through experimental design: § Groups randomly assigned § Control group § One variable manipulated Preparing a Diverse Population for Work.

How do you define a significant difference? A. A difference that is important. B. A difference that meets a specified significance level (p-value). C. When the null hypothesis is rejected. D. This is a trick question, it depends on the context.

Statistical vs. Practical Significance v Significant ≠ Important v Important ≠ Significant • • Group Retention Completion Group A 65% 50% Group B 66% 52% These group differences are not likely to be statistically significant, but if the school’s focus is on equity, this difference could still be important; i. e. , it could be practically significant. There are ways to actually measure practical significance from a statistical standpoint, but it can often be a judgment call based on context and the researcher’s focus. Preparing a Diverse Population for Work.

FUN FACT #3: Our degree (AAS or AAS-T) completion rate for the 2009 -2012 cohorts was ___%? A. B. C. D. E. 25. 1% 13. 8% 13. 2% 41. 5% 22. 3%

Data Types • Look at the following example. Can you calculate an average? 42% 45% # Responses 40% 0 -5 Hours 3 30% 6 -10 Hours 1 20% 11 -20 Hours 5 10% Response 20 -30 Hours 3 35% 25% 15% 25% 8% 5% 0% 0 -5 Hours 6 -10 Hours 11 -20 Hours 20 -30 Hours Technically, no. If you want to calculate averages, make sure you use scaled variables. If you are interested in frequency counts, you can use categorical variables. Preparing a Diverse Population for Work.

Let’s do some math! What is the mean, median, and mode for this series of numbers: 4, 4, 3, 8, 11 A. B. C. D. 6, 4, and 4 5, 3, and 4 6, 4, and 8 4, 4, and 4

Why the Median? • Why do we use the median when reporting on things like wage data? § Less susceptible to extreme values Wage $10, 000 $12, 500 Median = $11, 250 Mean = $18, 125 $40, 000 Preparing a Diverse Population for Work.

When conducting research, what kind of data is better, quantitative or qualitative? A. Qualitative for sure! B. Nope, definitely quantitative. C. You’re both wrong, they are equally important. D. Neither!

Quantitative vs. Qualitative Research Quantitative: • • Qualitative: Identify cause and effect relationships. Variables and methods defined beforehand. Focuses on the average behavior of people in a population. Data are numerical. • Focuses on understanding how people experience and interpret events. • Variables and methods may change during research. • Focuses on the similarities and differences in individual experiences. • Data are open-ended. WE NEED BOTH!! Preparing a Diverse Population for Work.

FUN FACT #4: RTC students who earned an AAS/AAS-T degree during the 2008 -2009 year had the highest wages in the state in 2013 (for all community/technical colleges). A. True B. False

DATA LIMITATIONS Preparing a Diverse Population for Work.

Sample Size • What can you conclude from these results? Course Success Rates by Modality, 2007 -2012 90% 82% 79% 73% 70% 60% 50% 40% 30% 20% 10% n = 13, 576 n = 5, 909 n = 123 Classroom Online Hybrid 0% Preparing a Diverse Population for Work.

Missing Data • What can you conclude from these results? Pre-Algebra College Algebra Completed 53. 13 42. 96 44. 31 Did not Complete 48. 23 38. 47 40. 69 # w/ Available Data 1, 725 536 49 % w/ Missing Score 36. 7% 80. 3% n = 12398. 2% Preparing a Diverse Population for Work.

Parameters (definitions) • How you ask the question really matters! • Get to know the RTC definitions in the Data Dictionary. I need the retention rates for my program! Cohort Comparison 2012 -2015 IPEDS New State Start PEP 555 600 483 500 476 390 400 334 311 300 200 164 139 100 0 2012 -2013 -2014 Don’t be afraid to tell us exactly what you want because we might interpret your question very differently or take our own liberties with definitions. 2014 -2015 n = 123 The answer can only be as detailed and accurate as the question itself. Preparing a Diverse Population for Work.

FUN FACT #5: What are the pass rates for our coop, internship, and/or practicum courses? A. B. C. D. 94% 88% 92% 75%

Database Limitations • • Time lag in Data Warehouse Preliminary vs. final data Wage and placement data • Significant delay • Only once per year • Only those in UI • Only those who left RTC and didn’t go elsewhere • Only those with available wage data • Sample size issues Testing data High school data n = 123 Detailed transfer data outside of WA state Income Preparing a Diverse Population for Work.

Summary • Familiarize yourself with common terms and definitions. • Don’t be afraid to ask questions, and then ask some more. • Consult with colleagues. • The more detail/information, the better! • Pay attention to footnotes. • Assessing effectiveness of interventions or changes takes time! • Don’t use data for evil. Preparing a Diverse Population for Work.

USING DATA TO DRIVE CHANGE Preparing a Diverse Population for Work.

Data-Driven Decisions • Faculty • Curricular changes • Financial impact • Admin/Staff • Staffing changes • Policy/procedure changes • Financial impact Preparing a Diverse Population for Work.

FUN FACT #6: __% of our Basic Studies students earned basic skills points in 2013 -2014, compared to __% statewide. A. B. C. D. 10%; 16% 8%; 16% 10%; 4% 16%; 8%

BREAK Preparing a Diverse Population for Work.

CASE STUDY SLIDE TITLE Preparing a Diverse Population for Work.

Case Study Directions Break into groups of 4 or 5 people. Each group will receive a program profile to review. Things to Consider: o This is a fictional program with data created by us, it does not represent any RTC program. o Do you need additional data? If so, ask us for the information. o What might be contributing to the low enrollment? o How do you evaluate the data? o What are other potential issues facing this program? o What interventions can be put in place to address these issues? o Should program revision occur? o What are the next steps? Remember, there are no wrong answers! Preparing a Diverse Population for Work.

FUN FACT #7: 1 st to 2 nd quarter retention in the five programs that piloted College Success increased from 62% to ___%? A. B. C. D. 78% 72% 65% 80%

DISCUSSION/QUESTIONS? Preparing a Diverse Population for Work.

Data Dashboards Several data dashboards are now available on the IR website! Data Dashboards • Enrollment • FTE • Retention & Completion • Placement and Wages • Outreach Look out for invitations to various dashboard trainings throughout the year!! Preparing a Diverse Population for Work.