PDI Data Literacy Busting Myths of Big Data

PDI Data Literacy: Busting Myths of Big Data Nairanjana (Jan) Dasgupta Professor, Dept. of Math and Stats Boeing Distinguished Professor of Math and Science Director, Center of Interdisciplinary Statistics Education and Research (CISER) Washington State University, Pullman, WA Email: dasgupta@wsu. edu

Part 1: Data in general v Data: Big or Small v SOURCE of data v Types of Data v Population versus Sample v Experiment versus observational studies v Exploratory and Confirmatory studies

Part 2: making sense of data v Distinction: Uni-variate, Bi-variate, Multi-variate, multiple v Graphical Summary of data v Numerical summary of data v Measures of Center v Measures of Spread v Measures of the Dimensionality v Summarizing multivariate data sets using clusters v Population versus sample what do we have data on?

Part 3: Making decisions from data: v Going from sample to population v Inference and decision making v Estimation and Intervals v Testing and Confidence Intervals v Errors in testing: Type I and Type II v Power v Statistical significance v P-value — good, bad or misused v ASA’s statement about p-values

Part 4: Big Data and its pros and cons v What are the advantages of big data v What do we mean by big? Big n or big p v Decision making with big data v Predictive analytics v Back to population versus sample v Overview and recap

Part 1: Data and its collection Types of Data: The good the bad and the ugly

Statistics and Data v Statistics has been often defined as the Science (or art) of collecting, compiling, summarizing, analyzing and inferring from data. v By this definition: it is the science (or art) that is meant to deal with data v So what is data science? The science dealing with big data?

Big Data: the elephant in the room

BIG Data? v What is big data? v How big is big? v Is it necessarily a good thing? v Some thoughts about how we deal with it from the Statistics point of view v We will start and end on this topic v But focus in the middle for the most part of “good” data

Some facts about BIG data v Most of the time big data is generated not collected. v No study design associated with its collection v Often unclear what we want it to tell us: we are often doing a stab in the dark approach. v I would like to coin the phase “opportunistic data” for big data that is not collected with a specific aim in mind, like social media data or phone data.

BIG Data: Some thoughts v Not much ACTUAL data analysis as the challenge is to actually manage and extract. v Mostly pretty pictures and “dashboards’ v An obsession now with decision makers: a buzzword. v Having more doesn’t solve the problem if the data is not “GOOD” to start off with. v Has more problems with BIAS as it is not collected in a systematic way. v Issues with dimensionality. v Extreme problems of multiple testing and false positives

Sobering Thoughts: Some Findings from BIG data that didn’t gel: v Prediction of flu outbreaks was off by a factor of 2 v Academy Award predictions were more off than correct v For the 2016 election Forbes predicted "If you believe in Big Data analytics, it’s time to begin planning for a Hillary Clinton presidency and all that entails. ” v Many other examples…

Data: GOOD, BAD or the culprit? v I will misquote Samuel Taylor Coleridge here (his quote was about water), when the ancient mariner was stuck in the middle of the ocean: Data data everywhere, and it really makes us blink Data, data everywhere but we’ve got to stop and think… v This is theme of today’s lecture: understanding data, types of data and what we can and cannot do with data. The Mathematics/Statistics that we need to understand to deal with data…

How can data be used? v Just because we have data does it mean we know something? v Can we say anything at an individual level? v I want to go to WH Auden and address this question…

THE UNKNOWN CITIZEN BY W. H. AUDEN(To JS/07 M 378 This Marble Monument. Is Erected by the State) He was found by the Bureau of Statistics to be One against whom there was no official complaint, And all the reports on his conduct agree That, in the modern sense of an old-fashioned word, he was a saint, For in everything he did he served the Greater Community. Except for the War till the day he retired He worked in a factory and never got fired, But satisfied his employers, Fudge Motors Inc. Yet he wasn't a scab or odd in his views, For his Union reports that he paid his dues, (Our report on his Union shows it was sound) And our Social Psychology workers found That he was popular with his mates and liked a drink. …

Unknown Soldier… Contd … He was married and added five children to the population, Which our Eugenist says was the right number for a parent of his generation. And our teachers report that he never interfered with their education. Was he free? Was he happy? The question is absurd: Had anything been wrong, we should certainly have heard. From Another Time by W. H. Auden.

Data collected because or just because v There is a distinction whether we are collecting data with a specific object in mind or we are using existing data that is already available. v Even a few years ago data was expensive and valuable and collected mostly with specific objects in mind v Now, there is a deluge in terms of data that is available for use as it is being collected ANYWAY.

Data Source: Stats is well equipped to deal with

Sources of Opportunistic data:

Let us start with collecting some data v Let us start with some information about you: 1. 2. 3. 4. What is your department or unit? How many Statistics classes you have taken? On a scale of 1 to 5 rate your liking for Statistics Your average blood pressure when faced with a Stats problem There is no source that I know of where such data is available so let us physically collect it.

Types of data Numerical Data Ordinal Categorical Discrete Continuous Nominal

What are these? v Nominal: Name, category v Discrete: what you count v Continuous: what you measure v Ordinal is in some no-man’s land mostly categorical but with a numerical flavor. 1. 2. 3. 4. What is your unit? On a scale of 1 to 5 rate your liking for Stats How many Statistics classes you have taken? Your blood pressure when faced with a Stats problem

How to collect this data? v Questions 1 -3 are self reported easily. v How about question 4? v Let us think about that

The big question is: WHY did I collect this information? v The reason we collected this information was to get some idea about all of you so I could come up with a plan of what to talk about, how much detail I go into etc. v So, the idea is I take the data I collected and LEARN something from this data? v Data by itself is just a bunch of numbers or categories and by itself it doesn’t mean much. v What we need to do is figure out how we LEARN from data.

Why collect Data? v In the past to answer a specific question or questions. v However, nowadays SOME data is collected without a specific object in mind, just because everything is data oriented, and collection is easy, internet, cell phone, credit card, Department stores etc. v But in general we have a purpose for collecting data, a specific questions or questions. v We want to learn something that we do not know from data.

Population versus Sample v Our question of interest is almost always about the big picture: something hard to study, but knowing would help us make decisions. v Population: Sum total of all individuals and objects in a study v Sample: part of the population selected for the study v Idea: Get a good sample, study this sample carefully and infer about the population based on the sample. v The data I just collected is this a population or a sample?

Population, Sample Inference

Where does statistical science come in? v If we could always study the population directly: we wouldn’t need Statisticians except for clerical jobs like summarizing… v If we are relying on samples: we need to take a GOOD sample: the good is defined by Statistics v What type of sample can I take? v How exactly can I “infer” attributes about the population (parameter) based on a sample (statistic)? v Caveat: the population at hand needs to be a REAL representative population.

A few sample types

Experiments versus Observational Studies v Experiments: You change the environment to see the effect your change had, trying to control all other potential factors that might affect your study. v Observational Study: You study the environment as is, and collect data on all possible factors that might be of interest to you. v Questions 1 -3 definitely observational v Question 4 could have been an experiment but possibly observational

Experiment versus Observational study

Questions to think about: v If you had a choice: would you want to do an experiment or an observational study? v Why? v Is BIG data always better data?

What does it matter the type of study we conduct? v It matters because how we proceed to analyze the data should differ in terms of the type of study we had. v Nowadays it is common to have data collected “just because” or “opportunistic data” and these are the extreme types of observational studies. As it wasn’t collected without any aim and it is hard to figure out if it is a population or a sample.

Exploratory versus Confirmatory studies v Exploratory studies: We do not have an idea about what we expect to find. So we study a bunch of factors to see how they affect what we are studying. It can be experimental or an observational study (though generally observational) v Confirmatory Studies: Generally have an idea what we are expecting to find and do a very focused study to give credence to our beliefs. It can be experimental or an observational study (though generally experimental) v Keep in my mind we cannot use data collected in an exploratory study to confirm our belief or hypothesis.

Exploratory versus confirmatory studies v Exploratory studies generate hypothesis and often we do know/expect certain patterns v We confirm these using confirmatory studies v Though in practice people often skip the confirmatory studies…

For Big data: what are we trying to learn v This is one of the harder questions about big data v Is it a population that we have or is it a sample? v If the latter, then what is the population? v How can this sample be used to infer about a population when no frame was used to draw the sample? v And statisticians and data scientists we need to think about this.

Part 1: recap v It matters what TYPE of data we have. v It matters how the data were collected v It matters whether we have a population or a sample v It matters if you randomized the process of data collection v If the population is studied all you need to do is to summarize, with a sample we need to think of inference. v If we are really dealing with a population when we talk about big data: then all we need to do is visualize and summarize. No inference required.

Work sheet for Part 1 v What type of data are the following: • Zip code, height, phone number, yearly income, size of family v If we randomly choose 50 apple trees in an orchard and measure its height, canopy cover, number of apples: • Would this be an observational study or experiment • Would the data be univariate or multivariate • Would this be an exploratory study or a confirmatory one?

Part 2 Summarization: Making sense of numbers

Summarizing Data v Let us go back to the questions we started with: 1. 2. 3. 4. What is your unit? On a scale of 1 to 5 rate your liking for Stats How many Statistics classes you have taken? Your blood pressure when facing a Stats problem Let us think about answering these questions and address things like: univariate, bivariate and multivariate in this context.

Univariate, Bivariate and Multivariate v Univariate: ONLY one variable is measured or disseminated at a particular time v Bivariate: Two variables that of equal importance are measured or discussed together v Multivariate: Multiple variables are measured and discussed together and each of the variables are of equal importance and collected using the same general method. v Most data we collect is multivariate in nature, but we can choose to discuss one variable at a time. Not a great idea, but often done in science.

Response and Explanatory Variables For the bivariate and multivariate case we can have two different types of scenarios: v ALL variables are equally important v We are REALLY interested in one variable but collect the others to understand the variable of interest. v The one that we are really interested in is called the RESPONSE variable. The others are called Explanatory variables. It was collected to explain the response. v Response variable is taken to be a RANDOM variable (or stochastic). Explanatory variables are assumed non-random.

Multiple versus Multivariate v These are associated with whether we have multiple responses or explanatory variables: v If we have multiple response variables and we are equally interested in them: multivariate v If we have ONE response variable and multiple EXPLANATORY variables: multiple

Analyzing UNIVARIATE data v With all the terminology intact, let us consider the most simple case. v ONE response variable. v We can graph it, and summarize it if it is a population from which we have the data. v We can conduct inference if it is a sample. v How and what we do, depends upon the data type we have. v Hence, knowledge of data types is crucial

Summarizing Univariate Categorical Data v What would be some methods used to summarize categorical data? v Graphical summary and numerical summary: v What graphs are relevant for univariate categorical data? v Pie chart v Bar chart v Line chart etc…

Numerical Data: Summary v Graphical summary: v Box plots v Histograms

Snap shot (out of 100 entries) v Unit Classes_taken v. F v. B v. C v. B v. A v. D 2 2 2 0 2 4 6 0 3 Scale 3 4 1 1 3 3 2 1 4 Blood_Pressure 139. 248 136. 148 107. 655 159. 391 121. 990 140. 697 94. 654 125. 146 83. 428

Graphs and Charts for our data Pie Chart for the units Bar Chart for classes taken

Histogram and Box-plots For Scale Summarizing BP

Summarizing Categorical data: Univariate v What numerical summaries would be relevant for categorical data: v For example let us take your MAJORs v How would you summarize all the information in our data by one (or a few numbers)? v Idea of Central Tendency: Most naturally arising data has a tendency to clump in the middle of the range of possible values.

Measures of Central Tendency v How does one measure what is happening in the CENTER of the data? v Thoughts? v Mean v Median v Mode

Mean, Median, Mode v What are the physical interpretations of these: v Center of Gravity v Middle-most point v Most frequently appearing number, category or group

Pictures of mean, median and mode

Measuring Central tendency for Categorical Data v When dealing with your majors: v Does Mean make sense? v Does Median make sense? v How about mode?

Numerical Data v Let us consider numerical data: v Which measure of central tendency makes sense here? v Which would you prefer? v Mean v Median v Mode

More Summarization: v Measure of Center, provide us with a first step for summarizing data. But often it cannot differentiate between data sets. v v Consider the following data sets: v v Set 1: 20 30 40 50 60 v Set 2: 20 35 40 45 60 v Set 3: 30 35 40 45 50 v v They all have same mean and median=40. v Are they identical? What makes them different?

Other summary measures v Measures of Spread v Shapes of the distribution of data v Where is the peak v Measures of symmetry v Percentiles

Measures of Spread v Standard Deviation v Variance v Range v Inter-quartile range v Median Absolute Deviation

Summarizing Bivariate and Multivariate data v Categorical: frequency tables v Numerical: Scatter plots v Many types of visualizations coming up because of “big data”. v Most of these dashboards use a combination of known techniques like graphs with a few bells and whistles thrown in.

Table of Major and Scale A B C D E F G H All 1 6 7 4 2 0 0 0 2 21 2 7 14 4 0 0 0 1 1 27 3 4 6 7 3 0 1 22 4 3 4 5 3 2 0 0 0 17 5 6 2 2 1 0 0 1 1 13 All 26 33 22 9 2 1 2 5 100

Scatter Plot

Measures of Association v For bivariate data we can calculate measures of association v Kendall’s tau (for ordinal data) v Correlation Coefficient (for numerical data) v For our data Kendall’s Tau for Major and Scale is. 07 v Correlation coefficient between BP and Classes. 03

Summarizing multivariate data v Principal Component Analysis v The following are loosely considered summarizing methods: • Clustering

What is PCA? v This is a MATHEMATICAL procedure that transforms a set of correlated responses into a smaller set of uncorrelated variables called PRINCIPAL COMPONENTS. v Essentially gives us an idea about the dimensionality of the data Used for summarizing the overlap in the variables v v v Uses: Data screening Clustering Discriminant Analysis

Objectives of PCA Reduce dimensionality, rather try to understand the TRUE dimensionality of the data v Identify “meaningful” variables v If you have a VARIANCE-COVARIANCE MATRIX, S: v PCA returns new variables called Principal Components that are: v • Uncorrelated • First component explains MOST of the variability • The remaining PC explain decreasing amounts of variability

Caveats The whole idea of PCA is to transform a set of correlated variables to a set of uncorrelated variables, hence if the data are already uncorrelated, not much additional advantage of doing PCA. v One can do PCA on correlation matrix or the Covariance matrix. v

What is Clustering? v Clustering is an EXPLORATORY statistical technique used to break up a data set into smaller groups or “clusters” with the idea that objects within a cluster are similar and objects in different clusters are different. v It uses different distance measures between units of a group and across groups to decide which units fall in a group.

Clustering in General Idea: to group observations that are “similar” based on predefined criteria. v Clustering can be applied to rows and/or columns v Clustering allows for reordering of the rows/columns to make it appropriate for visualization. v Fundamentally an exploratory tool, clustering is firmly imbedded in many people’s minds as the statistical method for the analysis… v

Why Cluster? There are very few formal theories about clustering though intuitively the idea is: v cluster the internal cohesion and external isolation. v Time-course experiments are often clustered to see if there are developmental similarities. v Useful for visualization. v Generally considered appropriate in typical clinical experiments. v

Clustering v How is “closeness decided”? v For clustering we generally need two ideas: v Distance: the original distance used to measure the distance between two points v Linkage: condensation of each group of observations into a single representative point

Types of Clustering Hierarchical and Non-hierarchical methods: v Non-Hierarchical (Partitioning): Have an initial set of cluster seed points and then build clusters around the point, using one of the distance measures. If the cluster is too large, it can split into smaller ones. v v Hierarchical: Observed data points are grouped into clusters in a nested sequence of groups.

Dendograms The dendogram should be interpreted with care, remember each branch of the dendogram is really like a mobile and can rotate, without altering the mathematical structure of the tree. v Neighboring nodes are “close” ONLY if they lie on the same branch. v It has been proposed one should slice the tree and look at the clusters produced therein. However, WHERE to cut the tree is subjective and there is no consensus about this. v Issue: mistakes made early have no way of being corrected later in this approach. v

Some remarks on clustering- 1 Simplistically, clustering cannot fail. That is, every clustering method will return clusters, whether the data are organized in clusters or not. v Clustering helps to group / order information and is a visualization tool for learning about the data. However, clustering results do not provide any kind of “proof” of anything. v

Some remarks-II v v One of the more paradoxical aspects of clustering is that it gets used in practice, even when class labels are available instead of using a discrimination method. The idea is: it is somehow seen as less “biased” to demonstrate the ability of the data to produce the class differences without using class labels. When the inferred clusters largely coincide with the known classes, this is thought to “validate” the class labels. The illogicality and inefficiency of this process does not seem to have become widely appreciated. One sees different “classifiers” (e. g. different gene sets) compared w. r. t their ability to separate known classes, simply by inspecting the clustering they produce, rather than by building classifiers.

Comparing clusters:

Back to Population versus Sample v The data that we collected, was it a population or was it a sample? v IF it is a population ALL we need to do is to summarize it or graph it. We have studied the population hence we do not need to do any more. v So, the question now is the data that we have what do we do with it. Stop now (we are done summarizing or should we do something more).

Recap of Part 2 v Use Pie charts, bar graphs for visualizing univariate categorical data v Use box plots, histograms for univariate numerical data v For bivariate data we can do scatter plots v For multivariate data we can do clusters v For numerical data use mean, median as measures of center v For categorical use mode v Use standard deviation, iqr for spread for numerical data v For categorical use the frequency plot or tables to summarize it v Summarization allows us to make sense of raw data and is crucial before we analyze it.

Worksheet for Part 2 v How would you summarize numerically and graphically the measures from the apple tree: • Height, canopy cover, number of apples, type of apple v There are 3 different types of apple in your data and when you cluster the apples you get the following dendogram. So you think your belief that Variety 2 and 3 were more similar than 1 and 3 are justified.

Dendogram

Overview What we wanted to learn and what we did learn

Take home messages (hopefully) v There are several types of data and each type has its own nuances. v Big data is still data and can be BAD data. Big isn’t always better. v The concept of population versus sample v Experimental, observational and opportunistic studies v Exploratory and confirmatory studies v Distinction between univariate, multivariate v To summarize type and dimension matters v Graphs are worth a thousand words

Part 3 Inference and analysis: Making Decisions from data

When and Why we need inference v IF the data we collected was really a population we do not need to do any inference. v If however, we assume that this was just a sample from a larger population of ALL WSU graduate students, staff and faculty, then we need to take this sample and make inferences about this sample to relate back to the population.

Inference for univariate response v Let us again consider this from an univariate, bivariate and multivariate stand point for both categorical and numerical data. v What are the questions we might want answered for the variable MAJOR. v What about BP?

Inference: v Use data and statistical methods to infer (get an idea, resolve a belief, estimate) something about the population based on results of a sample Inference Estimation Hypothesis Testing Point Estimation Interval Estimation

Estimation v We have no idea at all about the parameter of interest and we use the sample information to get an idea (estimate) the population parameter • Point Estimation: Using a single point to estimate the population parameter • Interval Estimation: We use an interval of values to estimate the population parameter

Hypothesis Testing v We have some idea about a population parameters. v We want to test this claim based on data we collect v Probably one of the most used and most misunderstood method used in science. v This provides us with the “dreaded p-value”.

Parameter v To understand inference, we really need to get a very clear idea about what is a parameter. v By definition: Parameter is a numerical characteristic (or characteristics if multivariate) of the population that is of interest. v To make this specific: let us consider the following example v Example: We are interested in the average number of statistics classes students have taken when they come to graduate school at WSU.

Population and Parameter: v Here the population is all graduate students at WSU. v We have to be careful here: is it all CURRENT graduate students or all students past, present and future. v To make matters easy let us say it is CURRENT graduate students. v The parameter is: the average number of statistics classes taken by the students.

Sample and Statistic v Our choices are to do a census and then compute the average from the entire census – this is the parameter v Or to take a sample and calculate the number FROM the sample. v If we use the sample and compute the number from the sample we call the sample average our STATISTIC.

How do we sample v Here we need to think of how we sample this very well defined population. v Thoughts? v Hallmarks of a good sample: representative, unbiased, reliable

Estimation: Complete Ignorance about parameter v If we use the sample statistic to get an idea of the population sample, what we are doing is inference, specifically ESTIMATION v What assures us that the sample statistic will be a good estimate of the population parameter? v This leads us to Unbiasedness and Precision

Where does Statistics come in?

Point Estimation v The idea of point estimation seems intuitive: we use the sample value for the population value. v The reason we can do this, is because we make certain assumptions about the probability distribution of the sample statistic v Generally we assume that the sampling scheme we pick allows us an unbiased and high precision distribution of the statistic. v If our method is indeed unbiased then the sample mean, should on average be a good estimator for the population mean.

Interval Estimation v Even if we believe in the unbiasedness of our estimator, we still often want an interval rather than just a single value for the estimates. v This allows us to have interval estimation. v This technique takes into account the spread as well as the distribution in the estimation. It gives us an interval of values, in which we feel that our parameter is contained with high confidence.

Confidence Interval: v In general a confidence interval for the population mean is given by: v Sample mean ± margin of error v Question is: how does one calculate “margin of error” v Answer: we need distributions and random variables to do that. v This means some mathematics and probability theory.

Confidence interval v Used quite a bit in the past v Gives similar information as hypothesis tests v Often can be inverted to construct tests v However, theoretically it is quite different, as here we talk about the SIZE of the effect rather than significance of the effect. v With all the bad press that p-values have received this might make a come back.

Hypothesis Testing: some idea about the parameter v We have some knowledge about the parameter v A claim, a warranty, what we would like it to be v We test the claim using our data; v First step: formulating the hypothesis v Always a pair: Research and Nullification of research v (affectionately called Ho and Ha)

How to formulate your hypothesis v First state your claim or research. • Let us say we believe that the average number of stats classes taken by graduate students coming into WSU is greater than 2. v Here our parameter is Population mean of statistics classes taken by graduate students at WSU • Claim: Mean > 2 v What nullifies this? • Mean ≤ 2 (Remember the “=“ always resides with the null)

Logic of Testing To actually test the hypothesis, what we try to do is to disprove or reject the null hypothesis. v If we can reject the null, by default our Ha (which is our research) is true. v Think of how the legal system works: v H 0: Not Guilty v Ha: Guilty v

How do we do this? v We take a sample and look at the sample values. Then we see if the null was true, would this be a likely value of the sample statistic. v If our observed value is not a likely value, we reject the null. v How likely or unlikely a value is, is determined by the sampling distribution of that statistic.

Example v In our example we were interested in the hypothesis about the average score in their last Stats class: v H 0: µ≤ 2 v Ha: µ > 2 v If we observed a sample with a mean of 4 and a standard deviation of 2 from a sample of 100 would you consider the null likely? ? v How about if the mean was 4 and the standard deviation was 20 from a sample of 100?

Players in decision making v Your observed statistic v Your sample size v Your observed standard deviation v Your capacity of being able to find probabilistically how likely our observed value is under the null.

Errors in testing: v Since we take our decisions about the parameter based on sample values we are likely to commit some errors. Type I error: Rejecting Ho when it is true (False Positive) Type II error: Failing to reject H 0 when Ha is true (False Negative) v In any given situation we want to minimize these errors. v P(Type I error) = a, Also called size, level of significance. P(Type II error) = b, Power = 1 -b, HERE we reject H 0 when the claim is true. We want power to be LARGE. Power is the TRUE Positive we want. v v v

Example v I am introducing a new drug into the market. The drug may have some serious side effects. Before I do so I will go through tests to see if is effective in curing disease. v H 0: not effective v Ha: drug is effective v What is Type I error and Type II error in this case? v Which is worse? More importantly think of the consequence of these errors.

One more example: v Ann Landers in her advice column on the reliability of DNA testing for determining paternity advises, “To get a completely accurate result you would have to be tested, so would the man and your mother. The test is 100% accurate if the man is NOT your father, and 99. 9% accurate if he is. Consider the hypothesis: v Ho: a particular man is the father v Ha: a particular man is not the father. v v Discuss the chances of probability of Type I and II errors.

Decision Making using Hypotheses: In general, this is the way we make decisions. v The idea is we want to minimize both Type I and II errors. v However, in practice we cannot minimize both the errors simultaneously. v What is done, is we fix our Type I error at some small level, ie 0. 1, 0. 05 or 0. 01 etc. Then we find the test that will minimize our Type II error for this fixed level of Type I error. This gives us the most powerful test. v v So in solving a hypothesis problem, we formulate our decision rule using the fixed value of Type I error. The decision rule is also called the CRITICAL VALUE.

How does rejection of null work with Critical values? v Here we calculate the value we can find a distribution of from the sample. v Then we look at this value and compare it with the distribution of the sample statistic to allow ourselves Type I error of alpha. v Based on this, if our observed value is beyond our critical value, we feel justified in rejecting the null. v CRITICISM: Choice of alpha is arbitrary. We can make alpha big or small depending on what we want our outcome to be…

P-values: Elephant in the room v Sometimes hypothesis testing can be thought to be subjective. This is because the choice of a-values may alter a decision. Hence it is thought that one should report pvalues and let the readers decide for themselves what the decision should be. v p-value or probability value is the probability of getting a value worse than the observed. If this probability is small then our observed is an unlikely value under the null and we should reject the null. Otherwise we cannot reject the null.

P-value for our example v For the hypothesis we talked about earlier: • H 0: µ≤ 2 • Ha: µ > 2 • If we observed a sample with a mean of 4 and a standard deviation of 2 from a sample of 100 would you consider the null likely? ? • How about if the mean was 4 and the standard deviation was 20 from a sample of 100? v P-value = P(Z > (4 -2)/(2)/sqrt(100) | µ≤ 2 ) <. 001 v P-value = P(Z > (4 -2)/(20)/sqrt(100) | µ≤ 2 ) <. 16

Criticism of p-values v As more and more people used p-values and with an effort to guard against the premise that “we can fool some of the people some of the time”, journals started having strict rules about p-value. v To publish you needed to show small p-values. v No SMALL p-values no publication… v This has often led to publication of ONLY significant results. v Also, led to let us get the p-value small by hook or crook attitude.

ASA Statement about p-value • It really tells us how incompatible the data are with a specified statistical model. • P-values do not measure the probability that studied hypothesis is true or that the data were produced by random chance alone. • Scientific conclusions and business policy decisions should not be based on only whether a p-value passes a specific threshold. • Proper inference requires full reporting and transparency • P-value does NOT measure the size of the effect or the importance of the result. • By itself the p-value cannot provide a good measure of evidence regarding the model.

A simple example: Term Constant X 1 Coef 3. 1586 -0. 002633 SE Coef 0. 0504 0. 000874 T-Value 62. 62 -3. 01 P-Value 0. 000 0. 003

Power: The other elephant in the room v Power is the TRUE positive v In other words what is the probability you would reject the null under a specified value of the alternative. v So first we need to figure out what value of the alternative we choose to calculate the power. This choice is up to us and we often call it the effect size.

Example of power v For the hypothesis we talked about earlier: • H 0: µ≤ 2 • Ha: µ > 2 • If we observed a sample with a mean of 4 and a standard deviation of 2 from a sample of 100. Calculate power when mu=2. 5, 3, 3. 5, 4 • Power = P (Z> (2. 5 -2)/2/10 ) = P(Z > 2. 5) =. 0162 • Etc.

Power in pictures

What are the players for power? v Sample size v Effect size v Standard deviation. v So to really calculate power one needs to have data to understand the distribution and have a feel for the standard deviation. Pre-hoc power calculation is often “trying to fool some of the people some of the time”

Recap of Part 3 v To make inferences our population of interest and parameter needs to be well defined. v Errors exist in testing and have to be considered v P-values are a measure of incompatibility of the existing data given the null hypothesis and cannot be used to PROVE anything. v To calculate power we need to look into values under the alternative and this can be subjective.

Worksheet for Part 3: v True or False: Type I error is always the worst hence w should focus on controlling it rather than Type II error. v You believe that the average time it takes students to walk from one class to another at WSU is more than the 10 minutes you are allotted. • Write out your null and alternate hypothesis • Write down what the type 1 error would be in this context. • You test it and get a p-value of. 13, does this indicate that the null is true?

Part 4: Big data, its pros and cons

What determines big data : The 5 V’s v Volume • Considered too large for regular software v Variety • Often a mix of many different data types v Velocity • Extreme speed at which data generated v Variability • Inconsistency of the data set v Veracity • How reliable is this data

How big is big? v By big we mean its volume is such that it is hard to analyze this on a single computer. v That in itself shouldn’t be problematic v But requiring specialized machines to analyze this has added to the myth and enigma of big data. v The problem with big data, at least as I see it, is some very pertinent statistical questions are bypassed when dealing with it.

Some statistical thoughts? v Is the big data a sample or a population? v If it is really a population: then analysis means constructing summary statistics. • This is bulky but not too difficult. v If it is a sample: what was the sampling frame? v If no population was considered when collecting this data, it is definitely not a representative sample. v So, should one really do inference on BIG data? v If one is allowed to do inference wouldn’t the sheer size of the data, give us so much power that we can pretty much come to any decision we test for.

Structure of data v Generally most data sets are rectangular in nature with p variables and n observations collected. v For the data we collected we have p=4 and n=100. v In big data we often have many more predictors than observations (the big p problem) v Many more (orders of magnitude more) observations than predictors, (the big n problem). v Both n and p are big and are fluid as they are constantly updated and amassed.

The Variety and Velocity Piece v Generally opportunistic data is a mix of categorical, discrete, ordinal, continuous and a mix of that as well. So if we use it as a multivariate we have to think about how to proceed. While not trivial this can be surmounted with too much difficulty. v The big issue is often the data is being amassed (I am not using collected intentionally) at a faster rate than it can actually be analyzed and summarized.

Variability and Veracity v This type of data is extremely variable and there is no systematic model in place to capture the components of variability. Modeling is very hard when you have no idea about the sources of variability in these types of data sets. v Veracity: is it measuring what we think it is? How truthful is this data? v Just because it is big, is it really good? v We all need to ponder these questions and thoughts.

Visualization of big data v Often called dashboards v Really a collection of well known age old graphs that most of you can do in excel! v It is really just summary data in pretty colors. v Don’t be fooled by these fancy terms.

Example of a Dashboard.

Analysis versus Inference v As the whole question of whether it is a sample or a population in itself is muddy let us leave inference out for now and now focus on analyzing. v A common analysis method associated with opportunistic data is predictive analysis.

Predictive Analytics and big data v Encompasses: prediction models, machine learning, data mining for prediction of the unknown using the history of the past. • If we are predicting are we inferring? ? I will assume it is okay to do that. v Exploits patterns found in historical and allows assessment of risk associated with a particular set of conditions. v Credit scoring has used predictive analytics for a long time v However, here at least in the past sampling was done to perform inference.

Techniques used in Predictive Analytics or supervised learning Regression techniques Logistic regression Time series models Survival or duration analysis Classification and Discrimination Regression Trees Neural networks Multilayer perceptron Radial basis functions Support vector machines Naïve Bayes k-nearest neighbors Geospatial predictive modeling Modeled by humans Done by machines: no model etc. , Analytical Methods Machine Learning Methods

Supervised Learning v Idea is learning from a known data set to predict the unknown. Essentially we know the class labels ahead of time. v What we need to do is find a RULE using features in the data that DISCRIMINATES effectively between the classes. v So that if we have a new observation with its features we can correctly classify it. v Machine Learning uses this idea and so it is very popular now. v v I will briefly cover this topic mostly with examples

Example 1: Turkey Thief v v v v There was this legal case in Kansas where a turkey farmer accused his neighbor of stealing turkeys from the farm. When the neighbor was arrested and the police looked in the freezer, there were multiple frozen turkeys there. The accused claimed these were WILD turkey that he had caught. The Statistician was called in to give evidence as there are some biological differences between domestic and wild turkey. So the biologist measured the bones and other body characteristic of the domestic and Wild turkeys and the Statistician built a DISCRIMANT function. They used the classification function to see if the turkeys in the freezer fell into he WILD or DOMESTIC class. THEY ALL fell in the DOMESTIC classification!

Steps Selection of features Model Fitting Model Validation using prediction of known classes Discriminant Analysis Feature selection is done by the computer No model, but computer determines the functions of the predictors used Model is validated based on prediction of known classes Machine Learning

Feature Selection v Find which of the observed variables can actually distinguish between the classes of interest. v This is variable selection v Don’t be confused and awed when people use terms like Elastic Net or Lasso at you in this context. These are fairly straightforward methods for model selection.

MODEL FITTING • • Commonly used: LDA K Nearest Neighbor QDA

Without models we can use Machine Learning methods v Neural networks v Naïve Bayes v Support Vector machines

Validation • See how well the classifiers classify the observations into the different classes. • Mostly commonly used method leave-one-out-cross validation. • Though test data set (holdout sample) and resubmissions are still used.

Recap of Part 4 v The sticky problem is if the data we have is a sample or a population. v Inference is tough, as it is hard to figure out to what population we are inferring for. v Predictive analytics often associated with big data v At the end of the day, machines are faster and more efficient but cannot create interpretative models (not yet). v We still don’t know if big data is good data, it depends upon who is collecting it and for what purpose.

Worksheet for Part 4 v A Genome wide association study was undertaken to see if we could identify the different SNPs (Single nucleotide Polymorphisms) from two groups using a case-control set up. Each group had 5000 units and we looked at 1 million SNPs in each case. We also collected other control data from bot groups. • Would you consider this a big data? • Would you consider this an opportunistic data? • Do you think we can do inferences in this case?

Overview What we wanted to learn and what we did learn

Take home messages (hopefully) v There are several types of data and each type has its own nuances. v The concept of population versus sample v Experimental, observational and opportunistic studies v Exploratory and confirmatory studies v Distinction between univariate, multivariate v To summarize type and dimension matters v Graphs are worth a thousand words

More take home messages v Estimation and Testing v Errors in testing v Decision making, power and p-values v Big data is still data and it can be bad data v Machine learning and Statistics v Dashboards are not in only in cars but a data visualization method used by “analytics” firms to produce well known graphs.

Jargon not to be awed by: v Elastic Net v Lasso v Bagging v Dashboard v Machine Learning v Neural networks v Feature selection v Deep learning v These are all statistics terms that is easy to explain and well- known and used to fool some of the people some of the time.

Myth of Big Data v There is no myth, it is just unwieldy, unstructured, undersigned data that is already being amassed. v It still has to be good data for us to make good analysis and predictions. v At the end of the day to make inferences on data (big or small) we need it to be representative.

Part 6: CISER: What we do and how we can help Graduate Students, post-docs and faculty

Statistical Help at WSU: Center for Interdisciplinary Statistics Education and Research (CISER) v Our mission • Education • Collaborative Research • Building a Community

Assistance types