Design of Experiments And Data Analysis Part 1

Design of Experiments And Data Analysis Part 1 Human Centered Robotics Tommaso Lisini Baldi and Prof. Domenico Prattichizzo

The term experiment is defined as the systematic procedure carried out under controlled conditions in order to discover an unknown effect, to test or establish a hypothesis, or to illustrate a known effect. When analyzing a process, experiments are often used to evaluate: • which process inputs have a significant impact on the process output, • what the target level of those inputs should be to achieve a desired result (output). Experiments can be designed in many different ways to collect this information. T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Motivation 2

You need a general knowledge of statistics: • Histogram, • Regression and Correlation Analysis • Continuous probability distribution T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Requested Knowledge 3

T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Experimental Design 4

Ideally, your experimental design should: • Describe how participants are allocated to experimental groups. • • A common method is completely randomized design, where participants are assigned to groups at random. A second method is randomized block design, where participants are divided into homogeneous blocks (for example, age groups) before being randomly assigned to groups. • Minimize or eliminate confounding variables, which can offer alternative explanations for the experimental results. • Allow you to make inferences about the relationship between independent variables and dependent variables. • Reduce variability, to make it easier for you to find differences in treatment outcomes. T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Experimental Design 5

Design of experiments involves: • The systematic collection of data • A focus on the design itself, rather than the results • Planning changes to independent (input) variables and the effect on dependent variables or response variables • Ensuring results are valid, easily interpreted, and definitive. The most important principles are: • Randomization: the assignment of study components by a completely random method, like simple random sampling. Randomization eliminates bias from the results • Replication: the experiment must be replicable by other researchers. This is usually achieved with the use of statistics like the standard error of the sample mean or confidence intervals. T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Experimental Design 6

Components of Experimental Design Consider the following diagram of a cake-baking. There are three aspects of the process that are analyzed by a designed experiment: T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Experimental Design 7

Components of Experimental Design Consider the following diagram of a cake-baking. There are three aspects of the process that are analyzed by a designed experiment: T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Experimental Design 8

Components of Experimental Design Inputs to the process (Factors). Can be classified as either controllable or uncontrollable variables. In this case, the controllable factors are the ingredients for the cake and the oven that the cake is baked in. The controllable variables will be referred to throughout the material as factors. • Note that the ingredients list was shortened for this example - there could be many other ingredients that have a significant bearing on the end result (oil, water, flavoring, etc). • Likewise, there could be other types of factors, such as the mixing method or tools, the sequence of mixing, or even the people involved. T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Experimental Design 9

Components of Experimental Design Levels, or settings of each factor in the study. • Examples include the oven temperature setting and the particular amounts of sugar, flour, and eggs chosen for evaluation. T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Experimental Design 10

Components of Experimental Design Output of the experiment (Response). In the example of cake baking: the taste, consistency, and appearance of the cake are measurable outcomes potentially influenced by the factors and their respective levels. Experimenters often desire to avoid optimizing the process for one response at the expense of another. For this reason, important outcomes are measured analyzed to determine the factors and their settings that will provide the best overall outcome for the critical-toquality characteristics - both measurable variables and assessable attributes. T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Experimental Design 11

Designed experiments have many potential uses in improving processes and products, including: • Comparing Alternatives. In the case of our cake-baking example, we might want to compare the results from two different types of flour. If it turned out that the flour from different vendors was not significant, we could select the lowest-cost vendor. If flour was significant, then we would select the best flour. The experiment(s) should allow us to make an informed decision that evaluates both quality and cost. • Identifying the Significant Inputs (Factors) Affecting an Output (Response) - separating the vital few from the trivial many. We might ask a question: "What are the significant factors beyond flour, eggs, sugar and baking? " T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Purpose of Experimentation 12

• Achieving an Optimal Process Output (Response). "What are the necessary factors, and what are the levels of those factors, to achieve the exact taste and consistency of the chocolate cake? • Reducing Variability. "Can the recipe be changed so it is more likely to always come out the same? " • Minimizing, Maximizing, or Targeting an Output (Response). "How can the cake be made as moist as possible without disintegrating? " • Improving process or product "Robustness" - fitness for use under varying conditions. "Can the factors and their levels (recipe) be modified so the cake will come out nearly the same no matter what type of oven is used? " T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Purpose of Experimentation 13

The Design of an experiment addresses the questions outlined above by agreeing on the following: • The factors to be tested. • The levels of those factors. • The structure and layout of experimental runs, or conditions. A well-designed experiment is as simple as possible - obtaining the required information in a cost effective and reproducible manner. T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Experiment Design Guidelines 14

When designing an experiment, pay particular heed to four potential traps that can create experimental difficulties: • In addition to measurement errors, other sources of error, or unexplained variation, can obscure the results. • Note that the term "error" is not a synonym with "mistakes". Error refers to all unexplained variation that is either within an experiment run or between experiment runs and associated with level settings changing. Properly designed experiments can identify and quantify the sources of error. T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Experiment Design Guidelines 15

• Uncontrollable factors that induce variation under normal operating conditions are referred to as "Noise Factors". These factors, such as multiple machines, multiple shifts, raw materials, humidity, etc. , can be built into the experiment so that their variation doesn't get concentrated into the unexplained, or experiment error. • A key strength of Designed Experiments is the ability to determine factors and settings that minimize the effects of the T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Experiment Design Guidelines 16

• Correlation can often be confused with causation. Two factors that vary together may be highly correlated without one causing the other - they may both be caused by a third factor. • Consider the example of a porcelain enameling operation that makes vassels. The manager notices that there are intermittent problems with an unacceptable roughness (peel of orange) in the surface. The manager also notices that the orange peel is worse on days with a low production rate. T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Experiment Design Guidelines 17

A plot of orange peel vs. production volume below illustrates the correlation: T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Experiment Design Guidelines 18

If the data are analyzed without knowledge of the operation, a FALSE conclusion could be reached that low production rates cause orange peel. The truth is that low production rates and orange peel are caused by employees with less skill. T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Experiment Design Guidelines They are correlated, but low production does not cause orange peel! This example highlights the importance of factoring in operational knowledge when designing an experiment. 19

Or even worse…. T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Experiment Design Guidelines 20

One of the most common errors in Experiment design is the Trial and Error approach. Roughly design the experiment Conduct an experimental campaign Check preliminary and not stable results T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Experiment Design Guidelines 21

A good practice: T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Experiment Design Guidelines 22

• One of the most common types of experiments is the comparison of two process methods, or two methods of treatment. • There are several ways to analyze such an experiment depending upon the information available from the population as well as the sample. T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Experiment Design Guidelines 23

• One of the most straight-forward methods to evaluate a new process method is to plot the results on an SPC (statistical process control) chart that also includes historical data from the baseline process, with established control limits. T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Experiment Design Guidelines 24

• Or use a scatterplot to investigate correlation T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Experiment Design Guidelines 25

• An alternative to the control chart approach is to use: • The T-Test (or F-test - F-ratio) to compare the means of alternate treatments. • The ANOVA (Analysis of Variance) T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Experiment Design Guidelines 26

Example • We want to find the quicker route home from work. • There are two alternatives to bypass traffic bottlenecks. • We timed the trip home over a month, recording ten data points for each alternative. T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Experiment Design Guidelines 27

T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Experiment Design Guidelines 28

• As shown on the table above, both new routes home (B e C) appear to be quicker than the existing route A. • To determine whether the difference in treatment means is due to random chance or a statistically significant different process, an ANOVA F-test is performed. • The test analysis is the basis for model evaluation of both single factor and multifactor experiments. T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Experiment Design Guidelines 29

• This analysis is commonly output as an ANOVA table by statistical analysis software (we will use SPSS), as illustrated by the table below: T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Experiment Design Guidelines 30

• The most important output of the table is the F-ratio (3. 61). • The F-ratio is equivalent to the Mean Square (variation) between the groups (treatments, or routes home in our example) of 19. 9 divided by the Mean Square error within the groups (variation within the given route samples) of 5. 51. T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Experiment Design Guidelines 31

• The Model F-ratio of 3. 61 implies the model is significant. • The p-value of 0. 0408 indicates that there is only a 4. 08% probability that a Model Fratio this large could occur due to random chance. In other words, the three routes differ significantly in terms of the time taken to reach home from work. T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Experiment Design Guidelines 32

T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Experiment Design Guidelines 33

• • If an interval includes the value of zero (meaning 'zero difference'), the corresponding pair of means do NOT differ significantly. You can use these intervals to identify which of the three routes is different and by how much. The intervals contain the likely values of differences of treatment means (1 -2), (1 -3) and (2 -3) respectively, each of which is likely to contain the true (population) mean difference in 95 out of 100 samples. Notice the second interval (1 -3) does not include the value of zero; the means of routes 1 (A) and 3 (C) differ significantly. In fact, all values included in the (1, 3) interval are T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Experiment Design Guidelines 34

T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 From Theory to Practice 35

T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Experimental goals 36

T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Experimental goals 37

• determine whether an association or correlation between two or more variables exists; • if such an association or correlation does exist, measure the strength and direction of the association/correlation. • Even though we use the words "association" and "correlation", you will often see research questions that measure associations/correlations using the word "relationship", or sometimes not mention any of these words. T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Association and Correlation 38

Examples • Is there an association between exam performance and time spent revising? (OR) Is there a relationship between exam performance and time spent revising? • Is there an association between depression and length of unemployment? (OR) Does the length a person is unemployed increase their level of depression? • Is there an association between maximal aerobic capacity and age? (OR) Is your maximal aerobic capacity related to your age? • … (OR) … T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Association and Correlation 39

Available test, selection based on # of variables • How many variables you have in your analysis? • The statistical tests that you can use to measure association can be broadly divided into those that assess the association between two variables and those that assess association between >= three or more variables. T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Association and Correlation 40

• How each variables was measured? • Are on continuous, ordinal, dichotomous or multinomial scale? • It is important to note that your two variables can be measured on a different scale or they can be both measured on the same scale T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Association and Correlation 41

Continuous variables: Continuous variables include both interval and ratio measurement types. • Interval variables can be measured along a continuum and have a numerical value. • Ratio variables are interval variables, but with the added condition that a 0 (zero) measurement indicates that there is none of that variable • Examples: • • • Distance in meters, which is measured on a ratio scale. Temperature in degrees Celsius, which is measured on an interval scale. Revision time in hours, which is measured on a ratio scale. Intelligence using IQ score, which is measured on an interval scale. Exam performance using a score of 0 to 100 T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Association and Correlation 42

Ordinal variables have two or more categories that are ordered or ranked. • Examples: • • • Attitude towards abortion, measured on a 5 point Likert scale: 1 - Strongly agree, 2 - Agree, 3 - Neither agree nor disagree, 4 - Disagree, 5 - Strongly disagree (N. B. , this is known as a 'rating' scale). Desire to join the donor list, measured on a 3 point scale: 1 - Not at all, 2 - A little, 3 - A great deal (N. B. , this is known as a 'rating' scale). Attitude towards debt, measured on a 10 -point scale: Open 1 2 3 4 5 6 7 8 9 10 Closed (N. B. , this is known as a 'semantic differential' scale). T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Association and Correlation 43

Multinomial variables are nominal variables with three or more categories. Multinomial variables do not have an intrinsic order (i. e. , they cannot be ranked, unlike ordinal variables) Examples: • • Mobile phone brand, with a choice of three categories: Samsung, Apple or Other. Degree type, with a choice of four categories: Business, Engineering, Medicine or Psychology. Hair colour, with a choice of five categories: Brown, Black, Blond, Red or Grey. Preferred season: Spring, Summer, Autumn, Winter T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Association and Correlation 44

T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Experimental goals 45

• • • Determine whethere are differences between two or more groups of related and/or unrelated (independent) cases on a dependent variable; if such differences exist, determine where these differences lie (i. e. , when you have three or more groups). Statistical tests that are used to determine whethere are differences between groups can be used for a wide range of study designs, • • • between-subjects designs that involve unrelated (independent) groups, within-subjects designs that involve related groups, mixed designs that have both related and unrelated groups. At their simplest, these involve testing for differences between just two related or unrelated groups, but they can be far more involved, incorporating multiple groups, multiple conditions/treatments and multiple dependent variables. Most often, you are comparing the mean T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Group Differences 46

Examples: • Is there a difference in salary between male and female doctors? • Is there a difference in daily cigarette use amongst heavy smokers before and after a hypnotherapy programme? • Is there a difference in productivity amongst packers at a factory based on the use of background music? • Is there a difference in the time it takes a new brand of boiler to reach boiling point based on the number of uses? T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Group Differences 47

Between-subjects designs, • the "cases" in your sample are assigned to groups based on: (a) a specific characteristic that the case has; (b) a different treatment or condition that is to be administered to each group. These "cases" could be people, animals, objects, organisations, and so forth. • • Remember that the central characteristic of between-subjects designs is that your independent variable(s) has "independent groups". This means that there must be different participants in each group with no participant in more than one group. T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Group Differences 48

Between-subjects designs – Example: Cases are assigned to groups based on a specific characteristic that each case has. • Imagine that the "cases" you are interested in are doctors in a hospital. • You want to determine whethere is a difference in salary between male and female doctors. • Therefore, your dependent variable would be "salary • The independent variable would be "gender". • This independent variable, "gender", consists of two "independent groups": • • Group #1 : Male doctors Group #2 : Female doctors T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Group Differences 49

T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Group Differences 50

Advantages and Disadvantages of Between Subjects Design. • Advantages. • • • Between subjects design is one of the simplest types of experimental design setup. Multiple treatments and treatment levels can be tested at the same time. This type of design can be completed quickly. T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Group Differences 51

• Disadvantages. • • • A major disadvantage in this type of experimental design is that as each participant is only being tested once, the addition of a new treatment requires the formation of another group. The design can become extremely complex if more than a few treatments are being tested Differences in individuals (i. e. age, race, sex) may skew results and are almost impossible to control for in this experimental design. Bias can be an issue unless you control for this factor using experimental blinds (either a single blind experiment–where the participant doesn’t know if they are getting a treatment or placebo–or a double blind, where neither the participant nor the researcher know). Generalization issues means that you may not be able to extrapolate your results to a wider audience. Environmental bias can be a problem with between subjects design. For example, let’s say you were giving one group of college students a standardized T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Group Differences 52

within-subjects designs ( or repeated-measures designs) • • the "cases" in your sample all form part of what are known as "related groups". These "cases" could be people, animals, objects, organisations, and so forth. "Related groups" means that all cases are present in all groups. Possibly the easiest way to think about related groups is to look at study designs that • • (a) have more than one time point and (b) involve cases undergoing two or more treatments/ conditions T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Group Differences 53

within-subjects designs – Example (1): • • • Cases undergo two or more treatments/conditions. Imagine that the "cases" you are interested in are packers within a factory. You want to determine whethere is a difference in productivity amongst these packers based on the use of background music. Therefore, your dependent variable would be "productivity" (e. g. , the number of boxes packed in a week), The independent variable would be "background music", consisting of two "related groups": • • T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Group Differences Related group #1: No music (Control conditions) Related group #2: Music (Treatment conditions) "No music" is the control condition because packers are not normally allowed to listen to background music, so providing background music makes "music" the treatment condition. All packers would be first measured under the control conditions (i. e. , no music), and then measured under the treatment conditions 54

within-subjects designs – Example (2): • • Cases are measured two or more times during an experiment. Imagine that the "cases" you are interested in are heavy smokers. You want to determine whethere is a difference in daily cigarette use amongst these smokers after a hypnotherapy programme. The dependent variable would be "cigarette use", whilst your independent variable would be "time", consisting of two "related groups": • • T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Group Differences Related group #1: Before hypnotherapy programme (time point #1) Related group #2: After hypnotherapy programme (time point #2) The smokers in related groups #1 and #2 are the same. The groups are "related" because the dependent variable, "cigarette use" is measured at two "time points" for the 55

T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Group Differences 56

Advantages and Disadvantages of Within Subjects Experimental Design. • Advantages. • • It requires fewer participants than the between subjects design. If a between subjects design were used for the blood pressure example above, double the amount of participants would be required. Within subjects design therefore requires fewer resources and is generally cheaper. Individual difference between participants are controlled for, as each participant acts as their own control. As the subjects are measured multiple times, this better enables the researcher to hone in on individual differences T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Group Differences 57

• Disadvantages. • • • Effects from one test could carry over to the next, a phenomenon called the “range effect. ” In the blood pressure example, if participants were asked to watch the scary movie first, their blood pressure could stay elevated for hours afterwards, skewing the results from the romantic comedy. Participants can exhibit “practice effects”, where they improve scores simply by taking the same test multiple times. This is often an issue on pre-test/post-test studies. Data is not completely independent, which may effect running hypothesis tests, like T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Group Differences 58

Mixed designs • include the characteristics of betweensubjects designs and within-subjects designs. • you can have one or more betweensubjects factor (i. e. , between-subjects independent variable) and one or more within-subjects factor (i. e. , within-subjects independent variable). Take the following example: • We are not interested in Mixed design T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Group Differences 59

T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 … and what about qualitative aspects of users’ experience? … 60

• The same tests and approaches are still exploitable, the problem is to quantify users’ feedback • Some questionnaires could be use to transform opinions and feedbacks into numbers T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Qualitative analysis 61

T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 Qualitative analysis 62

T. Lisini Baldi: Design of Experiments And Data Analysis – Part 1 ! 63