Lecture 14 Thurs Oct 23 Multiple Comparisons Sections
![Lecture 14 – Thurs, Oct 23 • Multiple Comparisons (Sections 6. 3, 6. 4). Lecture 14 – Thurs, Oct 23 • Multiple Comparisons (Sections 6. 3, 6. 4).](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-1.jpg)
![Compound Uncertainty • Compound uncertainty: When drawing more than one direct inference, there is Compound Uncertainty • Compound uncertainty: When drawing more than one direct inference, there is](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-2.jpg)
![Simultaneous Inferences • When several 95% confidence intervals are considered simultaneously, they constitute a Simultaneous Inferences • When several 95% confidence intervals are considered simultaneously, they constitute a](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-3.jpg)
![Individual vs. Family Confidence Levels • If a family consists of k confidence intervals, Individual vs. Family Confidence Levels • If a family consists of k confidence intervals,](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-4.jpg)
![Familywise Confidence Levels • K Lower Bound 3 85% Confidence level if independent 86% Familywise Confidence Levels • K Lower Bound 3 85% Confidence level if independent 86%](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-5.jpg)
![Multiple Comparison Procedures • Multiple comparison procedures are methods of constructing individual confidence intervals Multiple Comparison Procedures • Multiple comparison procedures are methods of constructing individual confidence intervals](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-6.jpg)
![Planned vs. Unplanned Comparisons • Consider one-way classification with 20 groups. • Planned Comparisons: Planned vs. Unplanned Comparisons • Consider one-way classification with 20 groups. • Planned Comparisons:](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-7.jpg)
![Families in Planned/Unplanned • Planned Comparisons: The family of confidence intervals is the family Families in Planned/Unplanned • Planned Comparisons: The family of confidence intervals is the family](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-8.jpg)
![Multiple Comparison Procedures • Confidence Interval: Estimate Margin of Error = (Multiplier)x(Standard Error of Multiple Comparison Procedures • Confidence Interval: Estimate Margin of Error = (Multiplier)x(Standard Error of](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-9.jpg)
![Tukey-Kramer Procedure • Based on computing the distribution of the largest |t| statistic under Tukey-Kramer Procedure • Based on computing the distribution of the largest |t| statistic under](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-10.jpg)
![Tukey-Kramer example • For multiplecomp. JMP, n=200, I=20, so from Table A. 5 using Tukey-Kramer example • For multiplecomp. JMP, n=200, I=20, so from Table A. 5 using](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-11.jpg)
![Tukey-Kramer in JMP • To see which groups are significantly different (in sense of Tukey-Kramer in JMP • To see which groups are significantly different (in sense of](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-12.jpg)
![Bonferroni Method • Bonferroni Method: If we have a family of k confidence intervals, Bonferroni Method • Bonferroni Method: If we have a family of k confidence intervals,](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-13.jpg)
![Bonferroni for tests • Suppose we are conducting k hypothesis tests and will “reject” Bonferroni for tests • Suppose we are conducting k hypothesis tests and will “reject”](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-14.jpg)
![Bonferroni for mice diets • Five comparisons were planned. Suppose we want the familywise Bonferroni for mice diets • Five comparisons were planned. Suppose we want the familywise](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-15.jpg)
![Exploratory Data Analysis and Multiple Comparisons • Searching data for suggestive patterns can lead Exploratory Data Analysis and Multiple Comparisons • Searching data for suggestive patterns can lead](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-16.jpg)
![Review of One-way layout • Assumptions of ideal model – All populations have same Review of One-way layout • Assumptions of ideal model – All populations have same](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-17.jpg)
![Review example • Case Study 6. 1. 1: Discrimination against the handicapped. • Randomized Review example • Case Study 6. 1. 1: Discrimination against the handicapped. • Randomized](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-18.jpg)
- Slides: 18
![Lecture 14 Thurs Oct 23 Multiple Comparisons Sections 6 3 6 4 Lecture 14 – Thurs, Oct 23 • Multiple Comparisons (Sections 6. 3, 6. 4).](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-1.jpg)
Lecture 14 – Thurs, Oct 23 • Multiple Comparisons (Sections 6. 3, 6. 4). • Next time: Simple linear regression (Sections 7. 1 -7. 3)
![Compound Uncertainty Compound uncertainty When drawing more than one direct inference there is Compound Uncertainty • Compound uncertainty: When drawing more than one direct inference, there is](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-2.jpg)
Compound Uncertainty • Compound uncertainty: When drawing more than one direct inference, there is an increased chance of making at least one mistake. • Impact on tests: If using a conventional criteria such as a p-value of 0. 05 to reject a null hypothesis, the probability of falsely rejecting a null hypothesis will be greater than 0. 05 if considering multiple tests. • Impact on confidence intervals: If forming multiple 95% confidence intervals, the chance that all of the confidence intervals will contain true parameter is less than 95%.
![Simultaneous Inferences When several 95 confidence intervals are considered simultaneously they constitute a Simultaneous Inferences • When several 95% confidence intervals are considered simultaneously, they constitute a](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-3.jpg)
Simultaneous Inferences • When several 95% confidence intervals are considered simultaneously, they constitute a family of confidence intervals • Individual Confidence Level: Success rate of a procedure for constructing a single confidence interval. • Familywise Confidence Level: Success rate of procedure for constructing a family of confidence intervals, where a “successful” usage is one in which all intervals in the family capture their parameters.
![Individual vs Family Confidence Levels If a family consists of k confidence intervals Individual vs. Family Confidence Levels • If a family consists of k confidence intervals,](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-4.jpg)
Individual vs. Family Confidence Levels • If a family consists of k confidence intervals, each with individual confidence level 95%, the familywise confidence levels can be no larger than 95% and no smaller than 100(1 -. 05 k)%. • Actual familywise confidence levels depends on degree of dependence between intervals. • If the intervals are independent, the familywise confidence level is 100(. 95)k%.
![Familywise Confidence Levels K Lower Bound 3 85 Confidence level if independent 86 Familywise Confidence Levels • K Lower Bound 3 85% Confidence level if independent 86%](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-5.jpg)
Familywise Confidence Levels • K Lower Bound 3 85% Confidence level if independent 86% 5 75% 77% 20 0% 36% 100 0% 1%
![Multiple Comparison Procedures Multiple comparison procedures are methods of constructing individual confidence intervals Multiple Comparison Procedures • Multiple comparison procedures are methods of constructing individual confidence intervals](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-6.jpg)
Multiple Comparison Procedures • Multiple comparison procedures are methods of constructing individual confidence intervals so that familywise confidence level is controlled (at 95% for example). • Key issue: What is the appropriate family to consider?
![Planned vs Unplanned Comparisons Consider oneway classification with 20 groups Planned Comparisons Planned vs. Unplanned Comparisons • Consider one-way classification with 20 groups. • Planned Comparisons:](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-7.jpg)
Planned vs. Unplanned Comparisons • Consider one-way classification with 20 groups. • Planned Comparisons: researcher is specifically interested in comparing groups 1 and 4 because comparison answers a research question directly. This is a planned comparison. In the mice diets example, the researchers had five planned comparisons. • Unplanned Comparisons: researcher examines all possible pairs of groups – 190 groups. As a result, researcher finds that only groups 5 and 8 suggest actual differences. Only this pair is reported as significant.
![Families in PlannedUnplanned Planned Comparisons The family of confidence intervals is the family Families in Planned/Unplanned • Planned Comparisons: The family of confidence intervals is the family](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-8.jpg)
Families in Planned/Unplanned • Planned Comparisons: The family of confidence intervals is the family of all planned comparison confidence intervals (e. g. , the family of five planned comparisons in mice diet). For small number of planned comparisons, it is usual practice to just use individual confidence intervals controlled at 95%. • Unplanned Comparisons: The family of confidence intervals is the family of all possible comparisons - (k*(k-1)/2) for a k-group one-way classification. It is important to control the familywise confidence level for unplanned comps.
![Multiple Comparison Procedures Confidence Interval Estimate Margin of Error MultiplierxStandard Error of Multiple Comparison Procedures • Confidence Interval: Estimate Margin of Error = (Multiplier)x(Standard Error of](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-9.jpg)
Multiple Comparison Procedures • Confidence Interval: Estimate Margin of Error = (Multiplier)x(Standard Error of Estimate). For multiple comparison procedures, the multipliers is greater than the usual 2. • Multiple comparisons procedures – Tukey/Kramer’s “Honest Significant Differences” – Bonferroni
![TukeyKramer Procedure Based on computing the distribution of the largest t statistic under Tukey-Kramer Procedure • Based on computing the distribution of the largest |t| statistic under](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-10.jpg)
Tukey-Kramer Procedure • Based on computing the distribution of the largest |t| statistic under the null hypothesis that all group means are equal. • Family of confidence intervals for all group mean differences that has 95% familywise confidence level: • can be found on Table A. 5. For df=n-I, use closest df > n-I on chart.
![TukeyKramer example For multiplecomp JMP n200 I20 so from Table A 5 using Tukey-Kramer example • For multiplecomp. JMP, n=200, I=20, so from Table A. 5 using](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-11.jpg)
Tukey-Kramer example • For multiplecomp. JMP, n=200, I=20, so from Table A. 5 using df=n-I= (chart only goes up to 120), • Tukey-Kramer family of confidence intervals with 95% familywise level: • Examples: Tukey-Kramer confidence interval for is and are not significantly different (in sense of statistical significance) using Tukey-Kramer since CI contains 0.
![TukeyKramer in JMP To see which groups are significantly different in sense of Tukey-Kramer in JMP • To see which groups are significantly different (in sense of](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-12.jpg)
Tukey-Kramer in JMP • To see which groups are significantly different (in sense of statistical significance), i. e. , which groups have CI for difference in group means that does not contain 0, click Compare Means under Oneway Analysis (after Analyze, Fit Y by X) and click All Pairs, Tukey’s HSD. • In table “Comparison of All Pairs Using Tukey’s HSD, ” two groups are significantly different if and only if the entry in the table for the pair of groups is positive.
![Bonferroni Method Bonferroni Method If we have a family of k confidence intervals Bonferroni Method • Bonferroni Method: If we have a family of k confidence intervals,](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-13.jpg)
Bonferroni Method • Bonferroni Method: If we have a family of k confidence intervals, to form individual confidence intervals that have a familywise confidence level of 95%, make the individual confidence intervals have confidence level 100(1. 05/k)%. • General method for doing multiple comparisons. • Bonferroni Inequality:
![Bonferroni for tests Suppose we are conducting k hypothesis tests and will reject Bonferroni for tests • Suppose we are conducting k hypothesis tests and will “reject”](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-14.jpg)
Bonferroni for tests • Suppose we are conducting k hypothesis tests and will “reject” the null hypothesis if the p-value is smaller than a cutoff p* (e. g. , p* =. 05). • Per-test type I error rate: the probability of falsely rejecting the null hypothesis when it is true. The per-test type I error rate is p*. • Familywise error rate: the probability of falsely rejecting at least one null hypothesis in a family of tests when all null hypotheses are true. • Bonferroni for tests: For a family of k tests, use a cutoff of p*/k to obtain a familywise error rate of at most p*, e. g. , for ten tests, reject if p-value <0. 005 to obtain familywise error rate of at most 0. 05.
![Bonferroni for mice diets Five comparisons were planned Suppose we want the familywise Bonferroni for mice diets • Five comparisons were planned. Suppose we want the familywise](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-15.jpg)
Bonferroni for mice diets • Five comparisons were planned. Suppose we want the familywise error rate for the five comparisons to be 0. 05. • Bonferroni method: We should consider two groups to be significantly different if the pvalue from the two-sided t-test is less than 0. 05/5=0. 01. •
![Exploratory Data Analysis and Multiple Comparisons Searching data for suggestive patterns can lead Exploratory Data Analysis and Multiple Comparisons • Searching data for suggestive patterns can lead](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-16.jpg)
Exploratory Data Analysis and Multiple Comparisons • Searching data for suggestive patterns can lead to important discoveries but it is difficult to test a hypothesis against a data set which suggested it. We must protect against “data snooping. ” • One way to try to protect against data snooping is to use multiple comparisons procedures (e. g. , example in Section 6. 5. 2). • The best way to protect against data snooping is to design a study to search specifically for a pattern that was suggested by an exploratory data analysis. In other words we convert an “unplanned” comparison into a “planned” comparison by doing a new experiment.
![Review of Oneway layout Assumptions of ideal model All populations have same Review of One-way layout • Assumptions of ideal model – All populations have same](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-17.jpg)
Review of One-way layout • Assumptions of ideal model – All populations have same standard deviation. – Each population is normal – Observations are independent • Planned comparisons: Usual t-test but use all groups to estimate. If many planned comparisons, use Bonferroni to adjust for multiple comparisons • Test of vs. alternative that at least two means differ: one-way ANOVA F-test • Unplanned comparisons: Use Tukey-Kramer procedure to adjust for multiple comparisons.
![Review example Case Study 6 1 1 Discrimination against the handicapped Randomized Review example • Case Study 6. 1. 1: Discrimination against the handicapped. • Randomized](https://slidetodoc.com/presentation_image_h/16ed7f457d905a9edb84d408649a986f/image-18.jpg)
Review example • Case Study 6. 1. 1: Discrimination against the handicapped. • Randomized experiment to study how physical handicaps affect people’s perception of employment qualifications. • Researchers prepared five videotaped job interviews with same two male actors in each. • Tapes differed only in that applicant appeared with a different handicap: wheelchair, crutches, hearing impaired, leg amputated, no handicap. • Seventy undergraduates were randomly assigned
Xkcd 882
01:640:244 lecture notes - lecture 15: plat, idah, farad
Example of mimd
Multiple probe vs multiple baseline
Alkenes chart
Działania na systemach liczbowych
Erg vizsgálat
Hidrocarbonetos ramificados
Mnemonic for hydrocarbons
Oct 31st sunset
Jhlt. 2019 oct; 38(10): 1015-1066
Ims design software
Propiedades químicas del carbono
Prop but pent hex hept oct
Oct 3 1993
Meth eth prop but pent hex hept oct non dec
Premium sanitas
Jhlt. 2019 oct; 38(10): 1015-1066
Quad pent hex hept oct