Experiments Statistics Experiment Design Playtesting Experiments dont have
![Experiments & Statistics Experiments & Statistics](https://slidetodoc.com/presentation_image_h/176ece3669b8cc627d4d1275bfb63d37/image-1.jpg)
![Experiment Design • Playtesting • Experiments don’t have to be “big”--many game design experiments Experiment Design • Playtesting • Experiments don’t have to be “big”--many game design experiments](https://slidetodoc.com/presentation_image_h/176ece3669b8cc627d4d1275bfb63d37/image-2.jpg)
![Control Group • Establish a baseline • Detect any outside factors that might influence Control Group • Establish a baseline • Detect any outside factors that might influence](https://slidetodoc.com/presentation_image_h/176ece3669b8cc627d4d1275bfb63d37/image-3.jpg)
![Countering Bias • Your bias: Predict and then test against new data, don’t just Countering Bias • Your bias: Predict and then test against new data, don’t just](https://slidetodoc.com/presentation_image_h/176ece3669b8cc627d4d1275bfb63d37/image-4.jpg)
![Measurement (and Statistics) Measurement (and Statistics)](https://slidetodoc.com/presentation_image_h/176ece3669b8cc627d4d1275bfb63d37/image-5.jpg)
![Example: Measuring Time • Play N turns of a game, measuring the time per Example: Measuring Time • Play N turns of a game, measuring the time per](https://slidetodoc.com/presentation_image_h/176ece3669b8cc627d4d1275bfb63d37/image-6.jpg)
![Accuracy vs. Precision • Experiments estimate values; they are never exact • Accuracy is Accuracy vs. Precision • Experiments estimate values; they are never exact • Accuracy is](https://slidetodoc.com/presentation_image_h/176ece3669b8cc627d4d1275bfb63d37/image-7.jpg)
![Population vs. Sample • Population statistics (truth): • μ = Mean (“average” or “expected Population vs. Sample • Population statistics (truth): • μ = Mean (“average” or “expected](https://slidetodoc.com/presentation_image_h/176ece3669b8cc627d4d1275bfb63d37/image-8.jpg)
![Is the Mean Accurate? Let N = sample size Let m = sample average Is the Mean Accurate? Let N = sample size Let m = sample average](https://slidetodoc.com/presentation_image_h/176ece3669b8cc627d4d1275bfb63d37/image-9.jpg)
![Exercise • Experimental Results*: • Played N = 20 turns of Carcassonne • Average Exercise • Experimental Results*: • Played N = 20 turns of Carcassonne • Average](https://slidetodoc.com/presentation_image_h/176ece3669b8cc627d4d1275bfb63d37/image-10.jpg)
![• • • Extrapolation We usually want to measure a relatively small fraction • • • Extrapolation We usually want to measure a relatively small fraction](https://slidetodoc.com/presentation_image_h/176ece3669b8cc627d4d1275bfb63d37/image-11.jpg)
![Is the Variance Accurate? • The previous slide assumed that we knew the population Is the Variance Accurate? • The previous slide assumed that we knew the population](https://slidetodoc.com/presentation_image_h/176ece3669b8cc627d4d1275bfb63d37/image-12.jpg)
![• • • Exercise We estimated that for Carcassonne, the turn time was • • • Exercise We estimated that for Carcassonne, the turn time was](https://slidetodoc.com/presentation_image_h/176ece3669b8cc627d4d1275bfb63d37/image-13.jpg)
![Hypothesis Testing 1. Form a hypothesis 2. Design an experiment to test • Analyze Hypothesis Testing 1. Form a hypothesis 2. Design an experiment to test • Analyze](https://slidetodoc.com/presentation_image_h/176ece3669b8cc627d4d1275bfb63d37/image-14.jpg)
![Objective and Quantitative • Bad! “People played our game and said that it was Objective and Quantitative • Bad! “People played our game and said that it was](https://slidetodoc.com/presentation_image_h/176ece3669b8cc627d4d1275bfb63d37/image-15.jpg)
![Exercises Design experiments to test the following hypotheses: • • “Our new rules increased Exercises Design experiments to test the following hypotheses: • • “Our new rules increased](https://slidetodoc.com/presentation_image_h/176ece3669b8cc627d4d1275bfb63d37/image-16.jpg)
![Exercises Design experiments to test the following hypotheses: • • “Our new rules increased Exercises Design experiments to test the following hypotheses: • • “Our new rules increased](https://slidetodoc.com/presentation_image_h/176ece3669b8cc627d4d1275bfb63d37/image-17.jpg)
- Slides: 17
![Experiments Statistics Experiments & Statistics](https://slidetodoc.com/presentation_image_h/176ece3669b8cc627d4d1275bfb63d37/image-1.jpg)
Experiments & Statistics
![Experiment Design Playtesting Experiments dont have to be bigmany game design experiments Experiment Design • Playtesting • Experiments don’t have to be “big”--many game design experiments](https://slidetodoc.com/presentation_image_h/176ece3669b8cc627d4d1275bfb63d37/image-2.jpg)
Experiment Design • Playtesting • Experiments don’t have to be “big”--many game design experiments take only 30 minutes to design and conduct, and the results are obvious • Two approaches: • • • Measure a Quantity Test a Hypothesis (Can do both in the same experiment) • Experiments are much weaker than
![Control Group Establish a baseline Detect any outside factors that might influence Control Group • Establish a baseline • Detect any outside factors that might influence](https://slidetodoc.com/presentation_image_h/176ece3669b8cc627d4d1275bfb63d37/image-3.jpg)
Control Group • Establish a baseline • Detect any outside factors that might influence the experiment • e. g. , location, testing process itself, temperature, day of week, recent events
![Countering Bias Your bias Predict and then test against new data dont just Countering Bias • Your bias: Predict and then test against new data, don’t just](https://slidetodoc.com/presentation_image_h/176ece3669b8cc627d4d1275bfb63d37/image-4.jpg)
Countering Bias • Your bias: Predict and then test against new data, don’t just fit a theory to existing data • Sample bias: • • Did you select playtesters who actually represent your target market? Is your experiment designed to reveal their true preferences? (beware of incenting them to “make you happy” or to seek outcomes that they don’t actually desire) • Did you prevent them from “cheating”? • Community bias: anonymous (blind) reviews
![Measurement and Statistics Measurement (and Statistics)](https://slidetodoc.com/presentation_image_h/176ece3669b8cc627d4d1275bfb63d37/image-5.jpg)
Measurement (and Statistics)
![Example Measuring Time Play N turns of a game measuring the time per Example: Measuring Time • Play N turns of a game, measuring the time per](https://slidetodoc.com/presentation_image_h/176ece3669b8cc627d4d1275bfb63d37/image-6.jpg)
Example: Measuring Time • Play N turns of a game, measuring the time per turn • We can now predict how long the game will run without further testing, even after we change the rules. • (How large should N be? )
![Accuracy vs Precision Experiments estimate values they are never exact Accuracy is Accuracy vs. Precision • Experiments estimate values; they are never exact • Accuracy is](https://slidetodoc.com/presentation_image_h/176ece3669b8cc627d4d1275bfb63d37/image-7.jpg)
Accuracy vs. Precision • Experiments estimate values; they are never exact • Accuracy is how close your measurement is to the true value (significant digits) • Precision is the number of decimal places in your measurement
![Population vs Sample Population statistics truth μ Mean average or expected Population vs. Sample • Population statistics (truth): • μ = Mean (“average” or “expected](https://slidetodoc.com/presentation_image_h/176ece3669b8cc627d4d1275bfb63d37/image-8.jpg)
Population vs. Sample • Population statistics (truth): • μ = Mean (“average” or “expected value”) • σ = Standard deviation • Sample statistics (measured): • N = Number of samples • m = Mean • s = Sample deviation Note the n-1 where you expected to see n
![Is the Mean Accurate Let N sample size Let m sample average Is the Mean Accurate? Let N = sample size Let m = sample average](https://slidetodoc.com/presentation_image_h/176ece3669b8cc627d4d1275bfb63d37/image-9.jpg)
Is the Mean Accurate? Let N = sample size Let m = sample average Let s 2 = sample variance Assume normal distribution For N = 10, the true population mean is on the interval: m ± s 3. 250 with 99% probability. http: //onlinestatbook. com/chapter 8/mean. html N 3 4 5 10 20 50 100 95% 4. 303 s 3. 182 s 2. 776 s 2. 262 s 2. 093 s 2. 010 s 1. 984 s 99% 9. 925 s 5. 841 s 4. 604 s 3. 250 s 2. 861 s 2. 680 s 2. 626 s t distribution
![Exercise Experimental Results Played N 20 turns of Carcassonne Average Exercise • Experimental Results*: • Played N = 20 turns of Carcassonne • Average](https://slidetodoc.com/presentation_image_h/176ece3669b8cc627d4d1275bfb63d37/image-10.jpg)
Exercise • Experimental Results*: • Played N = 20 turns of Carcassonne • Average turn time was m = 20 seconds • Sample deviation was s = 1. 9 • What range are you 95% confident contains the true mean? • 95% Confidence Interval: m ± 2. 093 s Sample Times: 18 19 20 20 21 18 21 20 23 25 19 18 21 18 17 21 22 19 19 21 Conclusion: More than 95% confident that the true average turn time is between 16 and 24 seconds *Artificial Results to make computation easier
![Extrapolation We usually want to measure a relatively small fraction • • • Extrapolation We usually want to measure a relatively small fraction](https://slidetodoc.com/presentation_image_h/176ece3669b8cc627d4d1275bfb63d37/image-11.jpg)
• • • Extrapolation We usually want to measure a relatively small fraction of the population and then generalize, e. g. , political polling data. Any Distribution: At least (1 -1/k 2)*100% of the values are within μ ± kσ. (Chebyshev’s Inequality) Normal Distribution: See table. k 1 2 3 4 6 Percent within μ ± kσ Normal (=) Any Distribution (≥) 68% 95% 99. 7% 99. 999999% 0% 75% 89% 94% 97%
![Is the Variance Accurate The previous slide assumed that we knew the population Is the Variance Accurate? • The previous slide assumed that we knew the population](https://slidetodoc.com/presentation_image_h/176ece3669b8cc627d4d1275bfb63d37/image-12.jpg)
Is the Variance Accurate? • The previous slide assumed that we knew the population variables μ and σ! • We know how to tell if m is accurate. . . • But is s accurate? • Good question. In this class, we’ll just assume that it is. . .
![Exercise We estimated that for Carcassonne the turn time was • • • Exercise We estimated that for Carcassonne, the turn time was](https://slidetodoc.com/presentation_image_h/176ece3669b8cc627d4d1275bfb63d37/image-13.jpg)
• • • Exercise We estimated that for Carcassonne, the turn time was m = 20 with s = 1. 9. There are 71 turns in the game. Assume turns times are normally distributed. How many turns per game do you expect to take more than 22 seconds? • 68% within [18, 22] • 32% outside [18, 22] • Half of the 32% are on the high side • 16% chance of one turn running long • Conclusion: 71 turns * 16% ≈ 11 turns • What is the range of total play times you expect for 99. 9% of all games? mgame = 71 * m = 71 * 20 seconds = 1, 380 seconds = 23 minutes • • s = 71 * 1. 9 ; s = 16 seconds • Normal distribution, so 99. 7% within 3 standard deviations (48 seconds) • Conclusion: About 99. 9% of games within 22 - 24 minutes. game 2 2 2 game
![Hypothesis Testing 1 Form a hypothesis 2 Design an experiment to test Analyze Hypothesis Testing 1. Form a hypothesis 2. Design an experiment to test • Analyze](https://slidetodoc.com/presentation_image_h/176ece3669b8cc627d4d1275bfb63d37/image-14.jpg)
Hypothesis Testing 1. Form a hypothesis 2. Design an experiment to test • Analyze the statistical validity of the test 3. Run the experiment 4. Evaluate results 5. (often. . . go back to step 1)
![Objective and Quantitative Bad People played our game and said that it was Objective and Quantitative • Bad! “People played our game and said that it was](https://slidetodoc.com/presentation_image_h/176ece3669b8cc627d4d1275bfb63d37/image-15.jpg)
Objective and Quantitative • Bad! “People played our game and said that it was fun, therefore it was engaging. ” • Better “On average, our game was 2 nd in a ranking from `most fun’ to `least fun’ of ten other commercial games in a survey of 100 players. 20% of subjects rated our game #1” • Good “ 100 subjects were randomly assigned to play our game or a hand-made version of Pit. They then decided individually which game to play again. 82% of respondents chose to play our game, so we conclude that it is about 4 times more engaging than Pit. ”
![Exercises Design experiments to test the following hypotheses Our new rules increased Exercises Design experiments to test the following hypotheses: • • “Our new rules increased](https://slidetodoc.com/presentation_image_h/176ece3669b8cc627d4d1275bfb63d37/image-16.jpg)
Exercises Design experiments to test the following hypotheses: • • “Our new rules increased engagement in the game. ” “The chance of drawing an unplayable tile in Carcassonne is less than 0. 1%. ” “Experienced players usually choose the highest resource intersection first and then maximize resource distribution second in Settlers of Catan. ” “In Guitar Hero, the intro for More Than a Feeling is harder than the chorus for most players. ”
![Exercises Design experiments to test the following hypotheses Our new rules increased Exercises Design experiments to test the following hypotheses: • • “Our new rules increased](https://slidetodoc.com/presentation_image_h/176ece3669b8cc627d4d1275bfb63d37/image-17.jpg)
Exercises Design experiments to test the following hypotheses: • • “Our new rules increased engagement in the game. ” “The chance of drawing an unplayable tile in Carcassonne is less than 0. 1%. ” “Experienced players usually choose the highest resource intersection first and then maximize resource distribution second in Settlers of Catan. ” “In Guitar Hero, the intro for More Than a Feeling is harder than the chorus for most players. ”
Dont ask dont tell political cartoon
Dont laugh at me dont call me names
Introduction to statistics what is statistics
Shape with 6 faces and 12 edges
Design of experiments doe
Jmp doe analysis
Design of experiments six sigma
Design of experiments pmp example
Design of experiments presentation
Worked out
Pmp stakeholder engagement plan
Plackett burman method
Design of experiments voorbeeld
Design of experiments
Torrent qq
What you dont know wont hurt
Dont lequel ou dans lequel
Please dont eat my cookie