Experiments in Computer Science Experiments in Computer Science

  • Slides: 9
Download presentation
Experiments in Computer Science

Experiments in Computer Science

Experiments in Computer Science "The fundamental principle of science, the definition almost, is this:

Experiments in Computer Science "The fundamental principle of science, the definition almost, is this: the sole test of the validity of any idea is experiment" – Richard P. Feynman • Tried and true experimental scientific • methodology from Physics, Biology, Chemistry. . . – Often not followed in Computer Science Let's be better Computer Scientists!

Scientific Methodology • Observe • • • – (Devise solution) Hypothesize Design Experiment Analyze

Scientific Methodology • Observe • • • – (Devise solution) Hypothesize Design Experiment Analyze Report

Methodology: Observe and Understand • Find Problem • – Test: make of Linux kernel

Methodology: Observe and Understand • Find Problem • – Test: make of Linux kernel – Build: memory intensive programs – Read: Linux Hacker’s guide says … Understand Relationships – Hard page faults are expensive – Logical memory larger than physical

Methodology: Devise and Hypothesize • Devise Solution (unless empirical) • – Claypool Reliable Audio

Methodology: Devise and Hypothesize • Devise Solution (unless empirical) • – Claypool Reliable Audio Protocol (CRAP) – Claypool buffering algorithm Make Hypothesis – Generalization about relationships – Soft page faults are common – Malloc does not cause page fault – Needs to be tested (not proven)

Methodology: Experiment • Design Experiment • – Variable: variable workload – Control: baseline workload

Methodology: Experiment • Design Experiment • – Variable: variable workload – Control: baseline workload Run Experiment “Whoa! That’s not what I expected!” – Bug in code + Back to “Run” – Uncontrolled event (system backup) + Back to “Design” – Insufficient understanding (Unix scheduling) + Back to “Understanding”

Methodology: Analyze • Interpretation and Evaluation – Statistical significance + mean, confidence intervals, correlation,

Methodology: Analyze • Interpretation and Evaluation – Statistical significance + mean, confidence intervals, correlation, goodness of fit – Does data support or reject hypothesis? – Explanation of other phenomena + Better code reduces page faults, improves performance

Graph: A Data Analysis Tool • • A picture is worth a thousand words

Graph: A Data Analysis Tool • • A picture is worth a thousand words Title, label axes (units!), legend

Dirty Little Secrets • Mini-experiments (no, “Pilot Tests”) • Hypotheses after the fact •

Dirty Little Secrets • Mini-experiments (no, “Pilot Tests”) • Hypotheses after the fact • • • – Running yields understanding Results here mean results there Controlled system still says meaningful things about the real world Observing a system will not change it

Computer Science Computer Science Computer Science a branchComputer Science Computer Science Computer Science a branch
Physical Experiments Computer Experiments Preliminaries Chapters 12 DesignPhysical Experiments Computer Experiments Preliminaries Chapters 12 Design
Experiments in Science at Primary School Experiments withExperiments in Science at Primary School Experiments with
Political Science 30 Political Inquiry Experiments Experiments ThePolitical Science 30 Political Inquiry Experiments Experiments The
Social Research Methods Experiments 1 Experiments l OneSocial Research Methods Experiments 1 Experiments l One
Wet Granulation Scaleup Experiments Design of Scaleup ExperimentsWet Granulation Scaleup Experiments Design of Scaleup Experiments
Design of Experiments 201819 Preface Experiments which processDesign of Experiments 201819 Preface Experiments which process
Experimental design Experiments vs observational studies Manipulative experimentsExperimental design Experiments vs observational studies Manipulative experiments