Design and Analysis of Experiments for Research Projects

Types of Experiments • Simulation – Numerical, event-driven, flow-level, etc. • Emulation • (Testbed)

Deciding factors • Nature of the project: – E. g. , Energy-efficient activity tracking

An example of hybrid approaches Abstracted Real Sensing Physical Environments Signal Processing Communication Actuation

Deciding factor • Scale, controllability, repeatability and fidelity – Simulation wins on the first

When do you design the experiments • From the onset of the project! –

Analyzing the results • Know what to expect – Don’t bring completely out-of-the-line results

An Example GNU Radio Signal Generator Concrete Channel 440 m. Vpp USRP 2 4.

Presenting your results • Statistics – Please give me error bars or confidence intervals

Slides: 9

Download presentation

Design and Analysis of Experiments for Research Projects Rong Zheng

Types of Experiments • Simulation – Numerical, event-driven, flow-level, etc. • Emulation • (Testbed) experiments Which one to pick?

Deciding factors • Nature of the project: – E. g. , Energy-efficient activity tracking using mobile phones – E. g, Influence maximization on online social networks – E. g. , structure health monitoring

An example of hybrid approaches Abstracted Real Sensing Physical Environments Signal Processing Communication Actuation Sensor node Our solution

Deciding factor • Scale, controllability, repeatability and fidelity – Simulation wins on the first 3 – Test experiments win on fidelity (but not always) – Emulation comes in between However, if you are doing systems research, the expectation is to have testbed experiments complimented by simulations

When do you design the experiments • From the onset of the project! – What datasets? – What hardware/software? – Which user group? – What metrics? – What to compare against?

Analyzing the results • Know what to expect – Don’t bring completely out-of-the-line results to your advisor • Understand what you didn’t expect – Verify the implementation is correct • Unit test (isolation), sanity check – Find an explanation • In CS, there is always an explanation to everything • When you are done with the experiments, you are still far from being done • Unexpected results can be quite useful in improving your solutions

An Example GNU Radio Signal Generator Concrete Channel 440 m. Vpp USRP 2 4. 161 Vpp Pre - Amp 238. 05 Vpp Trans Amp 1. 081 Vpp Oscilloscope Gain = -22. 23 d. B USRP Daughterboard 2. 5 m. Vpp File Sink

Presenting your results • Statistics – Please give me error bars or confidence intervals – Is the observation really statistically significant? • Obama vs Romeney Gullop poll 47% vs 49% – Both sides of the story: false positive and false negative • • Experimental settings please! 3 D graphs should be banned Up and down curves a bad sign Colors on black-white printout? Fonts? – Embedded font issues • Tell what you observe and provide insight gained – Don’t just run out of stream when you reach the evaluation section of your paper