Psychophysical methods Lavanya Sharan January 26 th 2011

Psychophysical methods Lavanya Sharan January 26 th, 2011

Announcements • • • Check class website: http: //graphics. cmu. edu/courses/P 2 P/ Pick presentations slots for Feb 9 th and Feb 14 th. One CVG and one P slot person (not on same day) Email slot preference to sharan@cs. cmu. edu Have to meet instructors 2 x before presentation!

Overview • Basics of designing a perceptual experiment • Types of experiments • Analysis of experiments • IRB

Basics • Independent variable • This is what varies • Can have several levels (i. e. values) • An experiment may have many indep. vars. • Dependent variable • This is what is measured

An example • How well do people recognize objects (as compared to a favorite CV algorithm)? Images from the Visual Object Classes Challenge 2010 Independent variable? Dependent variable? s s Cla c i t r pa ! n o i t a ip

An example • How well do people recognize objects? Images from the Visual Object Classes Challenge 2010 Independent variable Object class (20 levels) Dependent variable Accuracy, Reaction time (RT)

Understanding your variables • Discrete or continuous? (e. g. , object class is discrete) • Measurement scale - Nominal: there is a difference (e. g. , car vs. bus) Ordinal: <, =, > make sense (e. g. , rank in VOC 2010 challenge) Interval: size and sign of difference makes sense (e. g. , Fahrenheit scale) Ratio: all the above and having a 0 makes sense (e. g. , percentage correct)

Confounds • Ideally, stimuli only differ in the levels of the independent variable. Images from the Visual Object Classes Challenge 2010 Confounding variables Color, shape, size of object, contrast of images, background of objects, location in image, lighting, etc.

Confounds • Ideally, stimuli only differ in the levels of the independent variable. Images from the Visual Object Classes Challenge 2010 In practice, impossible to eliminate confounding variables. A well-designed study minimizes as many confounds as possible.

Dealing with confounds • Control for confounding factors by: • • • Removing confounding variables in stimuli Balancing the presence of confounding variables by adding control conditions* Separating participants into groups and comparing their responses i. e. , control group Condition = any manipulation of independent variables.

Controls Images from the Visual Object Classes Challenge 2010 Hypothesis: Size of object in image is enough to distinguish the 20 object classes. Confounding (or control) variable = Visual size of objects Option 1: Resize and re-crop all the images, never allow size to be an issue.

Controls Images from the Visual Object Classes Challenge 2010 Hypothesis: Size of object in image is enough to distinguish the 20 object classes. Confounding (or control) variable = Visual size of objects Option II: Create second set of images with resized objects, test if size is an issue. Question: Control condition (same subjects) or control group (different subjects)?

Experimental Designs Between-subjects: Different subjects in different conditions. Minimize learning effects, more subjects Within-subjects: Same subjects in all conditions. More power, fewer subjects Mixed design: Some conditions withinsubjects and some between-subjects.

Balanced designs Even after adding control conditions and/or groups, we have to worry about order effects Seeing bikes before buses Seeing dogs before buses

Balanced designs Even after adding control conditions and/or groups, we have to worry about order effects Seeing bikes before buses Seeing dogs before buses Does it matter? Priming can occur.

Balanced designs Even after adding control conditions and/or groups, we have to worry about order effects Solution: Seeing bikes before buses Seeing dogs before buses Randomization Counterbalancing

Balanced designs Use randomization when lots of subjects or each subject sees the same conditions many times. Use counterbalancing otherwise. E. g. , 3 conditions (Dog, Bike, Bus) Order 1 Order 2 Order 3 S 1, 4, 7, 10. . . S 2, 5, 8, 11. . . S 3, 6, 9, 12. . . Slide content: Aude Oliva, MIT OCW

Overview • Basics of designing a perceptual experiment • Types of experiments • Analysis of experiments • IRB

Thresholds Absolute: At which visual size can you detect that there is a bike? Difference: How much detail is needed to recreate the original shape?

Measuring Thresholds Method of Constant Stimuli: Present stimuli at several levels, some below and some above threshold, measure proportion detected. Method of Limits: Start at one level, if detected decrease else increase level. Converge to an estimate of threshold. Method of Adjustment: Let subjects adjust the level until they can just detect the stimulus.

Method of Constant Stimuli + Easy to conduct and analyze - Need to have an idea of threshold beforehand - Takes longer, more trials Psychometric function Image source: http: //www. ncbi. nlm. nih. gov/books/NBK 11513/ Slide content: Lynee Werner, University of Washington

Method of Limits + Fewer trials + Don’t need to estimate threshold beforehand - Noise prone Staircase procedure Image source: http: //www. ncbi. nlm. nih. gov/books/NBK 11513/ Slide content: Lynee Werner, University of Washington

Method of Adjustment + Intuitive for subject - Can be unreliable Image source: http: //psychology. wikia. com/wiki/File: Method_of_Adjustment. png Slide content: Lynee Werner, University of Washington

Overview • Basics of designing a perceptual experiment • Types of experiments • Analysis of experiments • IRB

Back to detection What about response bias? Subjects might be more prone to saying yes (or no). Image source: http: //www. walker. co. uk/walkerdam/getimage. aspx? id=9781406314403 -1&size=webuse Slide content: Lynee Werner, University of Washington

Signal detection theory Stimulus present Stimulus absent ‣ ‣ ‣ Subject said yes Subject said no Hit Miss False Alarm Correct Rejection d’ is computed from Hits and False Alarms Measure of sensitivity Bias-free Slide content: Lynee Werner, University of Washington

2 -AFC design ‣ Instead of asking for yes/no response on each trial, force subject to choose from two options ‣ Another way of removing response bias (interval bias can remain) Image source: http: //www. ratracetrap. com/wp-content/uploads/2009/09/Fork-in-the-road-300 x 237. png Slide content: Lynee Werner, University of Washington

How do you know subjects aren’t guessing? Need to calculate chance performance for every task Subjects might have been sleeping, distracted, perverse, or simply unable to do your task because it is humanly impossible. (Bad subjects. ) Image source: http: //academic. kellogg. edu/mckayg/buad 112/web/pres/coin%20 flip. jpg

Is performance better than chance? Statistics to the rescue. VOC 2010 Challenge example: 20 object classes implies chance = 1/20 = 5% Between-subjects design, Group ‘No Size Control’ (Performance = 80%, N=10) Group ‘Size Control’ (Performance = 75%, N=10) ✓ Use independent one-sample t-tests to compare performance in both groups to chance ✓ Choose significance threshold (usually p = 0. 05) ✓ Divide significance threshold by number of tests (Bonferroni correction), here 0. 05/2 = 0. 025 ✓ Quote the t-statistic and p-value if less than corrected threshold as showing statistical significance.

Is performance better in one condition than another? Again, statistics to the rescue. VOC 2010 Challenge example: 20 object classes implies chance = 1/20 = 5% Between-subjects design, Group ‘No Size Control’ (Performance = 80%, N=10) Group ‘Size Control’ (Performance = 75%, N=10) ✓ Use independent two-sample t-test to compare performance in two groups to each other ✓ Choose significance threshold (usually p = 0. 05, if combining with two previous tests then 0. 05/3 = 0. 0167) ✓ Quote the t-statistic and p-value if less than threshold as showing statistical significance.

Choosing the right tests is VERY important Otherwise you don’t know if you are measuring noise or a real effect. For example, for a within-subjects design, you would use a paired samples t-test. Get hold of a good statistics book and package and understand precisely what you are doing. I recommend using SPSS and material from: http: //www. statisticshell. com

Overview • Basics of designing a perceptual experiment • Types of experiments • Analysis of experiments • IRB

IRB and other legalities You are dealing with human subjects, this means you need to be very ethical and careful. CMU has a Regulatory Compliance Office: http: //www. cmu. edu/osp/regulatory-compliance/humansubjects. html You are encouraged to take the online human subjects training: https: //www. citiprogram. org/Default. asp? Image source: http: //www. icts. uiowa. edu/drupal/sites/all/themes/icts/custom/images/news/warning. jpg

IRB and other legalities For this class: If you will not use any perceptual data you gather ever again, you don’t need to write an IRB protocol and get approval. If there is even a tiniest chance, the perceptual data you gather will be show up anywhere, you NEED to write an IRB protocol and get it approved in time. Come talk to me about this. Image source: http: //www. icts. uiowa. edu/drupal/sites/all/themes/icts/custom/images/news/warning. jpg

Summary • • • Figure out your independent and dependent variables. Think of all possible confounds. Control for confounds, balance your designs. Get IRB to run study. Analyze data using standard statistical procedures. Don’t do all this a day before your deadline!