PSYC 3052 PSYC 2071 Advanced Social Psychology Judgment

PSYC 3052 / PSYC 2071 Advanced Social Psychology / Judgment and Decision Making T 4: Qualtrics survey best practices

Today’s Tutorial ● Designing a survey ○ ○ Question formatting/types Validation ■ Comprehension check ■ Content validation ○ ○ ○ Recoding values Answer randomisation Blocks Randomising conditions Preview ● Generating dummy data & exporting files

Process 1. Sign up for Qualtrics (if you haven’t already) 2. Read the guidelines: http: //mgto. org/qualtricsguide 3. Import the QSF file template in our replication folder & work on this 4. Design a Qualtrics survey featuring the most accurate replication of the experiment in the assigned article 5. Preview your survey yourself to check for error 6. Pre-test your survey on friends and family. Get feedback, revise accordingly and fix problems 7. Generate random responses from Qualtrics to create a dummy dataset 8. Conduct your analyses on the random dataset using R/JAMOVI 9. Revise Qualtrics and/or article analysis if needed

Important/Useful Links HKU Qualtrics: https: //hku. qualtrics. com/ Collaborative Qualtrics guide: http: //mgto. org/qualtricsguide Guide from Qualtrics website: https: //www. qualtrics. com/support/survey-platform/survey-module -overview/

Customised Qualtrics template ● Customised to cater MTurk (US data collection platform) ● With consent form, demographic information, etc. ● Please use the template for your Qualtrics survey (IMPORTANT!) ○ Template: https: //www. dropbox. com/sh/75 watpneuqxbzm 6/AAApa. Y 3 o_p. J 4 HNBJq 4 Zr. Co. XLa? dl=0

Creating a New Survey Include the author(s), year of article, phenomenon, and study number - Upload. qsf template from Dropbox E. g. Baron and Hershey (1988) Outcome Bias Study 1

Creating Questions *When writing answers/choices, follow the original - if it includes both the label and number, include both. If it includes either one but not both, follow the original.

Creating Questions - Formatting Needs to be consistent across all questions, especially when you have copied and pasted OR

Creating Questions - Question Types *Use matrix table when you have more than 10 items/questions that need to be rated - Choose repeat headers with the option “middle” - Options → Repeat Headers - Repeat Headers → Middle

Creating Questions - Validation Options *Remember to select this option for all questions, except for “Did you spot any errors? Anything missing or wrong? Something we should pay attention to in next runs? ”

Creating Questions - Validation Types For comprehension checks

Validation - Comprehension Check Example - Ziano et al. (2020) “Imagine that you are a government official of a small, developing country and you are asked to evaluate six government programs (Program A, B, C, D, E and F). All the six programs cost about the same and all are being considered for funding. Details of each programs are explained in the following. ” Please set force response for most/all questions

Validation - Comprehension Check Example - Ziano et al. (2020) Respondents need to choose ‘ 6’ to move on to the next page

Validation - Content Validation (Lyu et al. , 2019) For text entry only Restricting respondents to enter a number from 0 to 100

Creating Questions - Recoding Values By default, the first answer is assigned a value of 1, the second answer is assigned a value to 2 and so on Adding items, deleting one, and adding more items can mess up the coded values → Affects your data analysis if the values doesn’t tally with your scale number * Doesn’t correspond

Creating Questions - Answer Randomisation

Creating Questions Page break Icon meanings Validation Answer Randomisation Recode Values Separates questions onto the next page

Blocks For randomly or conditionally (e. g. control or experimental group) showing a block of questions One block = one condition * Remember to assign meaningful block names

Survey Flow - Randomiser For randomly presenting question blocks and other survey elements

Survey Flow - Randomiser Number of blocks you want to show your respondents Presents each block around an equal number of times across respondents

Preview Individual question Entire survey ● ● ● Rerun the survey at least per number of conditions Check if the comprehension check/attention check works Check if randomisation works as intended Check if validation works Check if the scale is shown clearly

Generating Test Responses (Dummy Data) To check how your dataset / reports will look (Note: custom validation will not be applied for these responses) To help you with designing a data analysis plan

Exporting Data Second resort - “use numeric values” *

Exporting Data From ‘More options’ Remember to select this *

Exporting the Survey -. qsf/. docx * For. qsf For. docx

References Feldman, G. , & Yeung, S. K. (2020). Collaborative Qualtrics guide. Work in Preparation Lyu, H. , Wang, M. , Feldman, G. , & Cheng, B. (2019). Miller & Ratner (1998): Replication and extension. Ziano*, I. , Xiao*, Q. , Yeung*, S. K. , *Narendra, G. I. , *Li, W. K. , … *Feldman^, G. (2020). Numbing or Sensitization? Replications and Extensions of Fetherstonaugh et al. (1997), “Insensitivity to the Value of Human Life”. https: //doi. org/10. 13140/RG. 2. 2. 10976. 71681