Preference learning Max Yi Ren MAE 540 Advanced

Preference learning Max Yi Ren MAE 540 Advanced Product Design 1

Understanding consumer preference is important for engineers! 2

How will electric vehicle designs affect the market? Kang et al. , A Framework for Quantitative Analysis of Government Policy Influence on Electric Vehicle Market, 2015 3

US consumer preference 4

Profit-driven Collaborative Policy design for energy security and emission reduction Ann Arbor 5

US consumer preference 6

Profit-driven Collaborative Policy design for energy security and emission reduction Beijing Figure from: Kang et al. , A Framework for Quantitative Analysis of Government Policy Influence on Electric Vehicle Market, 2015 7

Motivation The design process should have some form of user input How will the survey help you in the design process? 8

Today’s Outline • What is the goal of the survey? • What are the best questions to ask to achieve those goals? • Common problems to avoid • Choice-Based Conjoint (CBC) How do we prioritize our design objectives? 9

Goal: Understand consumers through data collection and analysis • • • Attitudes and Opinions/Beliefs Awareness and Knowledge Purchase Intentions Usage Behavior (what people actually do) Demographics 10

Types of Questions • Open Response • Closed Response – Itemized category – Comparative – Rank Order – Likert Scale – Semantic Differential – Constant Sum 11

Open Response Example: Why do you buy cellphone case? _________ 12

Closed Response Example: How often do you have problem with low speaker volume in a class setting? a) Very often b) Sometimes c) Rarely d) Never 13

Itemized Category What kind of food do you consume on a daily basis? a) Milk b) Egg c) Kool-aid 14

Comparative Compare to regular headphones, how much do you like bluetooth ones? _____. (a) Better (b) Worse (c) The same 15

Rank Order Please rank the following items in terms of importance to your daily life: (1 = I feel most stressed if I lose it, 4 = I will live) ___ Wallet ___ Cellphone ___ Laptop ___ Lunch box 16

Likert Scale Rate the following statement on a scale of 1 to 7: 1 = strongly agree, 7 = strongly disagree It would be great if I can track how much water I use and when I use them 1 2 3 4 5 6 7 17

Semantic Differential Please circle the number on the scale that best describes the difficulty in keeping your bike safe on campus: For sure will get stole 1 2 3 4 5 Don’t need a lock 18

Constant Sum How important are each of these when you choose a baby protection device? Please assign 100 points among the following: 15 ____ Easiness in installation 35 ____ Maintenance cost 15 False positive rate ____ 5 False negative rate ____ 30 Price ____ 19

Types of Questions • Open Response • Closed Response – Itemized category – Comparative – Rank Order – Likert Scale – Semantic Differential – Constant Sum 20

Taken from Fink, 1995 21

Common Problems to Avoid • • Misunderstanding/misinterpretation of question Specificity (too vague, too precise) Bias Inappropriate questions – Audience background/knowledge – Context (especially when questions may be out of order) – Offensiveness – Getting the right audience 22

Are the words understandable? Complex word Exhausted Candid Priority Leisure Employment Supervision Courageous Assistance Preserve Virtually Rectify Simpler substitute Tired Honest Most important Free time Work Care Brave Help Protect Nearly Correct 23

Vague Questions Example: • Do you work out regularly? – Yes – No Improved: • Do you work out at least three times a week? – Yes – No Even More Specific: • Physical activity can be defined as ____. Do you engage in physical activity at least 3 times a week? – Yes – No 24

What level of specificity is needed? Example: • How many times did you use a treadmill last month? ____ Improved: • Over the last month, about how often did you use a treadmill? – Never – 1 to 5 times – 6 to 10 times – More than 10 times 25

Bias Problems A question is biased if it influences people to respond in a manner that does not accurately reflect their position on the issue under investigation (Dillman, 1978) 26

Bias Problems • • • Order Priming Education/Information Social Desirability Wording/Connotations 27

Bias - Order 1. How smart is the typical professional designer? smart 1 2 3 4 very smart 2. How smart are you? smart 1 2 28

Bias - Priming 1. How are you feeling? 1. How is the weather? 2. How are you feeling? 29

Bias – Education/Information Read about our new product then answer the following questions… 30

Bias – Social Desirability • conforms to dominant belief patterns • people often respond to make themselves sound or seem “better” How often do you tell your significant other that you love them? How much do you recycle? 31

Bias Solutions • • Randomization Different versions Careful wording Think about potential bias when preparing surveys! 32

Wording Problems in Surveys (Dillman, 1978) 1. Will the words be uniformly understood? 2. Does the question contain abbreviations or unconventional phrases? 3. Is the question too vague? 4. Is the question too precise? 5. Is the question biased? 6. Is the question objectionable? 7. Is the question too demanding? 8. Is it a double question? 9. Does the question have a double negative? 10. Are the answers mutually exclusive? 11. Does the question assume too much about what the respondents know? 12. Is the question technically accurate? 13. Is an appropriate time referent provided? 14. Can the question be understood when taken out of order or context? 15. Can responses be compared to existing information? 33

Today’s Outline • What is the goal of the survey? • What are the best questions to ask to achieve those goals? • Common problems to avoid • Choice-Based Conjoint (CBC) How do we prioritize our design objectives? 34

Considerjointly Analysis § Systematic way to match product design with the needs and wants of customers, especially in the early stages of the New Product Development process • To understand how consumers make trade-offs • To help uncover customers’ most important product attributes Measuring Utility Design products to maximize market share 35

Utility model Random utility model Utility of Product j Error Deterministic component Product attributes Part worths In simple case: Product j vs. Product k Utility of j Utility of k Multinomial Logit Model Probability of choosing j = Probability that j has more utility than k 36

Utility model Product Utility Attributes Levels Partworth Vehicle Price $15 K Acceleration (0 60 mph) MPG $20 K $25 K 25 30 35 6 sec 8 sec 10 sec Conjoint Analysis where x is binary variables (x = 0 or 1) 37

Utility model Utility of vehicle j Product Attributes Levels Partworth Vehicle Price $15 K $20 K Acceleration (0 60 mph) MPG $25 K 25 30 35 6 sec 8 sec 10 sec $25 K 35 MPG 8 sec <Vehicle j> 38

Utility model Utility of vehicle j Product Attributes Levels Partworth Vehicle Price $15 K $20 K Acceleration (0 60 mph) MPG $25 K 25 30 35 6 sec 8 sec 10 sec $25 K 35 MPG 8 sec <Vehicle j> 39

Utility Model Compute relative importance for each product attribute Attribute Price Attribute MPG Attribute 0 to 60 Level Part-worth $15 K $20 K $25 K 6 3 -9 Level Part-worth 35 30 25 2 1 -3 Level Part-worth 6 sec 8 sec 10 sec 0. 5 0 -0. 5 40

Utility model Estimate market demand <Vehicle j> <Vehicle k> VS Potential Market Size 41

Conjoint survey design 1. Design the conjoint survey 1. 1: Select attributes relevant to the product or service category 1. 2: Select levels for each attribute Attribute Price MPG Acceleration (0 60 mph) $15 K 25 Level $20 K 30 $25 K 35 6 sec 8 sec 10 sec 1. 3: Develop “product bundles” to be evaluated ($15 K, 25 mpg, 10 sec) ($20 K, 30 mpg, 8 sec) Design of experiment (DOE) ($25 K, 35 mpg, 6 sec) … 42

Conjoint survey design Full factorial design Design of Experiment: D-optimal design det(XTX) = 0 det(XTX) = 2 det(XTX) = 3 X 1 X 2 43

Types of Conjoint survey <Metric/rating conjoint> <Ranking conjoint> Please rank all the cars Rank Price: $15 K MPG: 25 Acceleration: 10 sec Price: $30 K MPG: 20 Acceleration: 6 sec Price: $25 K MPG: 35 Acceleration: 8 sec How likely would you be to buy this car? 3 Price: $25 K MPG: 35 Acceleration: 8 sec 5 2 Definitely Would Not Probably Would Not Might or Might Not Probably Would Definitely Would … … 44

Types of Conjoint survey <Pair-wise comparison conjoint> <Choice-based conjoint> Which car would you prefer? Price: $25 K MPG: 35 Acceleration: 8 sec Strongly Prefer Left Somewhat Prefer Left Price: $15 K MPG: 25 Acceleration: 10 sec No Preference Somewhat Prefer Right Strongly Prefer Right If you were shopping for a car and these were your only option, which would you choose? $15 K 25 MPG 10 sec $30 K 20 MPG 6 sec $25 K 35 MPG 8 sec None: I wouldn’t choose any of these 45

Conjoint survey design 2. Conduct the conjoint survey and obtain data Evaluate bundles using either ranking, rating, or choice Choice based conjoint (CBC) If these were your only options, which would you choose? $15 K MPG 25 Acceleration 6 sec $25 K MPG 30 Acceleration 7 sec $15 K MPG 35 Acceleration 8 sec NONE: I wouldn't choose any of these. √ … 46

Estimate betas 3. Build the utility model Use MLE(Maximum likelihood estimation) Observed choice data 1: j is chosen 0: j is not chosen Choice data by survey Train model Estimate betas (β) 47

Contents 1. Why Marketing Modeling? 2. Conjoint Analysis 2. 1 Utility Model 2. 2 Conjoint Survey 2. 3. Spline Interpolation 3. Advanced Conjoint Analysis 48

Part-worth and Spline curve Price MPG Acceleration Level 15 K 20 K 30 K Part-worth (Beta) 0. 64 -0. 03 -0. 61 Level 25 30 35 Part-worth (Beta) -0. 67 -0. 07 0. 74 Level 6 8 10 Part-worth (Beta) 0. 74 0. 57 -1. 32 Spline Price MPG Acceleration Part-worth 0. 64 -0. 03 -0. 61 49

Compute interpolated values Product A Price MPG Acceleration Value Part-worth $24. 6 -0. 57 34 0. 56 7. 0 0. 87 Product B Price MPG Acceleration Value Part-worth $23. 9 -0. 49 30 -0. 07 6. 2 0. 80 50

Mixed Logit Model Preferences among consumers are usually heterogeneous, i. e. everyone likes something different. How do we capture these different preferences with limited number of responses from everyone? Hierarchical Bayes Model models the distribution of different preferences, thus leverages all responses for the estimation 51

Mixed Logit Model 52

Mixed Logit Model Instead, draw from the joint distribution: 53

Mixed Logit Model Metropolis-Hastings: A Markov chain Monte Carlo method to sample a distribution 54

Mixed Logit Model Summary: Mixed Logit is useful when heterogeneous preference exists For regular Discrete Choice surveys (~5 attributes, 3 -4 levels), a minimum of 200 respondents are needed, each with 10 -15 choice questions. 55

Overview of the class Design variables Design attributes Design Objective(s) Engineering models Preference model 56

Other Forms of Preference Learning Ratings and reviews from ecommerce sites 57

Other Forms of Preference Learning Ratings and reviews from ecommerce sites Challenges: Polarized ratings Unreliable comments 58

Other Forms of Preference Learning Io. T for tracking product usage Embed pressure sensors in shoes to understand which part of the shoe people care about the most during evaluation 59

Other Forms of Preference Learning Io. T for tracking product usage Eye tracking during a discrete choice survey to see how people compare two objects 60

Adaptive Questionnaire What is the right question to ask, given all answers you received? More specifically, what is the strategy to minimize the expected number of questions to achieve the goal of (understanding preference, guess what others are thinking, etc. )? 61

Adaptive Questionnaire 62

Adaptive Questionnaire Adaptive questionnaire (active learning) is an extension from traditional DOE. It can be applied to interactive surveys where data acquisition is expensive, or scenarios where data is abundant but importance of data is unclear. Example engineering application: In designing new materials, one may need to conduct a lot of experiments. What experiment shall we conduct given our current knowledge? At any point during the investigation, how do you evaluate the value of future experiments? 63