Modeling Online Browsing with Clickstream Data Using Path

Modeling Online Browsing with Clickstream Data Using Path Analysis Alan Montgomery Associate Professor Carnegie Mellon University Graduate School of Industrial Administration with Shibo Li, Kannan Srinivasan, John Liechty e-mail: alan. montgomery@cmu. edu web: http: //www. andrew. cmu. edu/user/alm 3 QME Conference, 24 October 2003 © 2003 by Alan Montgomery, All rights reserved.

Outline • • • Problem Data Model Predictive Results Model Properties and Applications Conclusions 2

Problem

Path Analysis Questions • Can we use the navigation path to predict future movement or subsequent events? – Is this user going to purchase? – Will he leave the web site within the next 5 clicks? • Does a path reflect a user’s goals? • How does web design and marketing mix variables effect browsing behavior and subsequent decisions? 4

Clickstream Example Information rules Clickstream Data generated from this Path: http: //www. barnesandnoble. com/index. asp? userid=xxx&vcqty=1 http: //search. barnesandnoble. com/booksearch/results. asp? WRD=information+rul es&userid=xxx http: //search. barnesandnoble. com/booksearch/isbn. Inquiry. asp? userid=xxx&isbn= 087584863 X http: //cart. barnesandnoble. com/shop/cart. asp? userid=xxx&vcqty=1 5

Purchase Conversion Application Home Categor y Product Shoppi ng Cart What is the chance of this user making a purchase during this session? 1 st viewing = 7% 2 nd viewing = 14% 3 rd viewing = 20% 4 th viewing = 60% 6

Contrast with Scanner Data • Scanner data does not include pathing information, and subsequently we lack tools for analyzing path information. • Fundamentally, previous datasets have lacked a temporal element of shopping that potentially could be valuable. • Some other examples of pathing information: – Gap shopper in the store (Underhill 1998) – Monitoring cart movements in the supermarket (Vide. Ocart) – PDAs/Cell phones with GPS tracking enabled 7

Problems with Clickstream Data • Huge – Terabytes of data • Collection Issues – Server versus client – Privacy and identifiability • Unstructured – User’s goals are not observed (Purchase? Search? Surfing? ) – Textual and graphical data • Conflicting Decision Problems – Redesign web site – Understand usage and implication on consumer behavior 8

Data

Data Source • Comscore Media Metrix Panelists from April 2002 who visit barnesandnoble. com – – 1, 160 users 1, 659 Sessions, 8. 75 Viewings per session 14, 512 viewings 114 purchases 7% conversion rate • Collect HTML content using Perl script – Parse the pages for content information, like the presence of a banner ad, price, … Could collect data directly from bn. com servers 10

Categorizing Pages Home Category Product Information Account Shop. Cart These categories derived from HCI literature based upon a task analysis of what users want to do at e- 11

Category Descriptions Abbreviatio n Category Name H Home page, common starting page for B&N visitors 6% A Account User account pages that allow a user to sign in, change address, and review order status 5% C Category Pages that have list of items (links like cooking, fiction, etc. from the home page) or results of a book search 32% P Product Pages with detailed product information, item description, price information, availability, and reviews 17% I Informatio n Pages with shipping, order status, and popup advertisements such as “Free Shipping” 23% S Shopping Cart Description Viewi ng Prob. Pages related to the shopping cart, such as 5% reviewing the cart, deleting items, entering 12 purchase information Categorization scheme for B&N web pages for 9, 180 unique pages requests and 14, 512 v

Transformed Clickstream Time URL Category Abbr. 1 8: 36: 11 pm /promo/coupon/popups/fs_usa_popup. asp? userid=xxx Information I 2 8: 36: 29 pm /booksearch/results. asp? wrd=70%2 d 215&userid=xxx Category C 3 8: 36: 48 pm /booksearch/results. asp? userid=xxx&mscssid=yyy&wrd=70%2 d 215& Category C 4 8: 37: 14 pm /booksearch/isbninquiry. asp? userid=xxx&mscssid=yyy&isbn=007213 Product P 5 8: 38: 10 pm /booksearch/results. asp? userid=xxx&mscssid=yyy&wrd=70%2 d 215& Category C 6 8: 44: 32 pm /textbooks/booksearch/isbninquiry. asp? userid=xxx&mscssid=yyy&isb Product P 7 8: 55: 12 pm /promo/coupon/popups/fs_usa_popup. asp? userid=xxx Information I 8 8: 55: 24 pm /booksearch/results. asp? wrd=70%2 d 215&userid=xxx Category C 9 8: 55: 36 pm /booksearch/results. asp? userid=xxx&mscssid=yyy&wrd=70%2 d 215& Category C 10 8: 56: 37 pm /shop/signin. asp? userid=xxx&mscssid=yyy Account A 11 8: 58: 16 pm /booksearch/results. asp? userid=xxx&mscssid=yyy&wrd=70%2 d 215& Category C 12 8: 58: 40 pm /booksearch/isbninquiry. asp? userid=xxx&mscssid=yyy&isbn=007222 Product P 13 opr=a&sort=p 4445 opr=a&sort=p n=0619034971 opr=a&sort=p 4983

Sample User Sessions Buyers Browsers User Path 1 2 3 4 5 ICCCCCPCCPCCCCCCCCCCCCCCCE IHHE IE IHICPPPCE IHHIIICIIE 6 7 8 9 10 HIAAAAIAIIIICICICCIPPIPPIPIICCSIIIPPPPPIPIPSISISISSSOII IIIHE HCCPPPCCPCCCCCPSCSCSPCCPCPCCCCCCCSAAAAASS OIIIIISASCCCE IIICICPCPPPCPCICICPCCCPCPPPIPSIIAASSSIIIIOIIE IISIASSSIOIE IPPPPSASSSSSOIAAAHCCPCCCCCCE 14

User Demographics Variable Age Mean Std Dev 45. 89 14. 62 2326. 4 8 1331. 6 8 Male . 47 . 50 White . 77 . 42 Children under 18 in the house . 40 . 49 Married . 29 . 45 Some college education . 82 . 39 High Income (>$50, 000) . 32 . 47 Medium Income ($25, 000 - . 35 . 48 Age 2 (square of Age) 2 Table 1. Demographic characteristics of 1, 160 panelists, all means except age and 15 age give

Page Covariates Variable Mean Std. Dev Presence of price information on page (Proportion) . 45 . 50 Promotional image present (Proportion) . 83 . 37 Presence of banner advertisement (Proportion) . 03 . 16 Number of links to a home page 2. 4 1. 0 10. 1 18. 1 2. 0 1. 1 28. 8 33. 9 Whether made a B&N purchase during last session . 03 . 18 Time Since Last Viewing (Seconds) 7. 2 66. 3 Whether the Visit is on Weekend (Proportion) . 28 . 45 Cum. no. of viewings at B&N during session (visit depth) 8. 8 16. 4 Cum. no. of viewings at other sites during session 44. 3 Number of links to a product page Number of links to an account page Number of links to an information page 84. 616 Cum. no. of viewings at other. Statistics bookstores during 4. 3 pages requested. 17. 3 Table 4. Descriptive for the 9, 180 unique B&N

Peculiarities of the Dataset • Users do not have to follow the hyperlinks the site presents, but frequently use browser controls (back, forward, history) • There are no forced transitions – It is possible to move from any page on the web site to any other page. One exception is the order page, which requres that the previous click be a shopping cart. • We model choice of the category associated with a page view, not the choice of which hyper link to follow. 17

Example: Path Problems Product 1 Web Design: Category 1 Product 2 Home Category 2 Potential Unexpected Sequences: Product 3 C 1 P 1 Google directs user to interior (doesn’t start at home) C 2 C 1 P 3 P 2 Two windows open, sessions intermixed H C 1 P 1 H Home is not linked, accessed from history P 3 HUser receives email with product link, then keys in home H P 3 Content page is cached by user’s browser (not observed) H H User refreshes home page/Intervening page not observed 18

Page Transitions Category of Previous Viewing Category Home Acct Home . 23 . 01 . 10 . 02 0 . 16 Account . 01 . 69 . 01 . 02 . 15 0 . 01 . 17 . 02 . 60 . 31 . 15 . 05 0 . 16 . 01 0 . 20 . 43 . 10 . 05 . 25 . 06 . 08 . 12 . 46 . 15 . 87 . 61 . 01 . 16 . 01 . 03 . 02 . 45 . 13 . 01 0 0 0 . 10 0 0 Exit . 32 . 06 . 09 . 14 . 02 0 0 Marginal . 06 . 05 . 32 . 17 . 23 . 05 . 01 . 11 Initial Prob. . 16 . 02 . 16 . 06 . 60 . 01 0 0 Category of Current Product Viewing Information Shop. Cart Order Cat Prod. Inform. Cart Order Exit 19 Table 6. Sample transition matrix for categories of viewings. (Notice that the columns sum to one, and there a total of 14, 51

Model

Choice Model of Browsing Switching: Hidden Markov Process Page and User Characteristics Category Latent Utilities Memory/Trends: Autoregressive Choices 21

Dynamic Multinomial Probit Model with A Mixture Process Observational Equation: User (i), Session (q), Viewing (t), Category (c) and indicates if a category is available. Model: s<iqt> denotes the state for this observation. 22

Modeling State Transitions • Propose zero- and first-order markov models to capture state transitions • Empirically estimate how many states: 1, 2, or 3 • Permit state transitions to occur at several levels: Page: State may shift after every viewing Session: States may shift after each session, but not within a session User: States may differ across users, but does not change across user’s viewings. This yields 3 x 2 x 3 = 18 model variations 23

Modeling State Transitions with Zero-Order Markov Process Users switch back and forth between states following an zero-order markov process: Implies that there is no memory in the process. If state transitions occur at a user level this is a latentclass model. 24

Modeling State Transitions with First-Order Markov Process There is a first-order Markov process governed by a hidden, continuous time Markov Chain, Diqt, which indicates the state. The time between transitions (wiqt) is exponentially distributed: Given that a transition has occurred: State probability of initial session viewing: 25

Relationship between Continuous and Discrete Time 1 1 2 2 2 1 1 Time If we assume that transitions only occur on discrete time intervals, then our continuous time model becomes the usual First-Order Markov Model: Notice that the waiting-time parameter of the exponential process populates the probabilities of the switching matrix. 26

Why include a Markov Model? Question: Isn’t there a redundancy between the VAR process and the markov model? Answer: Markov process can capture discrete jumps in relationship. Consider the conditional mean: Hamilton (1989) employed a markov switchingautoregression model to model US real GNP. 27

Model Elements • Dependent Variable is a Categorical Time Series – Choice modeled using Multinomial Probit (MNP Model) • Two Types of Time Series Behavior: – Continuous trends: VAR(1) Process – Abrupt changes: Mixture Process with a Hidden Markov Model • Covariates – Web design and marketing mix – Browsing behavior – Demographics • Consumer Heterogeneity – Hierarchical Bayesian Approach 28

Special Cases of our Model • Multinomial Probit Model (1 state) • Latent Mixture Model (Multiple states, zero-order markov) • Zero-Order Markov Process (only intercepts) • Approximates First-Order Markov Process with VAR(1) 29

Predictive Results

Comparison Model Specifications No memory Model Log Marginal In-Sample Density Hit Rate Out-of. Samp le Zero-Order Markov Model (1 State) -20410. 4 20. 5% 12. 6% Latent Class Model (2 States) -17673. 9 44. 3% 40. 2% Independent Probit -19086. 4 33. 2% 30. 4% Multinomial Probit with Intercepts Only -19335. 9 29. 4% 23. 1% First-Order Markov Model (1 State) -16444. 5 56. 1% 51. 6% -13768. 4 71. 1% 64. 4% 2 States, Zero Markov, User Transition -9124. 2 70. 0% 64. 1% 2 States, Zero Markov, Session Transition -9051. 0 77. 9% 70. 5% Best Dynamic Multinomial Probit: VAR only, no covariates 31

Findings • Dynamic multinomial probit model with two-state mixture process that follows a first-order markov process is best. – In sample hit rate of almost 90% – All criterion (in-sample, out-of-sample, posterior marginal density) support two-state process • State transitions appear to occur at a page level on average every 3 pages • Memory-less models: MNP, latent class, and zero-order markov model are the worst • Covariates are helpful, VAR model has hit rate of 71% • Browsing is better approximated by VAR process than a firstorder Markov process, which has a hit rate of 59% 32

Model Properties

Model Tasks • How well can we forecast a sequence? • What is the marginal distribution of the runs length between viewings? • Can we predict purchase conversion with limited information? These tasks illustrate both the adequacy of the model as well as potential applications. 34

How well can we predict a two-step sequence? Probability of Predicting Sequence Model CC PP CP PC Zero-Order Markov (1 State) 12. 2 % 3. 2% 5. 7% 5. 1% Multinomial Probit 15. 6 % 9. 1% 6. 2% 5. 4% Latent Class (2 States) 17. 1 % 11. 3 7. 2% % 5. 8% First-Order Markov (1 State) 31. 5 % 10. 8 8. 4% % 7. 5% VAR with Intercept 53. 9 % 21. 7 % 32. 4 % 33. 1 % Dynamic Probit (1 State) 63. 7 % 42. 9 % 54. 6 % 43. 1 % Best 35

How well can we predict the timing between viewings (C*C)? ote: the ‘*’ denotes any sequence of pages other than exit or category 36

Forecasting Purchase Conversion We wish to predict purchase conversion: “Is this user going to buy sometime in the remainder of their session? ” We currently observe the sequence: H I A Possible paths H H H. . . H I A S O I A ? ? ? S O or H I A * O Probability of purchase conversion is the chance that any of these sequences will occur. (Note: ? Does not include E since we are only interested in purchase in this session. ) 37

How well can we predict purchase? 38

Predicting Purchase Conversion 39

Predicted Purchase Conversions Sample Estimati on Holdout Session Type Forecast Origin/Number of viewings Number during session of 1 2 3 4 5 Sessions 6 Purchase 83 13. 3 16. 3 23. 4 30. 9 34. 4 41. 5 % % % (0. 48) (0. 52) (0. 60) (0. 65) (0. 67) (0. 69) No Purchase 1129 6. 1% 5. 4% 4. 6% 3. 7% 3. 4% 3. 1% (0. 33) (0. 32) (0. 30) (0. 27) (0. 26) (0. 25) All 1212 6. 6% 6. 1% 5. 9% 5. 6% 5. 5% 5. 7% (0. 35) (0. 34) (0. 33) (0. 32) (0. 33) Purchase 31 10. 4 12. 8 15. 2 18. 0 19. 1 21. 2 % % % (0. 97) (1. 06) (1. 14) (1. 21) (1. 24) (1. 29) No Purchase 416 6. 9% 5. 5% 5. 1% 4. 2% 3. 5% 3. 2% (0. 80) (0. 72) (0. 70) (0. 63) (0. 58) (0. 56) 447 7. 2% 5. 9% 5. 8% 5. 1% 4. 6% 4. 4%40 (0. 82) (0. 75) (0. 74) (0. 70) (0. 66) (0. 65) All

Conclusions

Summary • Propose new methodology to analyze page-level viewings – Structured path analysis as a categorical choice problem and applied dynamic multinomial probit model – Incorporate memory using vector autoregressive models and mixture models with hidden markov process • Good dynamic prediction of purchase conversion – Substantively we find that purchasers can be identified with a 40% probability within 6 clicks. – Users appear to be in two discrete states (deliberation oriented and purchase oriented) but can switch back and forth between these states from page to page. – Measurable impact of marketing mix and hypertext links • Conjecture: User’s goals are reflected in the path which is why we are able to predict so well. 42

Limitations and future directions • One retailer for an one-month period – Information sites may be quite different than shopping sites. • Timing information may be quite helpful. • Categorization results in valuable content information. – Predict choices at a link level instead of category • Better models of user behavior – Can we directly measure a user’s goals? • Can we adaptively design a web site using this model? – Consistency of user interface – Design influences goals 43

Appendix

Background Literature • Online Browsing and Purchase Behavior • Path Analysis in Computer Science • Human Computer Interaction • Consumer Search Behavior – Browsing: Bucklin and Sismeiro (2003) – Task completion: Sismeiro and Bucklin (2003) – Purchase conversion: Moe and Fader (2002), Park and Fader (2002), Moe et al (2002) – Common paths to improve designs: Yan et al (1996); Chen et al (1998); Spilopoulou et al (1999), Fu et al (1999) – Caching: Bestavros (1996), Zukerman et al (1999) – Visulation: Cadez et al (2000) – Web Site Design: Cooley et al (1999), Mobasher et al (2000), Nanopoulos and Manolopoulos (2000), Lee et al (2001) – Pirolli and Card (1999), Heer and Chi (2001, 2002) – Goal-Directed Search vs. Exploratory Search: Janiszewski (1998), Moe (2001), Brucks (1985), Jarboe and Mc. Daniel (1987) 45

Methodological Findings • Best model – 2 states, VAR(1), First-Order Markov, Page Level – 83% Hit Rate (Out-of-sample) • Memory is important – – Traditional Latent Class (40%) and Multinomial Probit (30%) First-Order (50%) dominates Zero-Order (20%) Page-Level States better than states at the Session or User VAR and Markov permit combination of smooth changes and jumps 46

Substantive Findings • Banner ads tend to encourage browsing oriented individuals but discourage continued browsing from deliberation oriented users • Including price information tends to discourage browsing oriented users but encourage deliberation oriented users • More home links leads to less use of home page by browsers and more by deliberation oriented users • Weekend users more likely to surf • Browsers who have purchased before are more likely to order • Rich set of relationships in dynamic browsing behavior 47

States at Page Transition Level State Time State Proce ss Zero. Order Page First. Order Number Log of Marginal States Density In- Out-of. Sample e Hit Rate (%) 1 -9378. 1 72. 05 65. 40 2 -9016. 9 79. 44 71. 42 3 -9064. 0 80. 34 70. 56 1 -8545. 4 83. 23 79. 95 2 -8428. 3 89. 71 83. 15 3 -8474. 0 89. 97 81. 14 Best 48

States at Session Transition Level State Time State Proce ss Zero. Order Sessio n First. Order Number Log of Marginal States Density In- Out-of. Sample e Hit Rate (%) 1 -9376. 1 73. 17 61. 56 2 -9051. 0 77. 90 70. 48 3 -9097. 7 78. 76 66. 14 1 -8573. 5 83. 05 73. 57 2 -8464. 9 88. 44 81. 48 3 -8487. 0 88. 73 78. 42 49

States at User Transition Level State Time User State Proce ss Zero. Order Number Log of Marginal States Density In- Out-of. Sample e Hit Rate (%) 1 -9411. 1 64. 38 61. 50 2 -9124. 2 70. 04 64. 12 3 -9193. 8 70. 85 63. 99 50

Alternative Model Specifications Log Margin al Density In-Sample Hit Rate (%) Out-of. Sample (%) Zero-Order Markov Model (1 State) -20410. 4 20. 48 12. 62 Zero-Order Markov Model (2 States) -19458. 3 28. 18 19. 02 First-Order Markov Model (1 State) -16444. 5 56. 06 51. 59 First-Order Markov Model (2 States) -16076. 0 58. 61 52. 08 Latent Class Model (1 State) -17849. 2 35. 47 30. 78 Latent Class Model (2 States) -17673. 9 44. 29 40. 21 Latent Class Model (3 States) -17722. 3 45. 29 36. 14 Independent -19086. 4 33. 23 30. 35 Only-Intercept -19335. 9 29. 37 23. 12 Intercept + VAR -13768. 4 71. 13 64. 38 Model 51

Parameter Estimates

Effects of Marketing Mix and Web Design on Hyperparameters Intercept Browsing Price Prese Prom nt o-tion Home Ads Links Acct Links Prod uct Links Shopping Cart -. 11 (. 02) -. 07 (. 02) . 05 (. 01) -. 04 (. 02) -. 04 (. 01) . 01 (. 01) Order -. 54 (. 08) -. 04 (. 01) . 05 (. 01) -. 06 (. 01) -. 02 (. 01) . 04 (. 01) . 01 (. 01) Shopping Cart . 27 (. 03) . 04 (. 01) -. 14 (. 01) -. 03 (. 01) . 01 (. 01) . 06 (. 01) . 04 (. 01) . 06 (. 02) . 08 (. 02) -. 03 (. 01) -. 07 (. 01) . 01 (. 01) -. 03 (. 01) . 02 (. 01) Deliberatio n Order 53

Effects of Session Context on Hyperparameters State Browsing Deliberation Categor y No. of Purc hase s Wee kend Othe r Site Visit s Other Book store s Time Dura tion Visit Dept h Shoppin g Cart -. 13 (. 06) -. 09 (. 01) . 03 (. 01) . 15 (. 03) -. 11 (. 02) -. 36 (. 06) Order . 33 (. 05) . 03 (. 02) . 01 (. 01) . 02 (. 03) . 01 (. 01) -. 02 (. 01) Shoppin g Cart . 01 (. 01) . 02 (. 01) . 04 (. 01) . 05 (. 01) . 02 (. 02) -. 05 (. 01) Order . 03 (. 01) -. 01 (. 01) . 05 (. 01) -. 01 (. 01) . 04 (. 01) 54

Effects of VAR Model State Browsing Categorie s Hom Acco e unt Cate gory Prod uct Infor Shop m Cart Orde r Shopping Cart . 02. 09 -. 01 -. 02 -. 01. 50 (. 012) (. 001) (. 002) (. 013) (. 003) Order -. 01. 13 -. 05 -. 02 -. 01. 03 -. 01 (. 008) (. 002) (. 001) (. 002) Shopping Cart . 01. 02 -. 06. 01 -. 01. 16 (. 002) (. 003) (. 004) (. 001) (. 006) Deliberati on Order . 02. 01 -. 03 -. 01. 03 -. 02 (. 003) (. 002) (. 004) (. 001) (. 002) 55

Estimated Results for the Two-state Hidden Markov Chain (Inverse of Waiting Time) (Starting Probabilities) P (Transition Probabilities) Browsing oriented state Deliberati on oriented state . 32 (. 01) . 26 (. 01) . 64 (. 01) . 36 (. 01) 0 1 1 0 56

Demographic Effects on Presence of Price Information Categori es Inter. Br D Age 2 Whit e Male Marri Colle Chil e g d d e Med. Inc High Inc Shopping -. 48. 04. 01. 06 -. 17. 26. 24. 22. 21. 03 Cart (. 02) (. 01) (. 02) Order -. 32. 16 -. 02. 18 -. 32. 07. 06. 05. 02. 07 (. 02) (. 01) (. 02) Shopping. 03. 01. 01 -. 07. 05 -. 01 -. 12. 10. 13 Cart (. 01) (. 02) (. 03) Order . 03. 01 -. 10. 04. 03 -. 19 -. 09. 04 -. 01 (. 01) (. 02) (. 01) (. 03) (. 02) 57

Consumer Heterogeneity: Price Presence Coefficient in Purchase State 58

Purchase Conversion

Will this user buy? 13. 8% 12. 3% 13. 2% 14. 3% 35. 3% Purchase User 1 Demographics Sex: Male Age: 55 Occupation: Service Worker State: Washington 52. 4%. . . 60

User 1: Purchase Conversion Predictions 61

0. 24% 0. 26% 0. 05% 0. 04% 0. 03%. . . No Purchase User 2 Demographics Sex: Female Age: 17 Occupation: Student State: Virginia 62

User 2: Purchase Conversion Predictions 63

Purchase Conversion Comparison Forecast Origin/Number of viewings during session Sample Estimatio n Holdout 1 2 3 4 5 6 First-Order Markov (1 State) 7. 2% (0. 26) 8. 2% (0. 28) 10. 1% (0. 31) 12. 7% (0. 34) 15. 3% (0. 36) 22. 4% (0. 43) Latent Class (2 States) 7. 4% (0. 27) 7. 8% (0. 27) 9. 5% (0. 30) 11. 3% (0. 32) 12. 8% (0. 34) 14. 5% (0. 36) Intercept + VAR 10. 4% (0. 31) 11. 6% (0. 33) 14. 9% (0. 36) 17. 0% (0. 38) 21. 4% (0. 42) 26. 0% (0. 45) Dynamic Multinomial Probit (1 State, Page-Level) 12. 4% (0. 34) 14. 0% (0. 35) 18. 7% (0. 39) 25. 1% (0. 44) 29. 0% (0. 46) 35. 8% (0. 49) Dynamic Multinomial Probit (2 States, Page-Level) 13. 3% (0. 48) 16. 3% (0. 52) 23. 4% (0. 60) 30. 9% (0. 65) 34. 4% (0. 67) 41. 5% (0. 69) First-Order Markov (1 State) 6. 5% (0. 35) 7. 5% (0. 37) 9. 6% (0. 42) 12. 6% (0. 47) 13. 7% (0. 49) 16. 6% (0. 53) Latent Class (2 States) 7. 2% (0. 82) 7. 3% (0. 82) 8. 6% (0. 89) 9. 5% (0. 93) 9. 9% (0. 94) 11. 2% (0. 99) Intercept + VAR 8. 4% (0. 88) 9. 3% (0. 92) 11. 8% (1. 02) 13. 5% (1. 08) 15. 3% (1. 13) 17. 7% (1. 21) Dynamic Multinomial Probit (1 State, Page-Level) 9. 3% (0. 92) 11. 4% (1. 01) 15. 2% (1. 13) 16. 4% (1. 17) 17. 8% (1. 21) 19. 0% (1. 24) Dynamic Multinomial Probit (2 States, Page-Level) 10. 4% (0. 97) 12. 8% (1. 06) 15. 2% (1. 14) 18. 0% (1. 21) 19. 1% (1. 24) 21. 2% (1. 29) Model 64

Hierarchical Specification

Hierarchical Specification 66