Searching Web Forums Amlie Marian Rutgers University Joint

Searching Web Forums Amélie Marian, Rutgers University Joint work with Gayatree Ganu 09/30/2013 Amélie Marian – Rutgers University

G a y a t r e e G a n u Forum Popularity and Search • Forums with most traffic [http: //rankings. big-boards. com] - BMW - 50 K uniq visitors/day - 25 M Posts - 0. 6 M Members - Despite popularity, forums lack good search capabilities Filipino Community Subaru Impreza Owners Rome Total War … Pakistan Cricket Fan Site Prison Talk Online Money making 09/30/2013 Amélie Marian – Rutgers University 2

G a y a t r e e Outline Multi-Granularity Search G a n Challenges u - Unstructured text - Background information omitted - Discussion digression Contributions Return each results at varying focus levels, allowing more or less context. (CIKM 2013) 09/30/2013 Egocentric Search Challenges - Multiple interpersonal relations with varying importance Contributions Proposed a multidimensional user similarity measure. Use authorship for improving personalized and keyword search. Amélie Marian – Rutgers University 3

G a y a t r e e Hierarchical Model • Hierarchy over objects at three searchable levels – pertinent sentences, larger posts, entire discussions or threads • Hierarchy – captures strength of association, containment relationship G a n u Dataset Thread 1 Post 1 Thread 2 Post 3 Post 4 Sent 5 Sent 6 2 Sent 1 Sent 2 Sent 3 Sent 4 2 2 Word 1 Word 2 09/30/2013 Word 4 • Lower levels for smaller objects • Edge represents containment • Edge weight of 2 indicates that the text in child was repeated in the text of parent Word 1 Amélie Marian – Rutgers University 4

Example Textual Results. G Query : hair loss a Top-4 Results Alternate Scoring Functions y. Post 1: (A) Aromasin certainly caused my hair loss and the hair started falling 14 days after the chemo. However, I bought myself a rather fashionable scarf to hide the baldness. I wear it everyday, a even at home. (B) Onc was shocked by my hair loss so I guess it is unusual on Aromasin. I had no tother side effects from Aromasin, no hot flashes, no stomach aches or muscle pains, no headaches or rnausea and none of the chemo brain. e Post 2: (C) Probably everyone is sick of the hair loss questions, but I need help with this falling hair. I ehad my first cemotherapy on 16 th September, so due in one week for the 2 nd treatment. (D) Surely the hair loss can’t be starting this fast. . or can it? . I was running my fingers at the nape of my neck and about five came out in my fingers. Would love to hear from anyone else have AC done G (Doxorubicin and Cyclophosphamide) only as I am not due to have the 3 rd drug (whatever that is - 12 a weekly sessions) after the 4 sessions of AC. Doctor said that different people have different side n effects, so I wanted to know what you all went through. (E) Have n’t noticed hair loss elsewhere, just the top hair and mainly at the back of my neck. (F) I thought the hair would start thining out u between 2 nd and 3 rd treatment, not weeks after the 1 st one. I have very curly long ringlets past my shoulders and am wondering if it would be better to just cut it short or completely shave it off. I am willing to try anything to make this stop, does anyone have a good recommendation for a shampoo, vitamins or supplements and (sadly) a good wig shop in downtown LA. tf*idf Sent (E) (4. 742) Sent (A) (4. 711) Sent (C) (4. 696) Sent (G) (4. 689) BM 25 Sent (D) (10. 570) Sent (B) (10. 458) Sent (H) (10. 362) Sent (E) (10. 175) HScore Post 2 (0. 131) Sent (G) (0. 093) Post 1 (0. 092) Sent (H) (0. 089) Post 3: My suggestion is, don’t focus so much on organic. Things can be organic and very unhealthy. I believe it when I read that nothing here is truly organic. They’re allowed a certain percentage. I think 5% of the food can not be organic and it still can carry the organic label. What you want is nonprocessed, traditional foods. Food that comes from a farm or a farmer’s market. Small farmers are not organic just because it is too much trouble to get the certification. Their produce is probably better than most of the industrial organic stuff. (G) Sorrytf*idf Jennifer, chemotherapy Score (t, d) = (1+log(tft, dand )) *treatment log(N/dffollowed t) * 1/Char. Length by hair loss is extremely depressing and you cannot prepare enough for falling hair, especially hair in clumps. (H) I am on femara and hair loss is non-stop, I had full head of thick hair. 09/30/2013 Amélie Marian – Rutgers University 5

G a y a t r e e G a n u Scoring Multi-Granularity Results Goal: Unified scoring for objects at multiple granularity levels – largely varying sizes – with inherent containment relationship Hierarchical Scoring Function (HScore) Score for node i with respect to search term t and having j children: =1 =0 … if i is a non-leaf node … if i is a leaf node containing t … if i is a leaf node not containing t ewij = edge weight between parent i and child j P(j) = number of parents of j C(i) = number of children of i 09/30/2013 Amélie Marian – Rutgers University 6

G a y a t r e e Effect of Size Weighting Parameter on HScore • Parameter controls the intermixing of granularities G a n u 20 18 16 Number of results in top-20 list 14 12 Threads 10 Posts 8 Sentences 6 4 2 0 0 0. 1 0. 2 0. 3 0. 4 0. 5 BM 25 Size parameter 09/30/2013 HScore Amélie Marian – Rutgers University 7

G a y a t r e e G a n u Multi-Granularity Result Generation 0. 1 Thread 1 2 Post 1 1. 6 Sent 1 1. 5 Sent 2 Thread 2 2. 5 Post 2 1. 4 Sent 3 Post 3 1. 3 Sent 4 0. 1 Sent 5 0. 1 Post 4 0. 4 Sent 6 Sorted Ordering: Post 3(2. 5), Post 1(2. 1), Post 2(2), Sent 1(1. 6), Sent 2(1. 5), Sent 3(1. 4), Sent 4(1. 3), Sent 6(0. 4), Sent 5(0. 1), Post 4(0. 1), Thread 1(0. 1), Thread 2(0. 1) For result size k=4, optimizing for the sum of scores: • Overlap: {Post 3, Post 1, Post 2, Sent 1} Sum Score = 8. 2 (minus 1. 6? ) • Greedy: {Post 3, Post 1, Post 2, Sent 6} Sum Score = 7. 0 • Best: {Post 3, Post 2, Sent 1, Sent 2} Sum Score = 7. 6 33% sample queries had overlap amongst at least 3 of top-10 results 09/30/2013 Amélie Marian – Rutgers University 8

G a y a t r e e G a n u Multi-Granularity Result Generation Goal: Generating a non-overlapping result set maximizing “quality” • Quality = Sum of scores of all results in the set • Maximal independent set problem (NP Hard) • Existing Algorithm: Lexicographic All Independent Sets (LAIS) outputs maximal independent set with polynomial delay in specific order 09/30/2013 Amélie Marian – Rutgers University 9

G a y a t r e e G a n u Optimal Algorithm for k-set (OAKS) • Fix node ordering by decreasing scores • Efficient OAKS Algorithm (typically k<<n): – Start with k-sized first independent set, i. e. , greedy – Branch from nodes preceding kth node of the set, check if maximal – Find new k-sized maximal sets, save in priority queue – Reject sets from priority queue where starting node occurs after current best set’s kth node 09/30/2013 Amélie Marian – Rutgers University 10

G a y a t r e e OAKS 0. 1 Thread 1 2. 1 G a n u 0. 1 2 Post 1 1. 6 Sent 1 1. 5 Sent 2 Thread 2 2. 5 Post 2 1. 4 Sent 3 Post 3 1. 3 Sent 4 0. 1 Sent 5 0. 1 Post 4 0. 4 Sent 6 Sorted Ordering: Post 3(2. 5), Post 1(2. 1), Post 2(2), Sent 1(1. 6), Sent 2(1. 5), Sent 3(1. 4), Sent 4(1. 3), Sent 6(0. 4), Sent 5(0. 1), Post 4(0. 1), Thread 1(0. 1), Thread 2(0. 1) For k=4, Greedy = {Post 3, Post 1, Post 2, Sent 6} Sum. Score=7. 0 In the 1 st iteration: Branches Branch from Sent 1, nodesremoving before Sent 6, all adjacent to Sent 1, {Post 3, Post 2, Sent 1} {Post 3, Post 2, Sent 1, Sent 2} Sum. Score = 7. 7 i. e. Sent 1, Maximal on Sent 2, first Sent 3, 4 nodes? Sent 4 YES! {Post 3 , Post 1, Sent 3, Sum. Score = 7. 3 {Post 3, Post 2, Sent 1, Sent 2} then complete to size Sent 4} k and insert in queue 09/30/2013 Amélie Marian – Rutgers University 11

G a y a t r e e G a n u Evaluating OAKS Algorithm Comparing OAKS Runtime Small overhead for practical k (=20) • Scoring time = 0. 96 sec • OAKS Result set generation time = 0. 09 sec Comparing LAIS and OAKS – 100 relatively infrequent queries with corpus frequency in range 20 -30, 30 -40… – OAKS is very efficient. Time required by OAKS depends on k 09/30/2013 OAKS improves over Greedy Sum. Score in 31% queries @top 20 Sets Evaluated Run Time (sec) Word Frequency LAIS OAKS 20 -30 57. 59 8. 12 0. 78 0. 12 30 -40 102. 07 5. 06 7. 88 0. 01 40 -50 158. 80 5. 88 26. 94 0. 01 50 -60 410. 18 6. 30 82. 20 0. 02 60 -70 716. 40 5. 26 77. 61 0. 01 70 -80 896. 59 8. 30 143. 33 0. 04 Amélie Marian – Rutgers University 12

G a y a t r e e G a n u Dataset and Evaluation Setting • Data collected from breastcancer. org – 31 K threads, 301 K posts, 1. 8 M unique sentences, 46 K keywords • 18 Sample Queries – e. g. , broccoli, herceptin side effects, emotional meltdown, scarf or wig, shampoo recommendation … • Experimental Search Strategies – top 20 results - Mixed-Hierarchy : Optimal mixed granularity result. - Posts-Hierarchy : Hierarchical scoring of posts only. - Posts-tf*idf : Existing traditional search. - Mixed-BM 25 09/30/2013 Amélie Marian – Rutgers University 13

G a y a t r e Graded Relevance Scale Exactly relevant answer, e Evaluating Perceived Relevance Mixed-Hierarchy G a n u Query = shampoo recommendation Relevant but too broad, Relevant but too narrow, Partially relevant answer, Not Relevant Crowd Sourced Relevance using Mechanical Turk - Over 7 annotations - Quality control -Honey pot questions - EM algorithm for consensus 09/30/2013 a = 0. 1 a = 0. 2 a = 0. 3 a = 0. 4 Rank = 1 Rel Broad Partial 2 Rel Broad Partial 3 Rel Broad Partial 4 Rel Broad Exactly Rel Broad 5 Rel Broad Exactly Rel Partial 6 Exactly Rel Rel Narrow 7 Rel Broad Exactly Rel Narrow Not Rel 8 Rel Broad Not Rel Partial 9 Rel Broad Rel Narrow Rel Broad Partial 10 Exactly Rel Narrow Partial Rel Narrow 11 Rel Broad Exactly Rel Not Rel 12 Rel Broad Exactly Rel Not Rel 13 Rel Broad Exactly Rel Partial Not Rel 14 Not Rel Exactly Rel Narrow Partial 15 Not Rel Exactly Rel Not Rel Broad 16 Not Rel Broad Rel Narrow Not Rel 17 Exactly Rel Broad Exactly Rel Not Rel 18 Exactly Rel Partial 19 Not Rel Broad Rel Narrow Not Rel 20 Not Rel Exactly Rel Partial Not Rel Amélie Marian – Rutgers University 14

G a y a t r e Mean Average Precision e Search MAP = = = Evaluating Perceived Relevance Discounted Cumulative Gain @20 @ 0. 1 0. 2 0. 3 = 0. 4 10 0. 98 0. 90 0. 70 20 0. 97 0. 95 0. 85 0. 66 25. 00 10 0. 76 0. 75 0. 77 0. 78 20. 00 Hierarchy 20 0. 72 0. 71 0. 73 0. 75 10. 00 Poststf*idf 10 0. 76 0. 73 0. 76 20 0. 74 0. 72 0. 73 Mixed BM 25 10 b=0. 75 0. 55 20 k=1. 2 0. 54 System G Mixed a Hierarchy n u Posts- Clearly, Mixed-H outperforms post only methods 09/30/2013 35. 00 30. 00 15. 00 0. 00 α=0. 1 α=0. 2 α=0. 3 α=0. 4 Mixed-hierarchy Posts-tf*idf Mixed-BM 25 b=0. 75 k 1=1. 2 Users perceive higher relevance of mixed granularity results Amélie Marian – Rutgers University 15

G a y a t r e e G a n u Ego. Centric Search • Previous technique did not take the authorship of posts into account • Some forum participants are similar, sharing same topics of interest or having the same needs, not necessarily at the same time – Rank similar author’s posts higher for personalized search • Some forum participants are experts, prolific and knowledgeable – Expert opinions carry more weight in keyword search • Author score to enhance personalized & keyword search 09/30/2013 Amélie Marian – Rutgers University 16

G a y a t r e e G a n u Author Score • Forum participants have several reasons to be linked auth 1 W(a, t) Topic 1 W(q, t) query 1 User Profiles: author 1 W(a 1, a 2) auth 2 Topic 2 query 2 author 3 - Location - Age - Cancer stage - Treatment -… author n Topic t query n auth n -Co-participation -Explicit References • Build a multidimensional heterogeneous graph over authors incorporating many relations • But, users assign different importance to different relations 09/30/2013 Amélie Marian – Rutgers University 17

G a y a t r e e G a n u Contributions Critical problem for leveraging authorship for search: Incorporating multiple user relations with varying importance learned egocentrically from user behavior Outline: • Author score computation using multidimensional graph • Personalized predictions of user interactions: authors most likely to provide answers • Re-ranking results of keyword search using author expertise 09/30/2013 Amélie Marian – Rutgers University 18

G a y a t r e e G a n u Multi-Dimensional Random Walks (MRW) c A= • Rooted RW for node similarity – Teleport back to root node with probability (1 -α) – Computes similarity of all nodes w. r. t root node 09/30/2013 Amélie Marian – Rutgers University a b c a 0 0 0 b 2 0 0 c 0 3 0 a b D= • Multidimensional RW– Heterogeneous Networks: – Transition matrix computed as A = 1 * A 1 + 2 * A 2 +. . . + n * An where i i = 1 and all i >= 0 – Egocentric weights For root node r : i (r) = j ew. Ai (r, m)/ Ak j ew. Ak (r, j) … m Ai and j Ak 3 a • Random Walks (RW) for finding most influential users – Pt+1 = M × Pt … till convergence – M = α(A + D) + (1 − α)E … relation matrix A, D for dangling nodes, uniform matrix E, α usually set to 0. 85 b 2 E= c a 0 0 0. 33 b 0 0 0. 33 c 0. 33 0 0 a b c a . 33 b . 33 c . 33 19

G a y a t r e e G a n u Personalized Answer Search • Link prediction by leveraging user similarities: – Given participant behavior, find similar users to the user asking question – Predict who will respond to this question • Learn similarities from first 90% training threads Link prediction enables • Relations used: – Topics covered in text, Co-participation in threads, Signature profiles, Proximity of posts suggesting which threads or which users to follow • MRW similarity compared with baselines: – Single relations – Path. Sim: • Existing approach for heterogeneous networks • Predefined paths of fixed length • No dynamic choice of path 09/30/2013 Amélie Marian – Rutgers University 20

G a n u Predicting User Interactions MAP for link prediction 0. 5 0. 4 MAP G a y a t r e e 0. 3 0. 2 0. 1 0 10 20 30 40 50 60 70 80 Top-K similar participants 90 100 Multidimensional RW has best prediction performance 09/30/2013 Amélie Marian – Rutgers University 21

G a y a t r e e G a n u Predicting User Interactions • Leverage content of the initial post to find users who are experts on the question – Topic. Score computed as cosine similarity between author’s history and initial post – User. Score = β * MRWScore + (1 - β) * Topic. Score Purely topical expertise MAP Purely MRW Neighbors β=0 β = 0. 1 β = 0. 2 β=1 Top 5 0. 52 0. 64 (8%) 0. 61 (4%) 0. 59 Top 10 0. 31 0. 50 (8%) 0. 49 (5%) 0. 46 Top 15 0. 24 0. 43 (8%) 0. 42 (6%) 0. 40 Top 20 0. 39 (6%) 0. 39 (7%) % Improvement over purely MRW 0. 37 09/30/2013 Amélie Marian – Rutgers University 22

• Non-rooted RW to find most influential expert users • Re-rank top-k results of IR scoring using author scores • Final score of post = ω*IR_score λ + (1 - ω)*Authority_score – Posts only, tf*idf scoring with size parameter 0. 81 0. 80 0. 79 0. 78 0. 77 0. 76 0. 75 0. 74 0. 73 0. 72 4% improvement 5% IR Score λ=0. 1 09/30/2013 1 00 00 70 0. 60 00 00 0. 5 0 00 01 00 00 1 0. 8 0. 9 4 0. 3 0. 2 0. 1 IR Score λ=0. 2 0 G a n u Enhanced Keyword Search MAP @ 10 G a y a t r e e Tradeoff Parameter ω Amélie Marian – Rutgers University Re-ranking search results with author score yields higher MAP relevance 23

G a y a t r e e G a n u Patient Emotion and st. Ructure Search USer tool(PERSEUS) - Conclusions • Designed hierarchical model and score that allows generating search results at several granularities of web forum objects. • Proposed OAKS algorithm for best non-overlapping result. • Conducted extensive user studies, show that mixed collection of granularities yields better relevance than post-only results. • Combined multiple relations linking users for computing similarities • Enhanced search results using multidimensional author similarity • Future Directions: – Multi-granular search on web pages, blogs, emails. Dynamic focus level selection. – Search in and out of context over dialogue, interviews, Q&A. – Optimal result selection for targeted advertising, result diversification – Time sensitive recommendations – Changing friendships, progressive search needs. 09/30/2013 Amélie Marian – Rutgers University 24

Thank you! 09/30/2013 Amélie Marian – Rutgers University

G a y a t r e e G a n u Why Random Walks? Rooted RW Examples (a) (b) r t r a b (c) 0. 6 r t 4 0. 4 2 c a Multi-dimensional Rooted RW Example 2 r 1 t r 2 r 1 1 0. 16 4 b A 1 1 c A 2 b (d) r 09/30/2013 a t c u 0. 26 score (b w. r. t r 1) = 0. 072 score (c w. r. t r 1) = 0. 096 0. 16 score (b w. r. t r 2) = 0. 097 score (c w. r. t r 2) = 0. 066 Amélie Marian – Rutgers University 26

G a y a t r e e G a n u LAIS 0. 1 Thread 1 2. 1 Thread 2 2 Post 1 2. 5 Post 2 Post 3 Sent 1 1. 5 Sent 2 Post 4 2 2 1. 6 0. 1 1. 4 Sent 3 1. 3 Sent 4 0. 1 Sent 5 0. 4 Sent 6 Sorted Ordering: Post 3(2. 5), Post 1(2. 1), Post 2(2), Sent 1(1. 6), Sent 2(1. 5), Sent 3(1. 4), Sent 4(1. 3), Sent 6(0. 4), Sent 5(0. 1), Post 4(0. 1), Thread 1(0. 1), Thread 2(0. 1) Greedy = {Post 3, Post 1, Post 2, Sent 6}. In the 1 st iteration: {Post 3, Post 2, Sent 1, Sent 2, Sent 6} {Post 3, Post 1, Sent 3, Sent 4, Sent 6} {Post 1, Post 2, Sent 6, Sent 5} {Post 3, Post 1, Post 2, Post 4) {Post 3, Sent 6, Thread 1} {Post 1, 09/30/2013 Post 2, Thread 2} Amélie Marian – Rutgers University 27

G a y a t r e e Current Search Functionality breastcancer. org G a n u Filtering criteria keyword search, member search Ranking based on date Posts only results 09/30/2013 Amélie Marian – Rutgers University 28